Modelled dietary scenarios often fail to reflect true dietary practice and do not account for variation in the environmental burden of food due to sourcing and production methods. Here we link dietary data from a sample of 55,504 vegans, vegetarians, fish-eaters and meat-eaters with food-level data on greenhouse gas emissions, land use, water use, eutrophication risk and potential biodiversity loss from a review of 570 life-cycle assessments covering more than 38,000 farms in 119 countries. Our results include the variation in food production and sourcing that is observed in the review of life-cycle assessments. All environmental indicators showed a positive association with amounts of animal-based food consumed. Dietary impacts of vegans were 25.1% (95% uncertainty interval, 15.1-37.0%) of high meat-eaters (≥100 g total meat consumed per day) for greenhouse gas emissions, 25.1% (7.1-44.5%) for land use, 46.4% (21.0-81.0%) for water use, 27.0% (19.4-40.4%) for eutrophication and 34.3% (12.0-65.3%) for biodiversity. At least 30% differences were found between low and high meat-eaters for most indicators. Despite substantial variation due to where and how food is produced, the relationship between environmental impact and animal-based food consumption is clear and should prompt the reduction of the latter.