Wouldn't this depend on location and source? I used to live in Scotland, where sheep graze on grass in fields that don't really take away space for vegetables (unless you want to survive on potatoes and cabbage), and they're local. So I assumed they're not that bad.
Most of the sheep grazing land in the UK isn't natural - natural trees and vegetation were cleared to make way for more sheep. It's had a large impact on the biodiversity here.
Sheep grazing also strips the land bare of deep and complex vegetation upending whole ecosystems. Perhaps in a futile or misguided gesture I try to keep my consumption of lamb to an occasional treat rather than a regular thing.
Ruminants inherently have large methane output that drives up climate impact. There are only very early efforts to reduce methane output through diet change.