In the UK, lamb is pretty much an everyday meat (well - every week...) but goat is regarded as pretty exotic - common mainly in Afro-caribbean takeaways or restaurants. (I'm from the South-East so this might not be true in other regions).
Mutton - I don't recall seeing very often at all in butcher shops. As the meat in takeway - it's more common but sometimes the terminology isn't precise. Some Indian takeaways use the word "meat" when they mean something sheep-based. And "mutton" sometimes means "goat" (and vice-versa). And "mutton" sometimes is just lamb...
Same in South Africa...."mutton" usually refers to lamb. almost no one has a "lamb curry" or "lamb bunny" (Google it). It's "mutton curry" or a "mutton bunny".
I heard this sentiment from co-workers and friends who didnt grow up eating goat. I think it boils down to two different things:
1. "meat musk" (lack of a better word, lol): People dont realize how pungent beef can be, its just that most Americans are accustomed to it. You can really smell this with grass-fed beef. I think goat/lamb meat is "musky" for the same reasons, as they eat mainly from foraging/hay.
2. preparation/cooking: If you plan on cooking goat like a steak, its not a gonna taste good. Generally, most dishes require that the goat is simmered or slow-cooked to make the meat more tender. Spices also make a big difference here with the dish.