It goes both ways. Aside from cultural differences, I would assume you would also feel excluded or isolated if you were the only non-American among a group of Americans who only spoke English, and you didn't understand or speak English.

Note that English is used as a universal language in international settings, and that you are a native English speaker. It's somewhat easier for you since they would know a least a little bit of English, while you wouldn't know their language unless you have some level of proficiency.

Also, Americans generally have a culture of accommodation or assimilation, while people from other cultures may not. The social environment when the host population interacts among each other as peers is also different than when interacting with foreigners, as it requires a separate form of interaction that isn't the norm. This applies to both Americans and other nationalities.

I agree, and it wouldn't surprise to hear that there are also Americans who are uninterested in trying to accomidate forgeiners, and I imagine that would be very discouraging for someone who tries to get out of their comfort zone as well.