I think the issue here is that it's hard work for a native English speaker to keep track of the correctness of every single vowel sound because in English so many are elided or become "uh".
Listen carefully to different English accents, or even better try and mimic them.
There's a massive variety of vowel sounds in English: Sydney, Irish, Boston, Indian, etcetera.
English speakers can often hear the differences, and many people can produce the different vowels when mimicking the accents (country, city, person, foreigner).
I did not deny the fact that there is a greater variety of vowel sounds available in English. I merely doubt its explanatory power for the phenomenon you describe. But perhaps I am confused about exactly what that phenomenon is.
