Are you saying that height doesn't matter, or just that being tall doesn't give you an advantage over being average? I've always heard it does, but that's just anecdotal. Short people are definitely at a disadvantage though. Height matters.
This is likely due to the fact that American men average a few inches taller than American women. Biology and statistical sampling plus a little bit of confirmation bias.