It's been well documented that around 8-10 months, infants lose the ability to distinguish between phonemes that do not meaningfully contrast in their linguistic environment. An English speaker can certainly hear the tones, and may even be able to pick out different tones in isolation, but it takes significant training to be able to interpret them as contrastive as opposed to inflectional, etc.