Everyone knows it, but you're not allowed to say it, and you're definitely not allowed to say it in the UK or you will literally be arrested for speech.
In the long run they're still way ahead because they exploited those "low-quality" people for centuries, extracting all their labor and material resources for virtually nothing.
It's actually not at all clear that the British colonial empire was a net positive economically. It's also not always the case that they were exploiting the people, rather than the mineral deposits etc.
It also wasn't "centuries". The British Crown ruled India (which at the time contained Pakistan and Bangladesh) for less than a century, for instance.
Everyone knows it, but you're not allowed to say it, and you're definitely not allowed to say it in the UK or you will literally be arrested for speech.