In theory they could live anywhere they chose. But the gap between theory and practice is narrower in theory than it is in practice.

My parents came to Toronto from Ireland in the 1960s and also experienced their share of covert racism. It may be hard to imagine that a city that is so cosmopolitan† today was so repressive and ethnocentric just a few decades ago.

† Notwithstanding the Toronto Sun, et al.

Given that it was 60 years ago, I'm not so surprised.

For what is worth, even today the majority of landlords in Japan will not rent to foreigners.