You don't have to. All it takes is a few that will soak up the population that want affordable housing. 99% of the rest of the land will be left for you to live in as you see fit. Just don't expect your housing to be an investment or subsidies for your extra cost of infrastructure and energy.