> why such stubborn holdouts exist and why can't the government work with them
Because if the government can't exercise eminent domain then those people are incentivized to extract maximum value from the government over the land. They may not want to sell it, or they may charge an exorbitant price.
It's kind of strange how the law recognizes that land ownership is a source of power yet does absolutely nothing against it. If the government has to override the law, just imagine how much they distort the private economy, where there is no easy way out. So many apartments couldn't be built and so many people could not be housed.
Because if the government can't exercise eminent domain then those people are incentivized to extract maximum value from the government over the land. They may not want to sell it, or they may charge an exorbitant price.