I wonder if they're optimising for profitability (while aiming to only keep user retention steady), or if this is really what the target users want.

I read anywhere that pages in japan are supposed to be chock full, you don't want any white space. Incidentally Yahoo Japan is very popular (although it does have different management, yahoo US only have as minority stake in it). There could be types of users elsewhere in the world who want the same kind of thing