What are the "more desirable parts of the city" with "restaurants and shops?" Union Square? No. SOMA? Not any more. Mission Bay? Boring, plus ballpark traffic. Dogpatch? The cheap warehouses are taken. Financial district? Old skyscrapers.
Mid-market, near Twitter and City Hall? You've got to be kidding.
Sounds like you just don't like SF. That's ok but it doesn't actually explain anything.
Same as NYC, there's a concentration of multi-millionaires that choose to live there. I highlight them specifically because they can live lavishly anywhere they want. That's not a... mistake?