In theory, if this system were to be adopted, the Stock Exchange Operator could charge a small tariff so that some of the money that now goes to HFT's goes to the stock exchange, with the rest going to the shareholders. So in theory, adopting this more optimal system would be financially beneficially to both the corporations and the Stock Exchange (and in theory, those two are the only two players who's agreement is needed to change the system). So I wonder why the optimal, auction based system does not get adopted - is there some reason for the status quo other than the normal inertia and stupidity of large institutions?
Not a good theory. All the players in this game believe they are above average among players in this game.
They all want the system to be exploitable because they all believe they will exploit it better than their peers.
At the very least better than the investors (the real suckers here).
You are falling into the game theory economics trap. People are highly irrational actors who don't even act in their own self interest most of the time, nevermind some form of optimal rational behavior.
You are falling into the behavioral economics trap. In general, the big movers and shakers in financial markets act in quite economically rational ways. There is a natural selection process at work where the people who are good at using their brain to accumulate money, accumulate money, and thus have more impact in the market. If you look at many cases where wealthy market players appear to be acting irrationally, what you'll often see is they are acting in a personally rational manner, but are operating in a context with very perverse incentives. So I am curious if anyone in the HN commentariat knows the specific perverse incentives involved that prevent this more optimal system from happening. ( Joe Retail investor can be quite irrational. But I'm wondering why the Stock Exchanges and big corporations accept the status quo with regards to HFT.)
"In general, the big movers and shakers in financial markets act in quite economically rational ways"
They are certainly more intelligent, knowledgeable and invest more rationally but I've seen nothing to make me believe that they lack or compensate for the types of positivity biases that makes everyone think they are above average and can beat their peers.
We weren't talking about the contexts of the investment choices they make (where I mostly agree with you) but about changing to a system where the winners can't exploit the system to extract far more money than the value they put in.
If they are A-type human beings they have a positivity bias if they have a human brain. You don't need perverse incentives to keep a system that over-rewards winners if everyone thinks they are, or soon will be, the winners.
That said, I would also be interested in knowing about any perverse incentives here, I just don't think they are necessary.