I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.
It doesn't really matter why anyone is in the market. The fact is that they are posting prices to the limit order book, and in doing so providing liquidity and accepting risk and providing a service. The only way the OP could get executed passively is by offering a price equal to or better than the market price, yes? So someone got a better price (or more liquidity at the prevailing price) by him being there.
There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.
Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.
The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.
It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.
Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.
I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.
I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.
We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.
By itself, closing a deal doesn't mean you provided value. It means the other party thinks you provided value to them. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)
If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray you are not the sucker. Locally, it's totally zero-sum.
Now maybe the whole system facilitates real transactions (with non-speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.
You're still missing it. Every single additional limit order in the market creates real value. If I want to buy 500 shares of company A, and one market participant is offering 250 at $10. Another is offering 250 at $11. It is going to cost me 250 x 10 + 250 x 11 = $5250. Now, the "short term speculator" you are complaining about shows up and puts his tiny little order in, he is offering 1 share at $10. Now the equation if you want to execute is 251 x 10 + 249 x 11 = $5249. He/she just saved you $1. One real dollar. As someone who wants to get a transaction done on a market, more people posting bids and offers cannot possibly hurt you. Only help you. You have the complete freedom to decide to meet their offered price, to cross the spread, to pick up that carton of milk at the 7-11 and walk to the till and make the transaction.
Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.
In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.
How wide are bid/offer spreads these days? How can that possibly be a bad thing?
(Edit: "limit order" looks like technical jargon. If so, I don't know what it means. Maybe you didn't say what I thought you said)
In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.
On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?
My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non-speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it harder, for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?
Now maybe a good speculator tends to provide a good service to non-speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. (Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)
I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side of the trade, that I'll have to pay an enormous spread or be left holding a position I want to be out of. An international corporation can be more confident of planning their cashflows knowing that the FX market will always be liquid, and will be quoting at most 2 ticks wide in the majors 24 hours a day every day. This is the oil of an economy, the freedom to allocate capital when you want, where you want and pay the smallest possible friction costs to do so.
Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand.
Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.
" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.
To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.
My definition of value is simple: something is valuable when it manages to raise people's utility functions (right now, that's about as rigorous as I can be). A trade that happens to be closed by a plain old investor is valuable. If it's another speculator, however, the value is zero. Only consequences ultimately matter.
But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.
> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?
The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.
When Pareto Optimum and Nash Equilibrium are at odds, life sucks.
I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.