Optane suffered the same fate as every other storage technology over the decades with a similar value proposition. Broadly, the notional value of storage technology like Optane is that it offers a drop-in improvement in storage performance without requiring any software modification, albeit at a significant cost premium. The theory that this market exists in any significant way never pans out in practice.
On one hand, you have a large market of people that are not sensitive enough to storage performance for the extra cost to be worth it. Ordinary storage is perfectly adequate for their needs and they would see little benefit in transparently making the storage their software was designed for faster.
On the other side, there are people who care about storage performance a lot. So much so, that they will design and modify their software to take full advantage of the storage and system characteristics. They use the storage so well that the gains by putting something in the middle will be marginal or non-existent, and certainly not worth the extra cost —- they would gain just as much by adding more storage devices.
The effective target market always ends up being “people who care a lot about storage performance but use their storage in the most naive way possible”, which isn’t that big of a market in practice. Using good software design to achieve similar results on commodity hardware is almost always the better option.
"The theory that this market exists in any significant way never pans out in practice."
The transition from HDD to SSD technology is a counter example to your claim. It was a drop-in replacement (same SATA interface) and the tech significantly improved performance without any other software modification needed.
>Octane was an incremental improvement in performance when compared to SSD
Optane PM was byte-addressable and had latency of ~300 ns. It rendered the entire block storage abstraction obsolete. It was so radical that using it effectively would require throwing out the assumptions all OSes have made about IO for the past fifty years.
The fact that it offered an incremental improvement even when its unique capabilities were completely ignored shows how phenomenally capable the technology was.
Didn't SSD's get a start with laptops and mobile devices? There the advantage of being shock resistant is massive and unsolvable with any other solution.
For a superior but new tech to stick, it has to find a viable market to be first self-sustaining, only then can it attack larger and more lucrative markets (Back in those days, PC was probably more lucrative than mobile). Only when SSDs matured enough, did they start eating up the entire storage market.
With SSDs you really feel the performance difference (especially latency) in everyday life. Someone who had experienced an SSD never wanted to go back to a HDD for storing anything but data, since program start-up times are just so much faster when there is no mechanical seeking involved.
The difference between Optane and an SSD was never so striking.
At least for consumer devices, I think the 'primordial soup' spark of motivation was the original lean and mean netbooks (with the initial push to web apps over running full clients locally, before they became low spec windows laptops), with compact flash ATA adapters to displace HDDs. Power consumption was a factor as well IIRC
>Broadly, the notional value of storage technology like Optane is that it offers a drop-in improvement in storage performance without requiring any software modification, albeit at a significant cost premium.
I don't think so at all. The value of Optane was that it was dramatically different, offering byte-addressable storage with latency of hundreds of nanoseconds. Getting the most out of this requires radically redesigning how modern systems store data, including ditching the idea of block storage.
Optane was the fastest traditional SSD on the market by a pretty wide margin, but that's not all it could have been.
>They use the storage so well that the gains by putting something in the middle will be marginal or non-existent, and certainly not worth the extra cost.
I don't agree with this either. How do you propose to get latency down to match Optane using ordinary SSDs? It doesn't matter how much hardware you throw at the problem; parallelism cannot reduce latency. You need another storage system to hit those targets. Maybe that can be a more traditional DRAM cache with battery backup, but that's only adequate in bursty workloads.
Putting "something in the middle" isn't worth it if the rest of the architecture stays the same. But it could bring about tremendous speedups if the new capabilities were used.
What about the 1-10 billion dollar company whose business isn’t technology, like say some mid tier insurance company running MySQL with a combination of off the shelf specialists software and in-house software. They would benefit from faster storage and they can’t tune their stack like Facebook
Obviously it failed as a product but I am not so sure that the cause is as you lay it out
On one hand, you have a large market of people that are not sensitive enough to storage performance for the extra cost to be worth it. Ordinary storage is perfectly adequate for their needs and they would see little benefit in transparently making the storage their software was designed for faster.
On the other side, there are people who care about storage performance a lot. So much so, that they will design and modify their software to take full advantage of the storage and system characteristics. They use the storage so well that the gains by putting something in the middle will be marginal or non-existent, and certainly not worth the extra cost —- they would gain just as much by adding more storage devices.
The effective target market always ends up being “people who care a lot about storage performance but use their storage in the most naive way possible”, which isn’t that big of a market in practice. Using good software design to achieve similar results on commodity hardware is almost always the better option.