The market is pricing in significant counterparty risk. It's possible Oracle does the buying, unpacking, and cooling of Nvidia servers and doesn't get paid.
> It's possible Oracle does the buying, unpacking, and cooling of Nvidia servers and doesn't get paid.
The beauty of being a cloud infra provider is you're selling shovels. OpenAI going bust? Doesn't matter too much for Oracle, there will always be someone willing to pay them for GPU compute capacity.
Even if the hype behind AI dies down, which I hope it does rather sooner than later, the fundamental aspects aren't vaporware like with the cryptocurrency craze - AI, even the relatively lackluster state we have today, has a ton of very useful usage cases that are actually working in the field.
> AI, even the relatively lackluster state we have today, has a ton of very useful usage cases that are actually working in the field.
Useful use cases at what price? It's totally possible that after VC money dries up the price increases far beyond what customers (both business and consumer) have been paying, resulting in demand destruction rendering the investment a net negative for Oracle.
Not nessesarily. We have providers of hosted oss models (even large sota ones), with competitive API pricing. They have no reason to subsidice their services, outside og initial market grab. But there is so many of them that it dosent look to bad.
Also buisnisses have invested to host things locally themselves as well.
> Not nessesarily. We have providers of hosted oss models (even large sota ones), with competitive API pricing. They have no reason to subsidice their services, outside og initial market grab. But there is so many of them that it dosent look to bad.
That price is holding at the current demand ratio of GPU availability/consumption. If the large companies such as OpenAI and Anthropic cease training new models and accepting loss-leader lines of business such as free consumer inference there is a reasonable chance of a GPU glut. This GPU glut may drive down prices Oracle can command for their cloud services.
The fundamental problem is not the market as it stands today. The problem is where will the market equilibrium shift from increased capacity and reduced demand due to VC subsidy reduction.
That might be precisely why OpenAI is pushing an over investment in infrastructure. When VCs are no longer willing to substitute compute, having more compute available than natural demand will drive the prices down.
Their customer base rapidly disappeared and they tried to paper over the losses with accounting fraud where they had fake sales of equipment because their actual customers disappeared. They had tons of equipment to sell with few customers left to buy them.
There's about to be a huge glut of GPU capacity and we still haven't figured out anything to actually do with all those GPUs that creates value at the scale of society.
Everything they do now steals value. Takes it from artists and bloggers and puts their money in the hands of CEOs and investors. Super efficient idea theft isn't creating new value for society because the value gained is always offset by the destruction of value we already had...
> there will always be someone willing to pay them for GPU compute capacity.
There is an hourglass shape to emerging technologies. Everyone uses commodity hardware and duct tape to try to win the race to the bottleneck, where everyone save 2 companies goes bankrupt. As it narrows companies have learned enough about the problem to start developing purpose built solutions. Once you pass the bottleneck enough of your engineers go to other companies that the custom solution becomes mainstream and competition grows again.
Google was working on custom chips for AI before we knew it was AI. They are going to survive either as the dominant player in AI or as the underlying platform everyone else builds on.
That leaves one other spot for everyone else racing to the eye of the needle. Anyone betting on Oracle to win on technology is silly, betting on Oracle with a bunch of generic GPUs that anyone can get is down right dumb.
Problem is, those Nvidia servers are useless to anyone who needs servers already racked in a datacenter. They're purposely made purely to run LLMs, and anything else would be a waste of time.
Nvidia is also fucking over anyone who buy these: datacenters depend on used hardware sales to recoup cost, sometimes getting up to half of what they originally paid for the hardware.... this hardware has no resale value.
It's literally garbage the moment it leaves the factory.
> datacenters depend on used hardware sales to recoup cost
Who? Ive worked for few big infra companies with millions to billions of DC assets. After depreciation and then some, say 3-5 years, theyre effectively scrap. Its more cost effective to buy new, denser, racks than continue to MRC on the stranded space and power. The reseller is cheaper/easier effectively paid to scrap the parts as e-waste and recover what they can.
Datacenters hire companies to do this on their behalf, as they don't have the internal know-how to do this. Even the big cloud companies do this.
They're on 3-5 year cycles, yet the hardware life has a good 8 years in it. The reseller sifts through what comes out, sees what is still alive, scraps what isn't, and resells the rest.
What you don't understand is this isn't a one way relationship. If you're a major cloud company, your hardware probably already is worthless. Meta (and other companies) use a non-standard case and rack "standard" called Open Compute (OCP)... there is no resale market for these, as normal datacenters use normal racks with normal cases. Meta has to pay a company to dispose of these and lose 100% of their investment.
But lets say we go to someone else that does use standard hardware, and the rest of the industry buys this stuff up (to maintain existing fleets that don't need upgraded yet, not everybody is on some ridiculous 3-5 year churn), the company you partnered with to deal with your e-waste is likely to either give you a much better rate or even pay you for your hardware (if it is in high demand).
These enterprise inference machines have zero resale value. Even the OCP stuff I mentioned above has a small market (theres a few smaller datacenters out there toying with OCP due to it having better density, but aren't willing to buy new to test it out), but there is no market for these inference SBCs.
See my sibling comment to this for more information on why the SBCs are uniquely weird.
Maybe i read too much in to “depend on used hardware sales.” Ive worked for 2/5 and 3/15 largest us companies doing cloud and infra stuff. Recovered costs from EOL hardware has just never ever mattered. Not even a rounding error on P&L and hardware/dc org has 1000 higher value priorities. Ill admit maybe the offset costs were squirreled away in finance but not visible to the business.
Even with zero resale value thats “fine.” Anytime Ive owned capacity planning it’d be more cost effective to pay someone a multiple of rack MRC to get the hardware out and free up the space and whips. The impediment was almost always free hands and coordination functions that were being spent on new adds rather than replacement.
E-waste disposal is a huge cost, and it might be entirely possible you're not seeing the cost, or you're not aware of what a badly negotiated contract looks like.
Also, a lot of the industry runs on incredibly poor margins. The only datacenter space in the world right now printing money is either owned by clouds or owned by the AI bubble (which are sometimes the same companies, or the cloud leasing space to the AI bubble).
Mostly, profits are eaten by power deals (this is why Facebook put their biggest important DCs up where the cheapest power in the US is) or property ownership (buying land, building the DC, paying property taxes, maintaining the building, etc, that shit aint cheap), and then you get to buy hardware and hopefully get customers.
Amazon, Google, Facebook, et al all cheat their way through every loophole known to man to keep the costs down and the profit high; not a lot of it is from scale, even though they're still trying to chase that to the end, too.
They're not GPUs in computers... they're effectively GPUs that are computers. They're a single board computer (SBC), with an ARM CPU that is approximately the scale for desktop but not the scale for enterprise, conjoined with 2 extremely large GPUs that are siblings to the desktop version.
Each GPU chip is approximately twice the size of the biggest desktop version, and it has two of them, and it roughly twice as dense as anything else you could do using normal enterprise-dense GPU deployments (ie, those rigs that have 4 x16 slots and fit dual-slot GPUs across 8 slots)
However, these are not the same chips, they are siblings: the major change is, depending on the generation, they contain 2x or 4x matrix math ALUs and no texture units.
Seeing as it is a "small" (for enterprise) CPU, and a GPU that is very poor at non-matrix GPU tasks; and since it doesn't have TUs or media controllers (thats the thing that does DP/HDMI and video decoding/encoding) it can't be some weird niche desktop.
Unless you're doing inference at home, this thing is useless. Inference at home, and even small scale enterprise inference, is very very niche. There isn't enough market to give these boards a second chance.
However, totally normal used server parts? Totally useful. Cases, fans, power supplies, motherboards, CPUs, RAM, HBA/RAID, NICs? All useful to end users, either smaller companies doing inexpensive onsite, smaller datacenters maintaining an existing fleet and not needing to upgrade yet, or people building home labs.
It's very rare that you'll be at the exact point where it does make sense to use the "not quite high-power GPU racks".
* Lowest short term costs favors "just use the Cloud" solutions, that will often have power, HVAC and space constraints that favors new GPUs.
* Lowest long term costs bias the cost analysis to OPEX, where old GPUs gets expensive very fast, as they use more power, HVAC and space for less compute.
