Per the author’s links, he warned that deep learning was hitting a wall in both 2018 and 2022. Now would be a reasonable time to look back and say “whoops, I was wrong about that.” Instead he seems to be doubling down.
> expert in human language development and cognitive neuroscience, Gary is a futurist able to accurately predict the challenges and limitations of contemporary AI
I'm struggling to reconcile how these connect and he has been installed as Head of AI at Uber. Reeks of being a huckster
>...held the position briefly after Uber acquired his company, Geometric Intelligence, in late 2016. However, Marcus stepped down from the directorship in March 2017,
Indeed. A mouse that runs through a maze may be right to say that it is constantly hitting a wall, yet it makes constant progress.
An example is citing Mr Sutskever's interview this way:
> in my 2022 “Deep learning is hitting a wall” evaluation of LLMs, which explicitly argued that the Kaplan scaling laws would eventually reach a point of diminishing returns (as Sutskever just did)
which is misleading, since Sutskever said it didn't hit a wall in 2022[0]:
> Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling
The larger point that Mr Marcus makes, though, is that the maze has no exit.
> there are many reasons to doubt that LLMs will ever deliver the rewards that many people expected.
That is something that most scientists disagree with. In fact the ongoing progress on LLMs has already accumulated tremendous utility which may already justify the investment.
a contrarian needs to keep spruiking the point, because if he relents, he loses the core audience that listened to him. That's why it's also the same with those who keep predicting market crashes etc.
The same can be said about hucksters of all stripes, yes.
But maybe not contrarians/non-contrarians? They are just the agree/disagree commentators. And much of the most valuable commentary is nuanced with support for and against their own position. But generally for.
Is deep learning approaching a wall? - He doesn't make a concrete prediction, which seems like a hedge to avoid looking silly later. Similarly, I noticed a hedge in this post:
Of course it ain’t over til it’s over. Maybe pure scaling ... will somehow magically yet solve ...
---
But the paper isn't wrong either:
Deep learning thus far is data hungry - yes, absolutely
Deep learning thus far is shallow and has limited capacity for transfer - yes, Sutskeyer is saying that deep learning doesn't generalize as well as humans
Deep learning thus far has no natural way to deal with hierarchical structure - I think this is technically true, but I would also say that a HUMAN can LEARN to use LLMs while taking these limitations into account. It's non-trivial to use them, but they are useful
Deep learning thus far has struggled with open-ended inference - same point as above -- all the limitations are of course open research questions, but it doesn't necessarily mean that scaling was "wrong". (The amount of money does seem crazy though, and if it screws up the US economy, I wouldn't be that surprised)
Deep learning thus far is not sufficiently transparent - absolutely, the scaling has greatly outpaced understanding/interpretability
Deep learning thus far has not been well integrated with prior knowledge - also seems like a valuable research direction
Deep learning thus far cannot inherently distinguish causation from correlation - ditto
Deep learning presumes a largely stable world, in ways that may be problematic - he uses the example of Google Flu Trends ... yes, deep learning cannot predict the future better than humans. That is a key point in the book "AI Snake Oil". I think this relates to the point about generalization -- deep learning is better at regurgitating and remixing the past, rather than generalizing and understanding the future.
Lots of people are saying otherwise, and then when you call them out on their predictions from 2 years ago, they have curiously short memories.
Deep learning thus far works well as an approximation, but its answers often cannot be fully trusted - absolutely, this is the main limitation. You have to verify its answers, and this can be very costly. Deep learning is only useful when verifying say 5 solutions is significantly cheaper than coming up with one yourself.
Deep learning thus far is difficult to engineer with - this is still true, e.g. deep learning failed to solve self-driving ~10 years ago
---
So Marcus is not wrong, and has nothing to apologize for. The scaling enthusiasts were not exactly wrong either, and we'll see what happens to their companies.
It does seem similar to be dot com bubble - when the dust cleared, real value was created. But you can also see that the marketing was very self-serving.
Stuff like "AGI 2027" will come off poorly -- it's an attempt by people with little power to curry favor with powerful people. They are serving as the marketing arm, and oddly not realizing it.
"AI will write all the code" will also come off poorly. Or at least we will realize that software creation != writing code, and software creation is the valuable activity
I think it would help if either side could be more quantitative about their claims, and the problem is both narratives are usually rather weaselly. Let's take this section:
>Deep learning thus far is shallow and has limited capacity for transfer - yes, Sutskeyer is saying that deep learning doesn't generalize as well as humans
But they do generalize to some extent, and my limited understanding is that they generalize way more than expected ("emergent abilities") from the pre-LLM era, when this prediction was made. Sutskever pretty much starts the podcast saying "Isn’t it straight out of science fiction?"
Now Gary Marcus says "limited capacity for transfer" so there is wiggle room there, but can this be quantified and compared to what is being seen today?
In the absence of concrete numbers, I would suspect he is wrong here. I mean, I still cannot mechanistically picture in my head how my intent, conveyed in high-level English, can get transformed into working code that fits just right into the rather bespoke surrounding code. Beyond coding, I've seen ChatGPT detect sarcasm in social media posts about truly absurd situations. In both cases, the test data is probably outside the distribution of the training data.
At some level, it is extracting abstract concepts from its training data, as well as my prompt and the unusual test data, even apply appropriate value judgements to those concepts where suitable, and combine everything properly to generate a correct response. These are much higher-level concepts than the ones Marcus says deep learning has no grasp of.
Absent quantifiable metrics, on a qualitative basis at least I would hold this point against him.
On a separate note:
> "AI will write all the code" will also come off poorly.
On the contrary, I think it is already true (cf agentic spec-driven development.) Sure, there are the hyper-boosters who were expecting software engineers to be replaced entirely, but looking back, claims from Dario, Satya, Pichai and their ilk were were all about "writing code" and not "creating software." They understand the difference and in retrospect were being deliberately careful in their wording while still aiming to create a splash.
Several OpenAI people said in 2023 that they were surprised by the acceptance of the public. Because they thought that LLMs were not so impressive.
The public has now caught up with that view. Familiarity breeds contempt, in this case justifiably so.
EDIT: It is interesting that in a submission about Sutskever essentially citing Sutskever is downvoted. You can do it here, but the whole of YouTube will still hate "AI".
In what way do you consider that to be the case? IBM's Watson defeated actual human champions in Jeopardy in 2011. Both Walmart and McDonald's notably made large investments shortly after that on custom developed AI based on Watson for business modeling and lots of other major corporations did similar things. Yes subsidizing it for the masses is nice but given the impressive technology of Watson 15 years ago I have a hard time seeing how today's generative AI is science fiction. I'm not even sure that the SOTA models could even win Jeopardy today. Watson only hallucinated facts for one answer.
When Watson did that, everyone initially was very impressed, but later it felt more like it was just a slightly better search engine.
LLMs screw up a lot, sure, but Watson couldn't do code reviews, or help me learn a foreign language by critiquing my use of articles and declination and idiom, nor could it create an SVG of a pelican riding a bicycle, nor help millions of bored kids cheat on their homework by writing entire essays for them.
What exactly are they doing? I've seen a lot of hype but not much real change. It's like a different way to google for answers and some code generation tossed in, but it's not like LLMs are folding my laundry or mowing my lawn. They seem to be good at putting graphic artists out of work mainly because the public abides the miserable slop produced.
Any fool could have anticipated the eventual result of transformer architecture if pursued to its maximum viable form.
What is impressive is the massive scale of data collection and compute resources rolled out, and the amount of money pouring into all this.
But 10 years ago, spammers were building simple little bots with markov chains to evade filters because their outputs sounded plausibly human enough. Not hard to see how a more advanced version of that could produce more useful outputs.
Any fool could have seen self driving cars coming in 2022. But that didn't happen. And still hasn't happened. But if it did happen, it would be easy to say:
"Any fool could have seen this coming in 2012 if they were paying attention to vision model improvements"
Everyone who lives in the show belt understands that unless a self driving car can navigate icy, snow-covered roads better than humans can, it's a non-starter. And the car can't just "pull over because it's too dangerous" that doesn't work at all.
That works fine. Self driving doesn’t need to be everything for all conditions everywhere.
Give me reliable and safe self driving for Interstate highways in moderate to good weather conditions and I would be very happy. Get better incrementally from there.
I live solidly in the snow belt.
Autopilot for planes works in this manner too. Theoretically a modern airliner could autofly takeoff to landing entirely autonomously at this point, but they do not. They decrease pilot workload.
If you want the full robotaxi panacea everywhere at all times in all conditions? Sure. None of us are likely to see that in our lifetime.
We do not. We have much better cruise control and some rudimentary geofenced autonomy. In our lifetimes, neither you nor I will be able to drive in a car that, based on deep learning training on a corpus of real world generated data, goes wherever we want it to whenever we want it to, autonomously.
> But for API use, the models are easily substituted, so market share is fleeting. The LLM interface being unstructured plain text makes it simpler to upgrade to a smarter model than than it used to be to swap a library or upgrade to a new version of the JVM.
Agree that the plain text interface (which enables extremely fast user adoption) also makes the product less sticky. I wonder if this is part of the incentive to push for specialized tool calling interfaces / MCP stuff - to engineer more lock in by increasing the model specific surface area.
Sure, every business owner has incentives that point to delivering a worse product (eg cheaper pizza ingredients increase margins). For most businesses there is a strong counteracting incentive to do a great job so the customer returns next week.
The key variable is how long that gap of time is. In the online dating example, if the dating app does a sufficiently great job you will never return. A milder version: if the used car salesman gives you great value, you might be back in 10 years. This creates very weak incentives for good service, so more predatory tactics dominate.
The universal theme with general purpose technologies is 1) they start out lagging behind current practices in every context 2) they improve rapidly, but 3) they break through and surpass current practices in different contexts at different times.
What that means is that if you work in a certain context, for a while you keep seeing AI get a 0 because it is worse than the current process. Behind the scenes the underlying technology is improving rapidly, but because it hasn’t cusped the viability threshold you don’t feel it at all. From this vantage point, it is easy to dismiss the whole thing and forget about the slope, because the whole line is under the surface of usefulness in your context. The author has identified two cases where current AI is below the cusp of viability: design and large scale changes to a codebase (though Codex is cracking the second one quickly).
The hard and useful thing is not to find contexts where the general purpose technology gets a 0, but to surf the cusp of viability by finding incrementally harder problems that are newly solvable as the underlying technology improves. A very clear example of this is early Tesla surfing the reduction in Li-ion battery prices by starting with expensive sports cars, then luxury sedans, then normal cars. You can be sure that throughout the first two phases, everyone at GM and Toyota was saying: Li-ion batteries are totally infeasible for the consumers we prioritize who want affordable cars. By the time the technology is ready for sedans, Tesla has a 5 year lead.
> The universal theme with general purpose technologies is 1) they start out lagging behind current practices in every context 2) they improve rapidly, but 3) they break through and surpass current practices in different contexts at different times.
I think you should say succesful "general purpose technologies". What you describe is what happens when things work out. Sometimes things stall at step 1, and the technology gets relegated to a foot note in the history books.
We don’t argue that microwaves will be ubiquitous (which they aren’t, but close enough). We argue that microwaves are not an artificial general barbecue, as the makers might wish were true.
And we argue that microwaves will indeed never replace your grill as the makers, again, would love you to believe.
Your reasoning would be fine if there were a clear distinction, like between a microwave and a grill.
What we actually have is a physical system (the brain) that somehow implements what we know as the only approximation of general intelligence and artificial systems of various architectures (mostly transformers) that are intended to capture the essence of general intelligence.
We are not at the microwave and the grill stage. We are at the birds and the heavier-than-air contraptions stage, when it's not yet clear whether those particular models will fly, or whether they need more power, more control surfaces, or something else.
Heck, the frontier models have around 100 times lower number of parameters than the most conservative estimate of the equivalent number of parameters of the brain: the number of synapses. And we are like "it won't fly".
There was a lot of hubris around microwaves. I remember a lot of images of full chickens being roasted in them. I've never once seen that "in the wild" as it were. They are good for reheating something that was produced earlier. Hey the metaphor is even better than I thought!
There are many years since I have switched to cooking only with microwaves, due to minimum wasted time and perfect reproducibility. And I normally eat only food that I cook myself from raw ingredients.
Attempting to roast a full chicken or turkey is not the correct way to use microwaves. You must first remove the bones from the bird, then cut the meat into bite-sized chunks. After using a boning knife for the former operation, I prefer to do the latter operation with some good Japanese kitchen scissors, as it is simpler and faster than with a knife.
If you buy turkey/chicken breasts or thighs without bones, then you have to do only the latter operation and cut them into bite-sized pieces.
Then you can roast the meat pieces in a closed glass vessel, without adding anything to the meat, except salt and spices (i.e. no added water or oil). The microwave oven must be set to a relatively low power and long time, e.g. for turkey meat to 30 minutes @ 440 W and for chicken to less time than that, e.g. 20 to 25 minutes. The exact values depend on the oven and on the amount of cooked meat, but once determined by experiment, they remain valid forever.
The meat cooked like this is practically identical to meat cooked on a covered grill (the kind with indirect heating, through hot air), but it is done faster and without needing any supervision. In my opinion this results in the tastiest meat in comparison with any other cooking method. However, I do not care about a roasted crust on the meat, which is also unhealthy, so I do not use the infrared lamp that the microwave oven has for making such a crust.
Vegetable garnishes, e.g. potatoes, must be cooked at microwaves separate from the meat, as they typically need much less time than meat, usually less than 10 minutes (but higher power). Everything must be mixed into the final dish after cooking, including things like added oil, which should better not be heated at great temperatures.
Even without the constraints of a microwave oven, preparing meat like this makes much more sense than cooking whole birds or fish or whatever. Removing all the bones and any other inedible parts and also cutting the meat into bite-sized pieces before cooking wastes much less time than when everybody must repeat all these operations every time during eating, so I consider that serving whole animals at a meal is just stupid, even if they may look appetizing for some people.
Neither "roasted" nor "steamed" are completely appropriate for this cooking method.
While there is steam in the vessel, it comes only from the water lost from the meat, not from any additional water, and the meat is heated by the microwaves, not by the steam.
Without keeping a lid on the cooking vessel, the loss of water is too rapid and the cooked meat becomes too dry. Even so, the weight of meat is reduced to about two thirds, due to the loss of water.
In the past, I was roasting meat on a covered grill, where the air enclosed in it was heated by a gas burner through an opening located on one side, on its bottom. With such a covered grill, the air in which the meat was cooked would also contain steam from the water lost by the meat, so the end result was very similar to the microwave cooking that I do now, also preventing the meat from becoming too dry, unlike with roasting on an open grill, while also concentrating the flavor and avoiding its dilution by added water or oil.
I greatly appreciate this type of thinking. If you debone the meat yourself, do you make stock or use the bones in any way? You obviously care about personal process optimization and health factors, and I'm curious to what extent you are thinking of the entire food/ingredient supply chain.
> There are many years since I have switched to cooking only with microwaves, due to minimum wasted time and perfect reproducibility. And I normally eat only food that I cook myself from raw ingredients.
Some of us also like the food tasting nice besides the reproducibility of the results.
Like I have said, meat prepared in this way is the tastiest that I have ever eaten, except if you are one of those who like real meat less than the burned crust of meat.
Even for the burned-crust lovers, many microwave ovens have lamps for this purpose, but I cannot say whether the lamps are good at making tasty burned crusts, because I have never used them, since I like more to eat meat than to eat burned crusts.
The good taste of meat prepared in this way is not surprising, because the meat is heated very uniformly and the flavor of the meat is not leached away by added water or oil, but it is concentrated due to the loss of water from the meat, so the taste is very similar to that of meat well cooked on a grill, neither burned nor under-cooked, and also without adding anything but salt and spices.
“Burned crust” is not what people want. If it’s burnt it went too far. The fact that you are equivocating the two makes me think you haven’t ever had properly cooked meat or are very confused about what “burned” means. https://www.thetakeout.com/how-to-brown-meat-while-cooking-1...
This convo is hilarious, you rock. I'm not surprised someone was open minded enough to master the art of cooking with a microwave. Also there are different type of fast cooking apparatus that are usually reserved for restaurants but I could imagine they might be up your alley. (I can't right now recall the name of such a device but its similar in function to microwave maybe it is a microwave at its heart?)
As I understood it, if you used the esoteric functions of the microwave, you COULD cook food like it was cooked on a range, but it required constant babysitting of the microwave and reinput of timers and cook power levels.
And what is the point? "We know that a microwave is not a grill, but pushy advertisers insist otherwise?" The analogy is plainly wrong in the first part. We don't know.
The second part is boring. Advertisers are always pushy. It's their work. The interesting part is how much truth in their statements and it depends on the first part.
Microwaves are pretty terrible, and proof that wide consumer adoption does not really indicate quality or suggest that consumers adopt technology which _should_ exist.
I lived without a microwave for a ~year and ended up buying one again because they are pretty convenient.
So maybe it's not high on the list based on the value you are measuring but it definitely has a convenience value that isn't replaced by anything else.
The generations of microwave after the first few were fantastic. They did what they were supposed to and lasted decades. Once that reputation was solidified, manufacturers began cutting corners, leaving us with the junk of today.
The same thing happened with web search and social media. Now, those selfsame companies are trying to get us to adopt AI. Even if it somehow manages to fulfill its promise, it will only be for a time. Cost-cutting will degrade the service.
I grew up in a country where microwaves were not a thing. When they suddenly got introduced, it felt like a miracle even just for the ability to quickly heat things up.
They warm up things that a) I don't want to put in a kettle and b) don't want to put in a dedicated pot to put on the stove.
Like the remainder of the soup i made yesterday that I've put in a china bowl in the fridge. I was able to eat warm soup out of that bowl without requiring to make any other dishes dirty. Pretty convenient if you ask me.
Bonus: you can take a cherry tomato or a single grape and make a small plasma arc in the microwave. Pretty cool trick to impress and scare people at house parties.
They are not. But they are the AI slop of cooking - it's easy to get an acceptable result, so people associate it with junk food made with zero effort.
That is not my experience or look at the history at all, wrt general-ish purpose technologies.
What usually happens is they either empower unskilled (in a particular context) personnel to perform some amount of tasks at "good enough" level or replace highly specialized machinery for some amount of tasks at again "good enough" level.
At some point (typically, when a general purpose technology is able to do "good enough" in multiple contexts) operating at "good enough" enough level in multiple verticals becomes profitable over operating ant specialized level and this is when the general purpose technology starts replacing specialized technology/personnel. Very much not all general-purpose technologies reach this stage at all, this is only applicable to highly successful general purpose technologies.
Then market share of general technology starts increasing rapidly while at the same time market share of specialized technologies drops, RnD in general tech explodes while specialized technologies start stagnating. Over time this may lead to cutting edge general purpose technologies surpassing the now-old specialized technologies, taking over in most areas.
> A very clear example of this is early Tesla surfing the reduction in Li-ion battery prices. <...> By the time the technology is ready for sedans, Tesla has a 5 year lead.
> everyone at GM and Toyota was saying: Li-ion batteries are totally infeasible for the consumers we prioritize who want affordable cars.
We are nearly two decades since the Tesla "expensive sports car" and pure BEVs are still the significantly more expensive option, despite massive subsidies. If anything, everyone at Toyota were right. Furthermore, they have been developing their electric drive-trains in parallel via the hybrid tech: surfing the same wave while raking in profits.
In fact, BEV sales outpace other drive-train sales only in regions where either registrations of those are artificially limited, or the government heavily subsidizes both purchase and maintenance costs. If you don't have government subsidized rooftop solar the cost per mile of BEV is more or less on par with a HEV and in most cases worse than diesel for long range trips.
> pure BEVs are still the significantly more expensive option
New technology often has ‘new’ tradeoffs, are GPU’s are sill only situationally better than CPU’s.
DC fast charging is several times more expensive than home charging which heavily influences the economics of buying an EV without subsidies. Same deal with plug in Hybrids or long range batteries on PEV, if you don’t need the range you’re just wasting money. So there’s cases when an unsubsidized PEV is the cheapest option and that line will change over time even if it’s not going away anytime soon.
AI on the other hand is such a wide umbrella it doesn’t really make sense to talk about specific tradeoffs beyond the short term. Nobody can say what the downsides will be in 10-20 years because they aren’t extrapolating a specific technology with clear tradeoffs. Self driving cars could be taking over industries in 15 years, or still quite limited we can’t say.
GPUs are a good example - they started getting traction in the early 2000s/late 90s.
Once in the mid 2000s we figured out that single-thread perf won't scale, GPUs became the next scaling frontier and it was thought that they'd complement and supplant CPUs - with the Xbox and smartphones having integrated GPUs, and games starting to rely on general purpose compute shaders, a lot of folks (including me) thought that the software in the future will constantly pingpong between CPU and GPU execution? Got an array to sort? Let the GPU handle that. Got a JPEG to decode? GPU. Etc.
I took an in depth CUDA course back in the early 2010s, thinking that come 5 years or so, all professional signal processing will move to GPUs, and GPU algorithm knowledge will be just as widespread and expected as how to program a CPU, and I would need to Leetcode a bitonic sort to get a regular-ass job.
What happened? GPUs weren't really used, data sharing between CPU and GPU is still cumbersome and slow, dedicated accelerators like video decoders weren't replaced by general purpose GPU compute, we still have special function units for these.
There are technical challenges sure to doing these things, but very solvable ones.
GPUs are still stuck in 2 niches - video games, and AI (which incidentally got huge). Everybody still writes single-threaded Python and Js.
There was every reason to be optimistic about GPGPU back then, and there's every reason to be optimistic about AI now.
Not sure where this will go, but probably not where we expect it to.
I heard a very similar sentiment expressed as "everything is not good enough to be useful until it suddenly is".
I find it a powerful mental model precisely because it is not a statement of success rate or survival rate: Yes, a lot of ideas never break any kind of viability threshold, sure, but every idea that did also started out as laughable, toy-like, and generally shit (not just li-ion batteries, also the wheel, guns, the internet and mobile computers).
It is essentially saying 'current lack of viability is a bad indicator of future death' (at least not any more than the high mortality of new tech in general), I guess.
Even if you consider a car a general purpose technology, Tesla displacing GM is a car displacing a car, so it's not really an example of what you're saying, is it?
You took a very specific argument, abstracted it, then posited your worldview.
What do you have to say about the circular trillions of dollars going around 7 companies and building huge data centers and expecting all smaller players to just subsidize them?
Sure, you can elide the argument by saying, "actually that doesn't matter because I am really smart and understood what the author really was talking about, let me reframe it properly".
I don't really have a response to that. You're free to do what you please. To me, something feels very wrong with that and this behavior in general plagues the modern Internet.
Is GPT-5.1-Codex better or worse than GPT-5.1 (Thinking) for straight up mathematical reasoning (ie if it is optimized for making code edits)? Said another way: what is the set of tasks where you expect GPT 5.1 to be better suited than GPT-5.1 Codex? Is it non-coding problems or non-technical problems?
> 1) Prioritize your ease of being over any other consideration: parties are like babies, if you’re stressed while holding them they’ll get stressed too. Every other decision is downstream of your serenity: e.g. it's better to have mediocre pizza from a happy host than fabulous hors d'oeuvres from a frazzled one.
The questions were just a proxy for the knowledge you needed. If you could answer the questions you must have learned enough to be able to do the work. We invented a way to answer the test questions without being able to do the work.
To continue the point. If the knowledge you need is easily obtained from an LLM then knowledge isn’t really necessary for the job. Stop selecting for what the candidate knows and start selecting for something more relevant to the job.
An accurate test would just be handing them a real piece of work to complete. Which would take ages and people would absolutely hate it. The interview questions are significantly faster, but easy to cheat on in online interviews.
The better option is to just ask the questions in person to prevent cheating.
This isn’t a new problem either. There is a reason certifications and universities don’t allow cheating in tests either. Because being able to copy paste an answer doesn’t demonstrate that you learned anything.
> When your throughput increases by an order of magnitude, you're not just writing more code - you're making more decisions.
> These aren't just implementation details - they're architectural choices that ripple through the codebase.
> The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable.
Enjoyed the article and the points it brought up. I do find it uncanny that this article about the merits and challenges of AI coding was likely written by ChatGPT.
Fortunately, we can have LLMs write code and keep all the benefits of normal software (determinism, reproducibility, permanent bug fixes etc.)
I don’t think anyone is advocating for web apps to take the form of an LLM prompt with the app getting created on the fly every time someone goes to the url.
reply