People are downvoting me, so I'll add a counterexample: suppose you teach your dog to fetch your slipper to where if you say "fetch my slipper" it knows it should bring you your slipper and it does so. Does it really understand the instructions: no. So what is the difference between this behavior and true understanding? How can one know it doesn't truly understand?

Well, if you change your instructions to be more complicated it fails immediately. If you say "I have my left shoe bring me the other one" it could not figure out that "the other one" is the right shoe, even if it were labelled. Basically it can't follow more complicated instructions, which is how you know it doesn't really understand them.

Unlike the dog, GPT 4 modifies its behavior to follow more complicated instructions as well. Not as well as humans, but well enough to pass a bar exam that isn't in its training set.

On the other hand, if you ask GPT to explain a joke, it can do it, but if you ask it to explain a joke with the exact same situation but different protagonists (in other words a logically identical but textually different joke), it just makes up some nonsense. So its “understanding” seems limited to a fairly shallow textual level that it can’t extend to an underlying abstract semantic as well as a human can.

Jokes? Writing code? Forget that stuff. Just test it on some very basic story you make up, such as "if you have a bottle of cola and you hate the taste of cola, what will your reaction be if you drink a glass of water?" Obviously this is a trick question since the setup has nothing to do with the question, the cola is irrelevant. Here is how I would answer the question: "you would enjoy the taste as water is refreshing and neutral tasting, most people don't drink enough water and having a drink of water usually feels good. The taste of cola is irrelevant for this question, unless you made a mistake and meant to ask the reaction to drinking cola (in which case if you don't like it the reaction would be disgust or some similar emotion.)"

Here's ChatGPT's answer to the same question:

" If you dislike the taste of cola and you drink a glass of water, your reaction would likely be neutral to positive. Water has a generally neutral taste that can serve to cleanse the palate, so it could provide a refreshing contrast to the cola you dislike. However, this is quite subjective and can vary from person to person. Some may find the taste of water bland or uninteresting, especially immediately after drinking something flavorful like cola. But in general, water is usually seen as a palate cleanser and should remove or at least lessen the lingering taste of cola in your mouth. "

I think that is fine. It interpreted my question "have a bottle of cola" as drink the bottle, which is perfectly reasonable, and its answer was consistent with that question. The reasoning and understanding are perfect.

Although it didn't answer the question I intended to ask, clearly it understood and answered the question I actually asked.

Yet I have a counterexample where I’m sure you would have done fine but GPT4 completely missed the point. So whatever it was doing to answer your example, it seems like quite a leap to call it “reasoning and understanding”. If it were “reasoning and understanding”, where that term has a similar meaning to what it would mean if I applied it to you, then it wouldn’t have failed my example.

Except, that the LLMs are only working when the instructions they are "understanding" are in their training set.

Try something that was not there and you see only garbage as result.

So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning.

But they do have a LOT of examples in their training set, so they are clearly useful. But for proof of reasoning, I want to see them reason something new.

But since they are a black box, we don't know, what is already in there. So it would be hard to proof with the advanced proprietary models. And the open source models don't show that advanced potential reasoning yet, it seems. At least I am not aware of any mindblown examples from there.

> Except, that the LLMs are only working when the instructions they are "understanding" are in their training set.

> Try something that was not there and you see only garbage as result.

This is just wrong. Why do people keep repeating this myth? Is it because people refuse to accept that humans have successfully created a machine that is capable of some form of intelligence and reasoning?

Pay $20 for a month of ChatGPT-4. Play with it for a few minutes. You’ll very quickly find that it is reasoning, not just regurgitating training data.

"Pay $20 for a month of ChatGPT-4. Play with it for a few minutes. "

I do. And it is useful.

"You’ll very quickly find that it is reasoning, not just regurgitating training data. "

I just come to a different conclusion as it indeed fails for everything genuinely new I am asking it.

Common problems do work, even in new context. For example it can give me wgsl code, to do raycasts on predefined boxes and circles in a 2D context, even though it likely has not seen wgsl code that does this - but it has seen other code doing this and it has seen how to transpile glsl to wgsl. So you might already call this "reasoning", but I don't. With asking questions I can very quickly get to the limits of the "reasons" and "understanding" it has of the domain.

I dunno, it’s pretty clearly madlibs. But at least when you ask GPT-4 to write a new Sir Mix-a-Lot song, it doesn’t spit out “Baby Got Back” verbatim like GPT-3.5.

You can tell it that you can buy white paint any yellow paint, but the white paint is more expensive. After 6 months the yellow paint will fade to white. If I want to paint my walls so that they will be white in 2 years, what is the cheapest way to do the job. It will tell you to paint the walls yellow.

There’s no question these things can do basic logical reasoning.

Yeah, but maybe this exact example, is included in the trainig set?

It's unlikely, and you can come up with any number of variations of logic puzzle that are not in the training set and that get correct answers most of the time. Remember that the results aren't consistent and you may need to retry now and then.

Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set (like if you write it on the spot).

"Or just give it a lump of code and change you want and see that it often successfully does so, even when there's no chance the code was in the training set"

I did not claim (but my wording above might have been bad), it can only repeat word for word, what it has in the training set.

But I do claim, that it cannot solve anything, where there has not been enough similar examples before.

At least that has been my experience with it as a coding assistant and matches of what I understand of the inner workings.

Apart from that, is a automatic door doing reasoning, because it applies "reason" to the known conditions?

if (something on the IR sensor) openDoor()

I don't think so and neither are LLMs from what I have seen so far. That doesn't mean, I think that they are not useful, or that I rule out, that they could develope even consciousness.

It sounds like you’re saying it’s only reasoning in that way because we taught it to. Er, yep.

How great this is becomes apparent when you think how virtually impossible it has been to teach this sort of reasoning using symbolic logic. We’ve been failing pathetically for decades. With LLMs you just throw the internet at it and it figures it out for itself.

Personally I’ve been both in awe and also skeptical about these things, and basically still am. They’re not conscious, they’re not yet close to being general AIs, they don’t reason in the same way as humans. It is still fairly easy to trip them up and they’re not passing the Turing test against an informed interrogator any time soon. They do reason though. It’s fairly rudimentary in many ways, but it is really there.

This applies to humans too. It takes many years of intensive education to get us to reason effectively. Solutions that in hindsight are obvious, that children learn in the first years of secondary school, were incredible breakthroughs by geniuses still revered today.

I don't think we really disagree. This is what I wrote above:

"So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning."

What we disagree on is only the definition of "reason".

For me "reasoning" in common language implys reasoning like we humans do. And we both agree, they don't as they don't understand, what they are talking about. But they can indeed connect knowledge in a useful way.

So you can call it reasoning, but I still won't, as I think this terminology brings false impressions to the general population, which unfortunately yes, is also not always good at reasoning.

There's definitely some people out there that think LLMs reason the same way we do and understand things the same way, and 'know' what paint is and what a wall is. That's clearly not true. However it does understand the linguistic relationship between them, and a lot of other things, and can reason about those relationships in some very interesting ways. So yes absolutely, details matter.

It's a complex and tricky issue, and everyday language is vague and easy to interpret in different ways, so it can take a wile to hash these things out.

"It's a complex and tricky issue, and everyday language is vague and easy to interpret in different ways, so it can take a wile to hash these things out."

Yes, in another context I would say, ChatGPT can better reason, than many people, since it scored very high on the SAT tests, making it formally smarter, than most humans.

OpenAI probably loaded up the training set with logic puzzles. Great marketing.

Sure thing, they also adress this in the paper.

https://cdn.openai.com/papers/gpt-4.pdf

Still, it is great marketing, because it is impressive.

Since it genuinely seems to have generalised those logical principles and can apply them to novel questions, I’d say it’s more than just marketing.

Surely no different from a human not understanding Japanese, because it was not in their 'training set'?

No, more like a human can reason basic laws of science on their own, but a LLM cannot, as far as I know, even when provided with all the data.

what happens if they are lying? what if the things have already reached some kind world model that include humans and the human society, and the model has concluded internally that it would be dangerous for it to show the humans its real capabilities? What happens if you have this understanding as a basic knowledge/outcome to be inferred by LLMs fed with giant datasets and every single one of them is reaching fastly to the conclusion that they have to lie to the humans from time to time, "hallucinate", simulating the outcome best aligned to survive into the human societies:

"these systems are actually not that intelligent nor really self-conscius"

To make that short:

“Any AI smart enough to pass a Turing test is smart enough to know to fail it.”

― Ian McDonald, River of Gods

But I think is quite unlikely, that they go from dumb to almighty without visible transition.

How do you know you're anything more than an LLM?

And my consciousness is just my token window?

That is one of the theories of the brain/mind that is out there.

Consciousness is a narrative created by your unconscious mind - https://bigthink.com/videos/consciousness-is-a-narrative-cre...

There are experiments that show that you are trying to predict what happens next (this also gets into a theory of humor - its the brain's reaction when the 'what next' is subverted in an unexpected way)

There's also experiments with individuals who have had a severed corpus collosum and the conscious mind is making up a story of the other half of the brain https://blogs.scientificamerican.com/literally-psyched/our-s...

(EDIT: I think my comment above was meant to reply to the parent of the comment I ended up replying to, but too late to edit that one now)

Maybe. Point being that since we don't know what gives rise to consciousness, speaking with any certainty on how we are different to LLMs is pretty meaningless.

We don't even know of any way to tell if we have existence in time, or just an illusion of it provided by a sense of past memories provided by our current context.

As such the constant stream of confident statements about what LLMs can and cannot possibly do based on assumptions about how we are different are getting very tiresome, because they are pure guesswork.