But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.
And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).
I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.
If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Do amnesiacs who are incapable of laying down long-term memories not think?
I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)
I mean, _we_ probably can't think with our wetware on a read-only substrate. It doesn't establish it as essential, just that the only sure example in nature of thought doesn't work that way.
I'm not an expert but as far I understand, plasticity is central to most complex operations of the brain and is likely to be involved in anything more complex than instinctive reactions. I'm happy to be corrected but it is my understanding that if you're thinking for a while on the same problem and establishing chains of reasoning, you are creating new connections and to me that means it's fundamental in the process of thinking.
Also not an expert :) I thought plasticity is an O(hours-days) learning mechanism. But I did some research and there is also Short Term Plasticity O(second) [1] which is a crucial part of working memory. We'd need that. But it seems it’s more of a volatile memory system, eg calcium ion depletion/saturation at the synapse, rather than a permanent wiring/potentiation change (please someone correct me if this isn’t right :) ).
So I guess I’d just clarify “read only” to be a little more specific - I think you could run multiple experiments where you vary the line of what’s modeled in volatile memory at runtime, and what’s immutable. I buy that you need to model STP for thought, but also suspect at this timescale you can keep everything slower immutable and keep the second-scale processes like thought working.
My original point still stands - your subjective experience in this scenario would be thought without long-term memory.
It would be funny if what you get from a read only human brain is a sort of memento guy who has no capacity to remember anything or follow a conversation... kind of like an LLM!
> If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?
Perhaps this is already known, but I would think there is a high chance that our brains require "write access" to function. That is, the very process of neural activity inherently makes modifications to the underlying structure.
> a high chance that our brains require "write access" to function
There are multiple learning mechanisms that happen on different time-frames, eg neural plasticity, hippocampus are both longer-term processes for memory consolidation. Whereas the content of “RAM” might be better modeled as a set of fast dynamic weights representing ions and neurotransmitter concentrations.
My hunch is that you could model the latter fast weights in volatile memory, but I wouldn’t count these as “modifying the structure”.
Do you have any particular systems in mind where you have reason to believe that permanent modification is required for brains to function?
This is just wrong though. They absolutely learn in-context in a single conversation within context limits. And they absolutely can explain their thinking; companies just block them from doing it.
But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.
And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).
I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.