GPT-3 is trained by looking at approximately all text in the world once. So the wall is when you run out of text.
And I don't think it's really gotten better recently. ChatGPT is not new, you just didn't know about it before. (Also, they're paying for it. You wouldn't enjoy using it nearly as much at full price.)
Humans become intelligent after reading / hearing several orders of magnitude fewer words than GPT-3 has access to.
There's dozens of ways GPT-4 could be improved over GPT-3 without needing more training data. Eg, through reinforcement learning (talking to itself), tweaking the neural architecture, spending more time training, etc.
No because multi-modal learning is next. A human can learn what a rock is after N=1 only because we have the visual/tactile of the rock (combined with our intuitive understanding physics etc) to go with the text label. We can immediately guess how it fits into our world from that visual/tactile. GPT needs very large N to grasp the concept of a rock because it requires a lot of textual associations to figure that concept out.
It is next conceptually, but it’s not proven you can do it till someone’s done it. Other rumors suggest whatever GPT4 is, it’s not multimodal.
I don’t believe large models are great multimodal demonstrations either, insofar as being large just lets you memorize different modalities side by side without necessarily integrating them.
I don't think that training a GPT on GPT output is likely to be helpful. If nothing else, it's going to be like a human getting stuck in a filter bubble - it's not going to improve correspondence with reality.
I’m curious whether there is a future coming where we praise researchers for any excellent, new, and novel research that can be added to the knowledge dataset.
ChatGPT is just InstructGPT with prompt engineering to make the model behave like it's having a conversation. There was probably some additional fine-tuning done to make it self aware (think of the "I am a chat bot trained by OpenAI" spiel), but personally it has comparable performance to text-davinci-003 which was released Nov of last year (Nov 2022).
And I don't think it's really gotten better recently. ChatGPT is not new, you just didn't know about it before. (Also, they're paying for it. You wouldn't enjoy using it nearly as much at full price.)