I'm not sure I follow. Knowledge based retrieval of what exactly? Outdated docs and dilapidated code? Aging internal wikis and chat histories erased for legal reasons?
Everyone also seems to overlook how much time and resources it takes for these models to be trained/fine tuned on a corpus of knowledge. Researches have calculated it probably took OpenAI the equivalent of 355 years on a single NVidia V100 to train up GPT 3. [1] Clearly they used more horsepower in parallel, which is a foreseen problem right now for other reasons. [2]
Everyone also seems to overlook how much time and resources it takes for these models to be trained/fine tuned on a corpus of knowledge. Researches have calculated it probably took OpenAI the equivalent of 355 years on a single NVidia V100 to train up GPT 3. [1] Clearly they used more horsepower in parallel, which is a foreseen problem right now for other reasons. [2]
[1] https://lambdalabs.com/blog/demystifying-gpt-3
[2] https://www.pcworld.com/article/2020375/the-ai-boom-could-cr...