My most exciting prediction is that we'll be able to some day have Optical Computing LLMs, where we grow crystals (instead of silicon wafers) such that they contain wave-guides in which a single ray of light is what makes up each "input" into a Neural Net, and as it shines thru, it does the "logic" of a MLP. The summation part is not hard because light naturally superposes to "add", but the multiplications and activation functions will be the hardest part.
But ideally once manufactured, a given LLM "model" will be a single solid crystal, such that shining an array of beams into it, will come out the other end of this complex crystal as an "inference" result. This will mean an LLM that consumes ZERO ENERGY, and made of glass will also basically last forever too.
We already have Optical Chips but they don't quite do what I'm saying. What I'm saying is essentially an "Analog LLM" where all the vector adds, mults, and tanh functions are done by the light interactions. It seems possible, but I think it's doable.I think there should theoretically be a "lens shape" that does an activation function, for example. Even if we have to do the multiplications by conventional chips, in a hybrid "silicon-wave system" such an "Analog Optical LLM" would still have huge performance and energy savings, and millions of times faster than today's tech.
And being based on light, could utilize quantum effects so that the whole thing can become a Quantum Computer as well. We could use effects like polarization and photon spin perhaps to even have 100s of inferences happening simultaneously thru a given apparatus, as long as wavelengths are different enough to not interact.
But ideally once manufactured, a given LLM "model" will be a single solid crystal, such that shining an array of beams into it, will come out the other end of this complex crystal as an "inference" result. This will mean an LLM that consumes ZERO ENERGY, and made of glass will also basically last forever too.
We already have Optical Chips but they don't quite do what I'm saying. What I'm saying is essentially an "Analog LLM" where all the vector adds, mults, and tanh functions are done by the light interactions. It seems possible, but I think it's doable.I think there should theoretically be a "lens shape" that does an activation function, for example. Even if we have to do the multiplications by conventional chips, in a hybrid "silicon-wave system" such an "Analog Optical LLM" would still have huge performance and energy savings, and millions of times faster than today's tech.
And being based on light, could utilize quantum effects so that the whole thing can become a Quantum Computer as well. We could use effects like polarization and photon spin perhaps to even have 100s of inferences happening simultaneously thru a given apparatus, as long as wavelengths are different enough to not interact.