> When you abandon O(N^2) attention, you are forced to start adding heuristics to choose what to correlate. Any time you see one of those giant context window LLMs, you need to be asking what heuristics they added, what is getting correlated, and what is not getting correlated.

Well, having a small context window and everything correlated with everything else is equivalent to having a large context window, but a particularly dumb heuristic.