Quadratic attention costs are becoming a practical constraint, not just a research concern

The architectural choice at the center of most modern AI systems, the attention mechanism that made transformers possible, carries a cost that compounds badly. Every time a model must attend over a longer sequence, the compute required scales not linearly but quadratically with context length. For years that was a manageable constraint. The evidence now suggests it may not remain one.

Ramin Hasani, whose work at Liquid AI sits at the frontier of alternative neural architectures, puts the case plainly: for extremely long context windows, linear attention is the only viable approach, because the quadratic cost of standard attention becomes prohibitive as context scales. That is not a claim about research preference. It is a claim about what is computationally possible when context length grows beyond a certain threshold.

The threshold matters because of where context demands appear to be heading. Dan Biderman projects that many AI-native companies could accumulate trillions of tokens of internal proprietary data within roughly 18 months. His framing is deliberate: he acknowledges the figure sounds exaggerated, but argues it is not an impossibility for companies that are genuinely AI-native. Knowledge workspaces of that scale, he contends, are not being fully reckoned with. If Biderman’s projection is even directionally correct, the quadratic cost problem shifts from an edge case into a practical constraint that organizations will encounter in the ordinary course of operating their systems.

Linear attention for extremely long context, you cannot do it any other way. The reason behind it is because context has become so large that that quadratic cost of attention just kicks in. Ramin Hasani

The research community has been circling this problem from several directions. Linear attention in theory can handle sequences of arbitrary length with fixed memory consumption, but practical implementations have faced their own limitations, and work continues on resolving them. The tradeoff space between full quadratic transformers and linear approximations is neither empty nor easily resolved, and the performance gap relative to full attention in certain tasks remains a live research question.

Hasani’s own research trajectory is relevant context here. Liquid AI’s foundational contribution, solving the liquid neural network neuron-interaction equations in closed form for the first time, appeared in Nature Machine Intelligence in November 2022. That result was not primarily about attention; it was about a different class of dynamical systems for modeling neural computation. But it is part of a broader program of building architectures that behave differently from conventional transformers, and the linear attention position Hasani now stakes out fits that orientation.

What makes this signal worth tracking is the combination of the architectural argument and the data-volume projection arriving together. Either, in isolation, would be interesting. Hasani making the case for linear attention is a research claim from a well-positioned team. Biderman projecting trillion-token proprietary corpora is a product and infrastructure claim from someone thinking about enterprise AI at scale. Together, they point toward a near-term environment in which the architectural defaults baked into most current systems may become genuinely untenable for the workloads those systems are expected to handle.

The honest caveat is that this is early signal, not settled verdict. The engineering challenges of linear attention are real, and the research questions are open. But the direction of pressure is clear enough: context demands are rising, quadratic costs scale badly, and the teams that wait for this to become an obvious crisis before rethinking their architectural assumptions will be solving the problem under worse conditions than the teams that begin now.