Context Windows Are Getting Cheaper to Run: What Subquadratic Attention Means for Your Long-Context AI Strategy

Most engineering leaders evaluating long-context LLMs focus on the headline number: how many tokens can the model accept? That question matters, but it obscures a more consequential one: what does it actually cost to run inference at that context length, and does that cost scale in a way your infrastructure budget can absorb? The attention mechanism sitting underneath that context window determines the answer, and the architecture choices being made in research labs right now will reshape the economics of long-context deployment within the next product cycle.

Why Quadratic Scaling Is a Budget Problem, Not Just a Research Problem

Standard transformer attention compares every token against every other token in the sequence. That all-to-all comparison is what gives attention its expressiveness, but it means compute and memory costs grow with the square of sequence length. Double the context, quadruple the cost. At 128K tokens this is manageable. At 1M tokens it becomes economically indefensible for most production workloads.

The memory pressure compounds the compute problem. During autoregressive decode, the model must re-read the entire key-value (KV) cache for every new token generated. That cache grows linearly with context length, which means memory bandwidth becomes the binding constraint long before raw compute does. If you have read our analysis of how KV cache pressure interacts with hardware constraints, you will recognise this as the same memory wall that limits throughput on current inference hardware.

The practical consequence is that many teams are running long-context models at context lengths well below their stated maximum, because the cost at the upper end is prohibitive. The architecture is the ceiling, not the advertised token count.

How Hierarchical Sparse Attention Changes the Scaling Curve

The core insight behind subquadratic attention is that not every token needs to attend to every other token with equal fidelity. A well-structured indexing scheme can identify which keys are relevant to a given query without exhaustively comparing all of them. The challenge has been building an indexer that is itself cheap to run.

Zyphra's Hierarchical Attention (ZHA) addresses this by constructing a tree of recursively compressed key summaries over the KV cache (Lufkin et al., arXiv 2026). A query traverses this tree from the top down, pruning irrelevant branches at each level and attending only to the sparse subset of keys that survive the traversal. The tree depth grows iteratively, which keeps total routing cost sublinear in sequence length.

The result is attention that scales subquadratically overall: faster than quadratic, slower than linear. In practical terms, this means the cost curve bends significantly at long context lengths. The authors' performance modelling shows the approach dramatically outperforms competing methods beyond 1 million tokens, which is the range where quadratic costs make standard attention economically untenable for most organisations (Lufkin et al., arXiv 2026).

The Distillation Adoption Path and What It Means for Procurement

One of the more commercially significant findings is that ZHA does not require training a model from scratch. The authors demonstrate that it can be applied to existing pretrained models through a short KL distillation stage over a few billion tokens, with performance maintained on long-context retrieval, reasoning, and agentic coding tasks.

This matters for procurement timelines. Historically, adopting a new attention architecture meant waiting for a vendor to release a purpose-built model and then re-evaluating it from scratch against your benchmarks. Post-training distillation compresses that adoption path considerably. It means the architecture can be retrofitted onto capable base models that teams have already evaluated and trust.

It also shifts the conversation with model vendors. The question is no longer only whether a vendor supports long context, but whether their long-context implementation uses an architecture whose costs are bounded as sequence length grows. That is a question worth putting directly to vendors during evaluation.

What Engineering Leaders Should Factor Into Infrastructure Planning

Cost Modelling at Target Context Lengths

When sizing infrastructure for a long-context workload, the standard approach is to benchmark at a representative context length and extrapolate linearly. That extrapolation is wrong for quadratic architectures and overly conservative for subquadratic ones. Build your cost model against the actual scaling exponent of the attention implementation, not the headline token count.

For workloads targeting contexts above 500K tokens, the difference between quadratic and subquadratic scaling is not marginal. It determines whether the workload is viable on your current hardware fleet or requires a significant capital commitment to additional memory capacity.

Model Selection Criteria

When evaluating long-context model vendors, we recommend treating the attention architecture as a first-class selection criterion alongside benchmark performance. Specifically, ask whether the vendor's long-context implementation achieves asymptotic subquadratic scaling or only constant-factor improvements over standard attention. Sparse attention methods that do not address the quadratic scaling of their own indexer fall into the latter category and will not hold up economically at extreme context lengths.

The robustness of ZHA to varied tree geometries, including trees up to six levels deep, suggests the architecture is not brittle to hyperparameter choices (Lufkin et al., arXiv 2026). That kind of stability is a signal worth weighting in vendor evaluation, because it reduces the risk of performance cliffs appearing in production at context lengths that differ from those tested during procurement.

KV Cache and Memory Planning

Subquadratic attention reduces the compute cost of prefill, but memory planning for the KV cache remains important. The tree structure in ZHA compresses key summaries at each level, which reduces the effective cache footprint compared to storing full keys for all tokens. Factor this into your memory capacity estimates, particularly if you are planning deployments on memory-constrained accelerators.

Where the Architecture Goes From Here

The authors suggest the approach has the potential to extend effective context windows to hundreds of millions or even billions of tokens (Lufkin et al., arXiv 2026). Whether that proves out in practice depends on factors beyond the attention mechanism itself, including positional encoding behaviour at extreme lengths, training data distribution, and the ability of the model to actually use retrieved context meaningfully.

What is already clear is that the cost curve for long-context inference is not fixed. The assumption that running at 1M tokens costs roughly 60 times more than running at 128K tokens is an artefact of quadratic attention, not a physical constraint. Engineering leaders who treat context window economics as settled are making infrastructure and vendor decisions based on a cost model that is already being revised.

The more productive frame is to treat attention architecture as an infrastructure variable, evaluate vendors on their scaling behaviour rather than their maximum token count, and build cost models that account for the actual exponent of the implementation you are deploying.

Companion piece to our broader work on LLM architecture evaluation and inference economics. See The Memory Wall Is Now Your Inference Problem for how KV cache pressure and hardware constraints interact with inference cost at scale.