Subquadratic attention is an old dream. SubQ is finally worth auditing.

An old dream, newly credible

SubQ reports 98 percent single-needle retrieval at 12 million tokens while attending to roughly 0.13 percent of the possible token pairs. The reaction split into the usual camps almost immediately: breakthrough versus repackaged research. Both are early.

Subquadratic attention was not invented in 2026. Reformer, Longformer, BigBird, Linformer, Performer: by 2020 the field already had a shelf full of methods betting that most token pairs don't matter. None displaced dense attention at the frontier; their asymptotic advantages did not consistently survive the combined demands of model quality, hardware efficiency, and production maturity. What SubQ may have done is make one version of the old idea commercially credible.

It helps to be precise about what “it” is. SubQ is a model, SubQ-1.1-Small, from a startup called Subquadratic. It is built on a mechanism the company calls Subquadratic Sparse Attention (SSA): a learned, content-dependent choice of which token pairs to actually compute. And by the company's own account, SubQ-1.1-Small is not trained from scratch; it is an existing open-weight frontier model with its dense attention replaced by SSA. The technical report does not name the donor; MIT Technology Review reports it was Qwen. That single fact is why “breakthrough” and “repackaged” are both premature. The interesting question isn't the lineage. It's whether the sparse selection holds up where it matters.

The selector is the whole game

SSA avoids computing every query-key interaction by first deciding which small subset of tokens each query should see. Attention over the selected subset may be exact, but the model is only as reliable as the causal paths its selector preserves. A relevant token need not be directly selected by every query; the failure occurs when its information never enters any path that materially influences the answer.

This is the load-bearing move, and it relocates the risk. Dense attention is expensive precisely because it refuses to decide in advance what matters: every token can see every other. A sparse model spends its budget on a selector and trusts it.

The speed result, and its ceiling

The speed claims have third-party measurement behind them. Appen, an evaluation firm commissioned by Subquadratic, benchmarked the SSA kernel against FlashAttention-2 on an NVIDIA B200: 381 milliseconds versus 21.4 seconds at one million tokens, a 56.2× wall-clock difference, scaling close to linear as context grows.

That is a serious systems result. It is also a result about the attention operation, not proof that the whole model runs 56× faster. Kernel latency and end-to-end latency are different quantities, and the gap between them is exactly the kind of thing a benchmark can quietly absorb.

Where the quality signal is

The quality results are more revealing than the speed ones. On a 128K RULER evaluation, SubQ scored perfectly on single-needle retrieval. On the multi-key tasks, accuracy fell from 96 percent with two keys, to 83 with four, to 68 with eight.

Those numbers identify a failure regime, not its cause. As the model must recover more targets from a crowded context, reliability falls sharply, and the public results do not say whether the misses came from routing, representation, reasoning, or output formatting. That ambiguity is exactly why selector-level diagnostics matter. The regime is also the one real work lives in: repositories rarely contain one cleanly isolated dependency. They contain aliases, repeated patterns, similar interfaces, stale implementations, and several files that all look locally relevant. For retrieving several items at once, the closer measure is Appen's eight-needle MRCR run at roughly one million tokens, where SubQ scores 86.2 percent.

How verified is this?

Before any of these numbers do argumentative work, it is worth being exact about where they come from. I checked each against primary sources, and they hold up as reported. What they do not yet have is open, unaffiliated reproduction.

There is also a provenance wrinkle worth resolving. SubQ's technical report claims 99.12 percent on the full 13-task RULER suite at 128K, while Appen's published breakdown at the same length averages 95.6 percent and shows the multi-key degradation above. A newer checkpoint, a changed task configuration, or different scoring could each explain the gap; the public materials do not make the mapping explicit. Before treating either number as definitive, the model version and harness need pinning down.

None of that makes the numbers wrong. It makes them claims awaiting the kind of adversarial, unaffiliated pressure that turns a demo into a result. That distinction, reported versus reproduced, is the whole reason to audit rather than react.

A repo that compiles and is wrong

Code is where I'd apply the pressure. Not because “every token matters.” Code actually has enormous exploitable sparsity: a function depends on a limited set of definitions; a module imports a fraction of the repository. The problem is the penalty for choosing the wrong sparse graph.

Picture a routine change: rename a field on a request object and update its callers. The selector pulls the type definition, the obvious call sites, the handler. It does not pull a validator six directories away that special-cases the old field name as a string, or an interface contract expressed only in a test fixture asserting the old shape. The model produces a clean, plausible patch. It compiles. It passes the tests the selector happened to surface. And it is wrong in exactly the way that reaches production, because the one file that encoded the real constraint was present in the repository and absent from the computation.

Many natural-language tasks tolerate partial correctness. Many software tasks have discontinuous acceptance criteria: a forgotten import, a config flag six directories away, or an interface contract that lives only in a test flips a confident answer from correct to broken.

SubQ's reported 81.8 percent on SWE-bench Verified, with extended thinking enabled, says the architecture can support serious coding. It is below Anthropic's separately reported 87.6 percent for Opus 4.7, though differences in harness, prompting, and inference budget prevent a clean model-only comparison, and the company itself concedes the margin is “as much harness as model.” A strong aggregate score is entirely compatible with a selector that occasionally drops the single dependency that decides the task.

Average visibility is not coverage

This is familiar territory from my day job. In silicon design-for-debug, we build scan chains and JTAG access because a chip's internal state is too large to inspect directly. Every debug engineer learns the same lesson early: average visibility is not coverage. You can observe 99.9 percent of a design and still miss the one state element corrupting the machine.

The selector is the visibility fabric through which a sparse model sees its context. If it routes the right dependencies, 0.13 percent of token pairs may be plenty. If it does not, a relevant token can be present in the prompt yet absent from every causal path that materially influences the answer. Availability is not use.

In silicon, an escape is the defect that passes through your test strategy and reaches the customer. Here it's the dependency that existed in the prompt but never reached the computation that needed it. A mean benchmark score hides that failure mode remarkably well, the same way a healthy average-coverage number can sit on top of a systematic escape.

Four questions to ask first

So before anyone writes “attention bottleneck solved,” four questions I'd want answered:

1. 01
   Selector recall

   For known-relevant information, can the system show whether it influenced the generated answer, directly or indirectly? Not average recall. The tail.

2. 02
   Full-model accounting

   Appen describes its kernel numbers as end-to-end wall clock for the SSA path. What are time to first token, peak memory, decode throughput, and total inference cost once MLPs, normalization, communication, and generation are included?

3. 03
   Failure attribution

   After a wrong answer, can we distinguish a routing failure (the token was never selected) from a reasoning failure over tokens that were?

4. 04
   Escape rate

   What is the failure rate on tasks where a single omitted dependency invalidates the whole result?

How to close the claim

The way to settle this is not another 12-million-token retrieval demo. It's selector diagnostics and adversarial tasks built to punish a dropped dependency.

Instrument the selector directly: for a task with known-relevant tokens, log which ones each query actually attended to, and measure selector recall against that ground truth. Build repository tasks where the deciding constraint lives off the obvious path, in a test, a config, or a distant validator, and score the escape rate, not the mean. Attribution needs more care than re-prompting: force-feeding the missing dependency into context changes salience, and can rescue a reasoning failure as easily as a routing one. The cleaner diagnostic holds the prompt, decoding parameters, and seed fixed and intervenes on routing itself, forcing the known-relevant spans into the selected set. If the answer changes reliably under that intervention and not under a control, the case for a routing failure is strong. These are feasible experiments. Their absence, not their results, is what should make a reader cautious.

Sparse attention needs the equivalent of a trace buffer: enough instrumentation to reconstruct not merely what the model answered, but what evidence was available to each stage of the computation and which paths actually influenced the answer.

My read: SubQ didn't invent subquadratic attention, and dismissing it on that basis would be lazy. The kernel is genuinely fast, the retrieval is strong, and 81.8 percent on SWE-bench Verified shows the architecture supports real coding. But the strongest claim is still open, and the evidence so far is vendor-originated and commissioned rather than openly reproduced. The way to close it is selector diagnostics and adversarial repository tasks.

The one it should not have missed

Sparse attention will not be judged by the millions of relationships it avoided computing. It will be judged by the one relationship it should not have missed.

The honest test is workload-shaped: there are tasks where I'd happily trust a model that reads 0.13 percent of token pairs, and tasks where I never would. Knowing which is which, before the selector decides for you, is the whole job.