Long-Context Memory For Constant-State Models
The conversation-needle benchmark, the transfer-gap finding, bitwise-exact streaming state carry, and the open programme for training past-window recall.
The conversation-needle benchmark, the transfer-gap finding, bitwise-exact streaming state carry, and the open programme for training past-window recall.
Byte-level training efficiency on one consumer machine: Muon, boundary patching, constant-state recurrence, and the hybrid trunk that beat both of its parents.
How an affine, fixed-cost-per-update model of training time turned a budgeted four-to-five-day run into a measured two-day run.
The three-phase recipe behind chat80m, the first conversing byte-level model in the project, its knowledge wall, and the identity-dosing discovery.
Falsified approaches at 10M+ params: fast-weight mem, test-time depth, byte-level selective loss, delta-rule instability, mem registers, and the MoE tax.
The v13 engine: Constant-size decode state, byte-for-byte parity to the reference and about one millisecond per byte on a CPU core.
Fine-tuning teaches behavior, RAG supplies knowledge, prompting shapes both. What the published comparisons found, and how to pick without burning a GPU budget.
The conversation-needle benchmark, the transfer-gap finding, bitwise-exact streaming state carry, and the open programme for training past-window recall.
The hybrid architecture from The composed efficiency stack carries its long-range memory in a fixed-size state that does not grow as a conversation lengthens. That is a real structural edge, and it is worth stating in its strongest honest form: a small, fast model could in principle hold a very long conversation at bounded cost, which a standard transformer, whose memory grows with every byte, cannot. The advantage is not a matter of a better cache or a cleverer eviction policy; it is a property of the shape of the model. The catch is equally structural. A fixed-size memory is a lossy summary, so it forgets, and the question is not whether it forgets but whether we can make it forget gracefully: gently, and preferentially, keeping what matters and shedding what does not. We built a benchmark that plants a fact in a synthetic conversation and asks for it back later, at controlled distances, and scores how recall decays with distance. We found two things, and neither is the thing we hoped to find. First, at small scale the model can be taught to retrieve facts that look like its training data, but it does not transfer that skill to facts written in unfamiliar surface forms. Second, recall past the local attention window is not trained at all, because the ordinary training loop resets the recurrent state at every window boundary, so the long-range pathway is never given a gradient. We also built and verified the streaming machinery that makes past-window recall measurable in the first place, and, in principle, trainable. We are candid about where this leaves us: no mechanism has yet flattened the forgetting curve. The honest deliverables so far are the benchmark, the finding, and the infrastructure. Everything downstream of those, the mechanism that actually makes a small model retrieve across a long conversation, is still open, and this paper does not claim to have solved it.
Terms are defined at first use. "Constant-state" (also "O(1) state") means the model's memory is a fixed size no matter how long the input; doubling the conversation does not double the memory. "Recall" here is exact-match accuracy on a planted fact: the model either reproduces the needle verbatim or it does not, and partial credit is not given. "In distribution" means test items that look like the training data; "out of distribution" means items whose surface form (the literal characters, the way the fact is spelled and punctuated) the model never saw in training. "The window" throughout means the span of the local attention component, the short-range mechanism that can look back a fixed number of recent bytes directly; anything the model needs from before that span must have been carried in the recurrent state.
The claim we can make honestly, and defend, is narrow and structural: memory cost does not grow with conversation length. The recurrent global mixer keeps one fixed matrix of state per head, updated once per byte, so a small model can carry a huge conversation at bounded cost. To see why this is a real edge rather than a bookkeeping curiosity, contrast the two ways a sequence model can remember its past. A transformer remembers by keeping every past token available for direct attention: its key/value cache grows linearly with the number of bytes seen, so a conversation twice as long costs twice the memory and, per new byte, more compute to attend over. That is a fine trade when context is short and hardware is generous. It is a poor trade when the goal is a long, coherent conversation held on a small, fast model on modest hardware, because the resource bill climbs without bound exactly as the conversation gets interesting. The constant-state mixer makes the opposite trade. It compresses the entire past into a fixed-size matrix and pays the same per-byte cost at byte one million as at byte one hundred. On the deployment axis this is decisive: bounded memory over unbounded conversation is precisely the property a lightweight always-on assistant needs, and it is the property a growing KV cache cannot offer at any cache-management cleverness, because the growth is intrinsic to how a transformer remembers.
What we do not claim is frontier quality. The single-machine compute wall from The composed efficiency stack still stands, and nothing here moves it. This is a capability target, long and coherent context on a small fast model, not a raw-quality target. The distinction matters because it tells us which failures are fatal and which are merely expensive: a mechanism that made recall cheap but slightly degraded raw quality could still be a win against the deployment brief, whereas a mechanism that bought quality at unbounded memory cost would defeat the entire point of a constant-state design.
A fixed-size state is a lossy summary, so it forgets, and this is not a defect to be patched but a consequence of the compression. If a bounded matrix must stand in for an unbounded past, information must be discarded, and the only real question is which information. In the current mixer the answer is unflattering: the forgetting is uniform. There is one scalar decay per head, applied to the whole state, so every stored quantity fades at the same rate regardless of its importance. A verbatim access code the user will need in twenty turns and an idle pleasantry from the same turn decay identically, because the decay does not know which is which. That is the mechanism-level reason a single-scalar decay cannot, by itself, produce graceful forgetting: gracefulness requires the model to fade the trivial faster than the load-bearing, and a scalar that multiplies everything has no way to tell them apart. When we say we want a memory that "does not degrade," we mean something precise and measurable: flatten the curve of recall against distance, so that a fact planted far back is retrieved nearly as often as one planted nearby. That is the target the rest of this paper is measured against.
Any claim about long-context memory has to be falsifiable and cheap to test, or it becomes a matter of anecdote. We wanted a harness that (a) runs without a GPU, so that mechanisms could be screened on the same hardware that serves the models; (b) controls the one variable that matters, the distance between where a fact is planted and where it is needed; and (c) separates the two things a memory can fail at, retrieving a fact and staying internally consistent about it. The benchmark is built around those goals.
We built a GPU-free harness (experiments/v2/longctx/needle_bench.py). It generates synthetic multi-turn conversations of a controlled byte length, plants a fact (a "needle") near the start, and at the end asks a question whose answer requires that needle. The needle comes in three kinds so that the result is not an artifact of one fact shape: an access code, a name, and a constraint. The conversation filler between plant and query is real dialogue-shaped text, so the needle has to survive genuine intervening content rather than blank padding, and the byte length is set explicitly so that plant-to-query distance is a dial we turn rather than a number we read off afterward.
The harness scores two distinct things. The first is recall as a function of plant-to-query byte distance, which is the degradation curve: the fraction of trials at each distance on which the model reproduces the needle exactly. The second is coherence, measured by self-consistency probes: having stated a fact earlier in a conversation, does the model later contradict it. A model can fail either test independently, and the two failures call for different fixes, so we keep them separate rather than folding both into one score.
A mechanism passes only if it measurably flattens the recall curve relative to the plain baseline. Raising recall at one distance while leaving the far distances at zero is not a pass, because the deployment claim is about the far distances; a memory that helps only within the window is a memory the local attention already provides. The harness scores any checkpoint, which lets us follow a strict order of operations: every mechanism is validated at 10M parameters before it is tried at 121M. The point of the 10M screen is economy. A mechanism that cannot show any signal at 10M is not worth the cost of a 121M run, and a mechanism that shows signal at 10M has earned the larger test. This ordering is what made the transfer-gap finding below both cheap to reach and, once reached, consequential.
The first degradation curve was measured on the chat80m SFT checkpoint, using the baseline uniform-decay state, with streaming carry enabled, 12 trials per distance, the 3 needle kinds, and greedy decode. The curve, stated in full and with every value preserved, is as follows. At a plant-to-query distance of about 190 bytes, recall is 0.92. At about 480 bytes, recall has already fallen to 0.25. At 2k, 8k, and 32k bytes, recall is 0.00 at every one of those distances. The coherence probe, run on the same checkpoint, gave a contradiction rate of 0.25.
This single curve contains two qualitatively different failures, and reading them apart is what organised the rest of the campaign into two hypotheses. Inside the window, recall does not hold: it collapses from 0.92 to 0.25 as the needle moves from roughly 190 to roughly 480 bytes back, even though local attention can in principle still see the fact at those distances. Past the window, recall is not merely low but exactly zero, at 2k and 8k and 32k alike, with no gradient of decay between them, which is the signature of a pathway that carries nothing rather than a pathway that carries a little and fades.
We named the two hypotheses so that later work could be filed under one or the other. H1, the mechanism hypothesis: past-window recall is 0 because the uniform-decay state carries nothing usable across windows, so the failure is in what the state can represent and preserve. H2, the training-pressure hypothesis: recall collapses from 92 to 25 percent inside the window, where local attention can still reach the fact, which means ordinary chat data never demands long-range exact recall, so the skill is simply never trained. The flat zero past the window is H1's territory; the in-window collapse is H2's. The distinction is not academic, because the two hypotheses imply different fixes, and the open programme below is organised precisely around them.
The transfer-gap finding is the result of this paper we are most confident in, partly because it began as a wrong reading that the evidence then corrected. At 10M, all three arms we ran, the baseline, a pinned register, and a recall-pressure-trained arm, scored 0.00 on the needle benchmark at every distance. The obvious first reading was that retrieval is simply absent below some scale: that a 10M model is too small to retrieve at all, and the mechanism question is moot until the model is larger. That reading was wrong, and it is worth dwelling on how we know, because the correction is the substance of the finding.
We ran an in-distribution diagnostic that removed the benchmark's hardest confound. Instead of the alien needle surface forms the benchmark deliberately uses, we scored greedy answer completion on held-out conversations drawn from the same recall corpus the recall arm was trained on, with a last-byte exact match over 40 trials each. On that diagnostic the recall-trained arm scored 19/40 = 0.47, against the baseline at 2/40 = 0.05. That is a tenfold lift, and it is not compatible with "retrieval is absent at this scale." The corrected finding is therefore precise: recall-pressure data teaches genuine retrieval even at 10M. What a 10M model lacks is not the capacity to retrieve but the capacity to transfer, to recognise a fact it has never seen spelled this way as the same kind of thing it learned to fetch.
The reason the benchmark exposes this gap by design is a contamination guard, and it is a feature, not a leak. The benchmark's needle surface forms are made deliberately alien to the everyday-facts training pool: hyphenated codes and five-digit dollar amounts, shapes that do not occur in ordinary chat data. The guard exists so that a model cannot pass the benchmark by having memorised the needle during training; it must actually retrieve it from the conversation. The side effect is that passing the benchmark requires surface-form generalisation, and a 10M model cannot bridge that gap. The clean statement of the whole effect is a pair of numbers: in-distribution recall 0.47, alien-surface-form recall 0.00. Together they falsify "sub-threshold capability" and replace it with the correct model of the 10M behaviour: a narrow skill with no surface-form generalisation.
Why the correction matters methodologically is worth making explicit, because it is a lesson the campaign will keep paying forward. A single benchmark number, 0.00 across the board, invited a scaling conclusion ("too small, come back bigger") that would have been expensive and misdirected: it would have moved the whole programme to 121M in search of a capability that was in fact already present at 10M and merely masked by a deliberate confound. The in-distribution diagnostic cost almost nothing and inverted the conclusion. The general point is that when a benchmark is built with an adversarial guard, a zero on that benchmark has two readings, "the capability is absent" and "the guard is doing its job," and only a second measurement that removes the guard can tell them apart. We now treat a flat zero on a guarded benchmark as a question, not an answer.
For a constant-state trunk to serve or train beyond a single window, its recurrent state must carry from one window to the next without approximation, because any drift introduced at a window boundary would accumulate over a long conversation and would make past-window measurements untrustworthy: we would not know whether a low far-distance recall reflected the model or the seam. We therefore verified carry exactly rather than approximately. Carrying both the state matrix and the convolution tail across two window boundaries reproduces the one-pass output and the final state bitwise, with a maximum absolute difference of 0.0, for all three state rules: uniform-decay, delta, and pinned. The convolution tail at the window seam was the correctness crux. The state matrix is the obvious thing to carry, but the short convolution has a receptive field that straddles the boundary, so the last few bytes of one window and the first few of the next share support; carrying only the matrix and dropping the tail leaves a discontinuity exactly at the seam, and the bitwise identity fails there before it fails anywhere else. Carrying the tail closes it.
Two consequences follow, and both are load-bearing for the rest of the programme. First, past-window needle distances are now real measurements: when the benchmark reports 0.00 at 8k bytes, that is a statement about the model's memory and not an artifact of a lossy window stitch, because window-carried streaming equals a single pass exactly. Second, chat length is no longer capped at the training sequence length. Because a streamed run is provably identical to a single pass over the whole sequence, a conversation may run arbitrarily long at inference by feeding it window by window, without the model ever seeing a discontinuity it was not trained through. This is genuine infrastructure. It is also, importantly, not yet a capability: proving that state can be carried exactly across windows is a different thing from proving that a model trained with that carry actually retrieves anything across them, and we are careful not to conflate the two.
No memory mechanism has yet flattened the curve, and the reason the surviving routes look the way they do is that the obvious routes have already been tried and killed. Negative results, and why they are worth keeping records both learnable-mechanism failures at 10M: the delta-rule state (M1) would not train stably, and the pinned register (M2) cost quality without buying recall. Those two negatives matter here because they rule out the tempting move of designing a cleverer decay rule and hoping it flattens the curve. The surviving routes are therefore not about a smarter forgetting scalar at all; they are about training pressure and inference-time refresh. Read against the two hypotheses above, the first route attacks the flat past-window zero (H1), the second and third attack the in-window collapse and the transfer gap (H2).
The stateless training forward resets the recurrent state at every window, so retrieval beyond one window is never trained; the long-range pathway receives no gradient, which is the direct mechanical reason past-window recall is 0 at every scale we have measured. Carrying the state across the contiguous chunks of a training step, rather than resetting it, makes cross-window retrieval trainable at all: the gradient can now flow from a query in a later chunk back to a fact planted in an earlier one. This is built, and it is verified bit-identical to the old trainer when the carry is off, so turning it on changes only the thing it is meant to change. What remains is the run. The validation run is queued, not yet done, and its success criterion is deliberately concrete and currently impossible to meet by construction: any in-distribution recall beyond 512 bytes, a distance at which the current trainer, resetting every window, cannot by construction produce anything but zero. Until that run is done, train-time carry is a mechanism we have made possible, not a mechanism we have shown works.
Surprise-gated rehearsal is an inference-time mechanism, and its appeal is that it sidesteps the training loop entirely. It tracks per-turn prediction surprise and, when surprise spikes on back-referencing content (the model is suddenly uncertain in a way that suggests it is reaching for something it can no longer see), re-encodes a compressed digest of the conversation so far and re-injects it into the state. Because it changes nothing about training, it is immune to training interference by construction, which is exactly the failure mode that killed the M2 pinned register: a mechanism that cannot damage training cannot repeat that mistake. The A/B against the same checkpoint is in flight. Its kill condition is stated in advance and is strict: no recall lift kills it. We are not looking for a plausible story about why rehearsal should help; we are looking for the curve to move, and if it does not, M3 joins the negatives.
The third route is the retry condition drawn directly from the recall-SFT failure in How a 121M byte model learned to talk, and it is precise about what went wrong and how to avoid it. A small in-pretrain dose of recall data taught genuine retrieval at 10M, which is the same capability the transfer-gap diagnostic measured at 0.47 in distribution. A concentrated late-phase dose, by contrast, overwrote the model's copy behaviour: it taught the narrow recall skill at the cost of the general ability that made the model useful. The retry condition therefore has two parts, and both are load-bearing: keep the dose low, so it does not overwrite, and make the surface forms diverse, so the skill that is learned is retrieval-in-general rather than retrieval-of-these-exact- shapes. Diverse surface forms are the direct answer to the transfer gap: if the training pool contains hyphenated codes and dollar amounts and many other shapes, the alien-surface-form recall of 0.00 is the number this route exists to move.
For a general reader, the shape of the situation is this. The design gives cheap long memory by its shape: the model can hold state over a very long conversation without its memory bill growing, which a conventional model cannot. But "can hold state" is not the same as "does retrieve," and the gap between those two is where all the difficulty lives. At small scale the model can learn to fetch facts that look like its training data, yet it fails on facts written in unfamiliar ways, because it has learned a narrow skill rather than a general one. And it never learns to reach past its short-term window at all, because the training procedure wipes that memory between windows, so the long-range ability is never practised. The fixes we are pursuing are about how we train and refresh the memory, not about inventing a smarter forgetting rule, because the two smarter rules we tried both failed, and failing them told us where not to look. A useful way to picture it: the model has a filing cabinet it can carry cheaply no matter how full it gets, but so far we have taught it to file only the kinds of documents it has seen before, and the ordinary training day never asks it to retrieve anything filed more than a few minutes ago.
We keep the limitations in view rather than at the end. The headline recall numbers, 0.92 near the plant falling to 0.25 and then to 0.00, are measured on synthetic needle conversations under greedy decode. That is a deliberate stress test, chosen because it is hard and reproducible, and it is not a natural-conversation evaluation; a real conversation would be easier in some ways and harder in others, and these numbers should not be read as an estimate of everyday behaviour. The transfer gap is a 10M finding. Whether 121M or a larger model bridges alien surface forms on its own, without the diverse-surface-form data above, is untested, and we should not assume scale alone closes it. Streaming carry is proven exact, with a maximum absolute difference of 0.0, and it is real infrastructure, but proving that a model trained with state carry actually retrieves past its window is still ahead of us, and the exactness of the carry says nothing about whether the retrieval will materialise. The campaign's current honest position, stated without hedging in either direction, is that the benchmark and the streaming machinery are done and trustworthy, the training-pressure lever is real but unproven past the window, and the mechanism race so far has produced a set of negatives and one corrected finding. That is a genuine foundation and it is not a solved problem, and we decline to describe it as either less or more than that.