Negative LLM Training Results
Falsified approaches at 10M+ params: fast-weight mem, test-time depth, byte-level selective loss, delta-rule instability, mem registers, and the MoE tax.
Falsified approaches at 10M+ params: fast-weight mem, test-time depth, byte-level selective loss, delta-rule instability, mem registers, and the MoE tax.
Byte-level training efficiency on one consumer machine: Muon, boundary patching, constant-state recurrence, and the hybrid trunk that beat both of its parents.
How an affine, fixed-cost-per-update model of training time turned a budgeted four-to-five-day run into a measured two-day run.
The three-phase recipe behind chat80m, the first conversing byte-level model in the project, its knowledge wall, and the identity-dosing discovery.
The conversation-needle benchmark, the transfer-gap finding, bitwise-exact streaming state carry, and the open programme for training past-window recall.
The v13 engine: Constant-size decode state, byte-for-byte parity to the reference and about one millisecond per byte on a CPU core.
Fine-tuning teaches behavior, RAG supplies knowledge, prompting shapes both. What the published comparisons found, and how to pick without burning a GPU budget.
Falsified approaches at 10M to 121M parameters: fast-weight memory, test-time depth loops, byte-level selective loss, delta-rule instability, pinned memory registers, and the mixture-of-experts throughput tax on this hardware.
Most of the ideas we tested did not work. Each dead end is recorded here with the measurement that killed it and the condition under which it would be worth trying again. The reason for writing them up is practical, and it is twofold. First, a negative result with a clearly identified cause is saved compute: it stops the next person, or the same person three weeks later, from re-spending the training runs we already spent to learn the same thing. Second, several of these kills left behind an artifact that outlived the idea, a device or a method that we kept and that shaped the models that did work.
We describe six falsified approaches. Two are memory add-ons that stored soft impressions but never retrievable facts. One is a "think longer at test time" depth loop that, when measured, did not think longer. One is a token-skipping training trick that skipped exactly the wrong tokens at byte level. One is an error-correcting memory rule that is exact on paper and in double precision but would not train in single precision without overflowing, and whose failure we trace all the way down to the arithmetic. One is a hand-designed memory register that cost quality without buying recall. The sixth is not a failure of an idea at all but of a machine: a mixture-of-experts design that is throttled by this specific hardware, and whose verdict is therefore a sequencing decision rather than a kill. Three of the six left something we kept: a carried memory that turned out to be a free context extender, a numerical fix for the delta rule that is now a general-purpose stabiliser, and a tail-averaged evaluation method that is now standard across the project. None of the six is a claim that the underlying published idea is wrong. Each is a claim about what happens to that idea at 10M parameters, on one Apple M3 Ultra, in single precision, with our data, and each is bounded by a retry condition that says exactly what would reopen it.
Terms are defined at first use. "Val" is validation loss in bits per byte (bpb), lower being better. "10M" refers to the canonical 10-million-parameter byte-level rig used for the architecture race (FineWeb-Edu corpus, 12000 steps, one seed, single-change arms), the same rig as The composed efficiency stack. "Byte-level" means the model reads text one byte at a time, 256 possible values, with no word-piece dictionary. A "single-change arm" differs from its baseline by exactly one thing, so any difference in the outcome is attributable to that one thing.
The project keeps two ledgers, successes.md and failures.md, and treats them as equally load-bearing. The failures ledger is not a diary of disappointments; it is a kill list, a working instrument. Each entry states three things: what was tested, the measured number that killed it, and a retry condition, which is the specific new evidence or setting that would reopen the question. An approach on the list is treated as dead until its retry condition is met, and not one day sooner. This is deliberate discipline. The alternative, the default failure mode of any long research programme, is to re-run a falsified experiment every few weeks because nobody wrote down why it failed, or to re-run it because the intuition that motivated it is genuinely appealing and keeps returning. A retry condition converts that recurring temptation into a checklist item: the idea comes back only when the world has changed in the one way that would matter.
Two properties of these kills are worth stating plainly, because they are what make a negative result trustworthy rather than merely discouraging. The first is that a kill here is always scoped. We do not write "fast-weight memory does not work"; we write "fast-weight memory does not store retrievable facts at 10M parameters on this data, and here is the structural reason." The scope is the difference between falsifying an idea and falsifying an idea at a particular operating point. The published versions of every method in this paper were demonstrated at scales, on kernels, and with training budgets we do not have, and nothing here contradicts those demonstrations. What we falsify is the hypothesis that the method survives transplant to our regime unchanged. That is a narrower claim, and a defensible one, precisely because it is narrow.
The second property is that the killing measurement is pre-registered. Before a run, each arm carries a falsifier: a threshold, decided in advance, that the run must clear or die. The point of committing to the number beforehand is to remove the researcher's freedom to move the goalposts after seeing the curve, which is the single most common way a dead idea stays alive. When an arm misses its pre-registered falsifier, the verdict is mechanical, and the ledger records the miss rather than a rationalisation of it.
Two standing methods in the project were themselves born from the failures below, which is the clearest possible argument that negatives are first-class. The looped arm (section 2) and the selective-loss arm (section 3) both produced validation curves so noisy that a single milestone evaluation could report a win or a loss essentially at random. That noise is not a nuisance to be ignored; it is a measurement about the arms, and it forced a change in how every arm is judged. Verdicts across the project now use tail-averaged validation loss, the average over the final stretch of a run, rather than any single checkpoint. The habit of quoting a tail standard deviation alongside a tail mean also comes from these arms. A negative result that changes your evaluation protocol has paid for itself several times over before it is even filed.
The idea belongs to the fast-weight lineage: a network that writes transient associations into a rapidly changing set of weights during the forward pass, rather than only into the slowly changing weights fixed by training. A "fast weight" is exactly that, a weight updated during a forward pass rather than by gradient descent. Revived recently as the Titans line of learned long-term memory, the appeal for a small model is obvious and seductive: if the model can write a fact into its fast weights at inference time, it could learn a new fact without a full training pass, effectively carrying an editable scratch memory that ordinary attention lacks. For a byte-level model that will never be large enough to memorise the world in its slow weights, a working fast-weight store would be a genuine capability, not just an efficiency.
We attached such a memory trunk to the 10M rig and asked one blunt question: can it hold a fact you state once and quiz later. We ran it in two configurations, differing only in whether the memory is reset or carried between windows.
E4 (memory reset every window). Plain-text language quality was essentially preserved: val 1.0026 versus dense 0.9990, a regression of 0.004, comfortably under the 0.05 kill line for a single change. So the memory trunk did no harm to ordinary modelling. But the capability it was built for did not appear. A knowledge-injection test (state an invented fact, then quiz it after 8000 distraction bytes) gave a recall lift of 0.0 points: 0 of 12 correct on both the memory-on and the memory-off arms, where the pre-registered falsifier needed +10. The two arms were indistinguishable, which means the memory contributed no discriminating signal at all. The diagnosis was structural, not a matter of tuning. If the memory is reset at every window boundary, and attention already spans the whole window, then within any window attention can already see everything the memory could hold, and across windows the memory is gone. The training loss therefore never has a reason to write anything into the memory that it will need to read later, because there is no later. Gradient descent optimises what the loss rewards, and this configuration rewards nothing for cross-window retention. The store stayed empty because nothing ever asked it to fill.
E4b (memory carried across windows). The concrete retry followed directly from that diagnosis: give the memory a job that attention cannot do. We carried the memory across the four contiguous 512-byte chunks the data loader already feeds per step, so that the memory serves bytes 513 to 2048, beyond the 512-byte attention window. Now the loss has a reason to use the store, because information from early bytes can only reach late bytes through it. This time language quality was not merely preserved, it was excellent: val 0.9867, which beats dense-muon's 0.9990. The reason is illuminating. The carried memory works as a free context extension: attention still sees only 512 bytes, but the memory spans 2048, so the model gets a wider effective horizon at no extra attention cost. That is a real and useful effect, and it is the constant-state long-range idea of The composed efficiency stack arriving through a side door. But explicit recall still failed. Exact-match recall lift was again 0.0 at every distraction length tested. Where the memory did leave a trace, it was a soft one and only within the horizon it was actually trained on: within the trained 1024-byte range there was a fact-span bits-per-byte win rate of 0.667 (5.42 bpb on the fact span with memory on, versus 5.52 off), meaning the model was slightly less surprised by the fact when the memory held it, which is an impression, not a retrieval. Beyond 4096 bytes the memory did not merely fail to help, it hurt: a stale carried state actively degraded prediction, win rate 0.167, memory-on worse than memory-off. A memory with no mechanism to forget or overwrite becomes a source of confidently wrong context once its contents go out of date.
The fast-weight route to explicit, exact fact recall is dead at 10M. The reason is not that the mechanism is broken but that at this scale and horizon the training signal never rewards writing a retrievable fact, and a soft impression is not a retrievable fact. What survives is real and was kept: the carried memory is a free long-context device, so the memory trunk remains in the codebase as a context extender, just not as a knowledge store. That surviving use reappears as the streaming state carry studied in Long-context memory for constant-state models, and the recall failure here is the same wall that paper measures head-on. Retry conditions are specific: train with carry horizons at or beyond the recall distance (8k bytes or more), so that the loss is forced to depend on long-range writes; go to a larger scale, since the Titans evidence floor is 170M and nothing here contradicts it above that; or switch to retrieval-based memory, which sidesteps the whole problem because it needs no gradient into the store at all.
The idea comes from the recurrent-depth family: reuse a single block of weights several times in sequence ("weight-tied loops"), train with a random number of loops so the block learns to be iterated, then run more loops at test time to "think longer" on hard inputs. It is the universal-transformer notion of tying depth into a repeated block, sharpened in the Ouro and recurrent-depth line into a concrete recipe. The promise is test-time compute scaling: a single trained model that you can dial up on demand, spending more forward passes for more accuracy without retraining. We built trunk=looped: patched local blocks plus half the usual number of unique global blocks, iterated R times, with R drawn uniformly from 1 to 4 each training step (the Huginn recipe of a random loop count during training), parameter-matched to dense at 10.12M versus 10.08M so the comparison isolates the loop, not a parameter change.
The claim is a claim about what happens when you push R past what the model trained on, so we tested exactly that. On the trained checkpoint we swept R over 64 identical validation batches and read the cross-entropy at each: 1.0103, 0.9975, 0.9965, 0.9983, 1.0050, 1.0142 at R = 1, 2, 3, 4, 6, 8. Quality peaks at R = 3, which is essentially the training-time mean depth of 2.5, and then degrades monotonically as R climbs to 6 and 8. This is the signature of interpolation, not extrapolation: the model performs best at the centre of the depth distribution it was trained on and gets worse the further you push it outside that distribution. There is no test-time compute scaling here. Running more loops than the model saw in training does not buy more thinking, it buys drift away from the regime the weights were fit to. A second measured cost is methodological and turned out to matter project-wide: random-R training makes the validation loss noisy, with a tail-20 standard deviation of 0.0087, about nine times that of the other arms. A single evaluation of a looped run is therefore untrustworthy on its own, which is one of the two arms that pushed the whole project onto tail-averaged evaluation.
The think-longer claim is dead at this scale: measured, the loop does not scale with test-time depth, it interpolates within its training distribution. One result survives, but we are careful not to scale it, because it is not cleanly attributable. At matched parameters the looped arm beats dense-muon (0.9920 versus 0.9990, at 1.63x wall-clock, ahead at 111 of 120 evaluations). That is a real number, but it is unattributed: we never trained the loop-free control with the same halved set of unique global blocks, so the gain may belong to the patched front-end rather than to the loop itself, and honesty requires we not credit the loop for it. Against the trunks we do scale, the looped arm loses on both quality and wall-clock: it is behind full patched (0.9776) and behind the hybrid (0.9707), so it is simply not the trunk to build on. Retry conditions: a parameter-bound deployment where the loop-free control is trained first, so the win can be attributed cleanly to the loop or to the front-end; or reasoning-style executed benchmarks with loop counts trained beyond 4, since the failure here is about extrapolating past the trained depth, and a model trained to loop deeper is a different and untested proposition.
The idea is RHO-1's Selective Language Modeling: rather than compute the loss on every token, train only on the tokens where the student model's loss most exceeds a frozen reference model's loss, on the theory that the easy, already-predictable tokens are wasted gradient and the learning happens at the surprising ones. On subword vocabularies the published result is strong, and the intuition is sound: if a token is trivially predictable, the gradient it contributes is near zero anyway, so dropping it costs little and concentrates the compute where it helps. We ran slm_keep=0.6, training on the surprising top 60 percent of tokens as scored by a frozen dense-muon reference, applied as a single change on top of the patched trunk.
Tail-10 val was 1.0638 versus the patched baseline's 0.9763, a regression of 0.0875, about eight sigma beyond the tail noise. This is not a marginal loss, it is a decisive one. The arm never reaches the baseline's final quality at all, so the byte-savings-to-target, the metric that would justify a selective method, is zero: you cannot save bytes reaching a target you never reach. The shape of the curve is the instructive part and explains the mechanism of the failure. Selective loss is fast to a mediocre plateau: it reaches the 1.064 level 1.24x sooner in steps than the baseline passes that same level, and then it flatlines. Early speed followed by a ceiling is exactly what you would expect if the method is over-weighting a subset of tokens that carries the easy early gains but not the structure needed for the last stretch of quality. A second observation, now a standing method, is that per-evaluation variance on this arm was large (standard deviation about 0.035), so single milestone evaluations lied in both directions; together with the looped arm, this is why verdicts moved to tail-averaged val.
At byte granularity the premise fails for a concrete reason: the "easy" 40 percent of tokens is not skippable filler. At the byte level a token is a single character-code, and the low-loss bytes are the whitespace, the morphology, the punctuation and the tag syntax, that is, the structural scaffolding that the hard, high-loss bytes sit inside. Down-weighting the structure to concentrate on the surprising bytes removes the frame that makes the surprising bytes predictable in the first place. The RHO-1 result is not in question; what fails is the assumption that a byte is a semantic unit whose easiness implies redundancy. Retry conditions follow directly: apply the selection at subword or patch granularity, where a token is a semantic unit and "easy" more plausibly means "redundant"; use a reference model markedly stronger than the student, so the excess-loss signal points at genuine learnable difficulty rather than at student noise; or restrict selective loss to a mid-training anneal phase, where the early-speed effect the curve does show is real and the eventual plateau does not matter because the phase is short by design.
This is the centerpiece kill, because it is the one where we followed the failure all the way down to the arithmetic and found a general trap that any implementer of this class of model can fall into.
The idea comes from Gated DeltaNet, and behind it the classic delta rule, the Widrow-Hoff error-correcting update, transplanted into the recurrent state of a linear-attention memory. The constant-state trunk of The composed efficiency stack keeps its long-range memory in a fixed-size matrix that is updated once per byte. The default update fades the whole memory uniformly and adds the new content. The delta rule is smarter: before writing a new key-value pair, it selectively subtracts out whatever the memory currently associates with that key, so the new value overwrites the stale one in that slot rather than piling on top of it. This is an error-correcting write, and the literature shows it improves recall precisely because it stops old and new associations for the same key from interfering. The per-byte update is
S <- alpha (I - beta k kT) S + beta k vT
where S is the state matrix, k and v are the key and value, alpha is the per-byte decay, and beta is the write gate. We tried it as state_rule=delta on the hybrid trunk, twice, and both times it diverged. The interesting part is why.
Run 1 (pure-chat data). Training crept to divergence at step 4420 of 12000: after a grad-norm spike the loss went non-finite and stayed there. The root cause is numerical, and it is worth stating precisely because it is a general trap that hides well. Training the delta rule efficiently uses a chunkwise-parallel formulation: instead of stepping the recurrence one byte at a time, you process a whole chunk of bytes in parallel by solving for the within-chunk interactions in closed form. That closed form requires the inverse of a matrix of the shape (I + M), where M encodes the in-chunk key-key interactions and is strictly lower triangular. Because M is strictly lower triangular it is nilpotent: its powers eventually vanish, so the inverse is a finite series, and the implementation forms it as a nilpotent expansion with terms up to the 32nd power of M. In exact arithmetic this is beautiful. The high powers of M can individually be enormous, but they alternate in sign and cancel, and a contraction argument guarantees the whole thing collapses to a bounded result: at every checkpoint the float64 (double precision) maximum entry of the true inverse is 1.000. The trouble is that the cancellation is only exact if the arithmetic is exact.
In 32-bit floating point, which carries only about 7 significant decimal digits, the cancellation is not survivable once the intermediate terms grow large. To land on a bounded value of order 1 by subtracting quantities of order 1e17, you would need roughly 17 digits of significance, and fp32 has about seven. Everything below the seventh digit is rounding noise, and after the cancellation the rounding noise is all that is left. We measured it directly. The fp32 inverse maximum entry grew from 1.3e8 at step 800, to 3.4e10 at step 4000, to 5.2e10 at step 4400, at which point the run diverged, while the true value the computation was supposed to produce was at most 1.000 the entire time. The computed inverse was not slightly wrong, it was wrong by ten orders of magnitude, and it had been wrong for thousands of steps.
The trigger for the growth was a specific, identifiable behaviour of the model, not a random accident. The entries of M scale with the write gate beta and with the alignment of the in-chunk keys, amplified by the per-byte decay. When beta saturates toward 1, and the keys within a chunk point in similar directions, and the decay is close to 1 so nothing shrinks, the off-diagonal entries of M grow and their high powers explode. That is exactly what happened on one head. The write gate beta saturated on the slowest-decay head, the head the architecture designates as the long-memory head, and it was this head that drove the overflow. At the failure point that head had beta mean 0.93, beta max 0.999, a third of its gate outputs above 0.99, per-byte decay near 0.998, and its in-chunk keys aligned with a dot-product mean of 0.55. Every one of those is a term that pushes M larger and its powers larger. The head was, in fact, emitting fp32 noise from step 800 onward, which is where the measured maximum entry had already reached 1.3e8.
The reason the run did not die at step 800, when the head first went numerically bad, is the most cautionary part of the whole episode. A downstream normalisation layer sat between the broken head and the loss, and a norm layer rescales its input to a bounded output. So even though the long-memory head was producing garbage of magnitude 1e8 and rising, the norm downstream squashed it back to a normal scale, and the run trained around the broken head for thousands of steps as if nothing were wrong. The loss kept descending; the gradients looked reasonable. The failure was masked, not absent. It stayed masked until the intermediate finally grew past the fp32 representable ceiling and produced an actual NaN, which the norm layer cannot rescale, and then a single optimiser step spread that NaN into every weight and the run was gone. A method can be numerically dead for thousands of steps and still look alive if something downstream is quietly hiding the corpse.
The fix, which is shipped and triple-verified, computes the same exact inverse without ever forming the large intermediates. Rather than the nilpotent power series, which builds up the enormous high-power terms and then relies on catastrophic cancellation to knock them back down, it uses a block-recursive triangular inversion with a small block size. A lower-triangular matrix can be inverted block by block, expressing the inverse of the whole in terms of the inverses of small diagonal blocks and a bounded combination with the off-diagonal blocks, and this path never raises M to a high power at all. The worst-case amplification along any path of that computation (the "worst path weight") is 20, against roughly 4.5e17 for the power series. That is the whole story of the fix in one comparison: the old method's answer had to pass through a quantity of order 1e17 on its way to a result of order 1, and the new method's answer never exceeds order 20. On a deliberately saturated stress test, one built to provoke the exact conditions the long-memory head fell into, the old code reaches 2.4e31 and fails, while the new code matches the exact double-precision oracle to 5.4e-7. And a 50-step exact resume of the death window, replayed with the fix in place, is finite throughout with flat gradient norms. The stabiliser does what it was designed to do: it makes the arithmetic robust, so the algorithm's exactness on paper becomes exactness in single precision.
Run 2 (m1delta2, recall-mix data, fixed inverse). With the numerical fix in place, we retrained from scratch on a recall-pressure data mix, and it diverged again, at step 2094, with a completely different signature. Run 1 had a slow, visible build-up of a bad intermediate over thousands of steps. Run 2 had none of that: a sudden single-step NaN with no precursor at all. The gradient norms were flat at 0.37 to 0.44, and the loss was descending healthily through 0.77, and then in one step it was gone. The stabiliser demonstrably worked, in that a replay of Run 1's specific divergence is now finite; but training from scratch on different data found a second, different path to failure that the first fix does not address. This matters for the verdict: the numerical trap was real and is now closed, and the delta formulation is still not trainable to completion on this stack.
Killed on trainability: two divergences under two different numeric regimes, one retry spent per failure. The scope of the kill is important and narrow. The dense baseline and the pinned arm (section 5) both trained cleanly on the same configurations, so the instability is specific to the delta formulation on this stack (Muon, fp32, Apple MPS, byte level) run without the specialised fused kernels the published work trains with. The oracle-exact math and the literature's recall results are not in question; what we falsify is that the delta rule survives naive transplant into single-precision MPS without a hardened kernel. Retry conditions are concrete: cap beta away from saturation, so the head cannot drive M into the regime where even the stabilised inverse is stressed; use a markedly lower learning rate on the delta projections, so the write gate and key directions evolve more gently; or move to a proper fused kernel path if the platform ever leaves MPS, since the published implementations depend on exactly that. The general lesson is the one the fp32 forensics teach: an algorithm that is exact in real arithmetic can still be untrainable in single precision if its intermediates must cancel from 1e17 down to 1, and, worse, the failure can hide behind a downstream norm layer for thousands of steps before it finally overflows.
This one is our own invention, not a transplant, which makes its kill cleaner: there is no published result to caveat against, only our own hypothesis to test. The idea is to reserve a small fixed set of memory slots, "pins", that are exempt from the decay gate, so that a handful of important facts do not fade the way the rest of the state does, with a learned controller deciding what salient content to write into the pins. The intuition is that a constant-state memory forgets uniformly, and that a few protected slots could hold a needle against the tide of decay. We tested it twice at 10M.
On pure-chat data the arm trained stably, at 77k bytes per second, 97 percent of the baseline throughput, so the mechanism is cheap; but it carried a real quality tax. Tail-10 val was 0.7637 versus the baseline's 0.6907, a regression of 0.073, about seven sigma, while needle recall stayed at the floor: 0.00 versus 0.00, no discriminating power at all. So the pins cost quality and bought nothing. On a fairer recall-pressure mix, judged against the correct baseline and applied as a single change, quality came out neutral (0.7010 versus 0.6990), which removes the quality objection, but in-distribution recall was now consistently below baseline rather than above it: 0.338 versus 0.423 at 130 trials, and 0.28 versus 0.47 at 40 trials, never above at any sample size we ran. The register did not merely fail to help recall, it measurably hurt it.
Killed on the pre-registered condition, "pinned recall no better than the baseline": there was no measured benefit in any sample, and there was a measured cost. The mechanism of the failure is that at this scale raw state capacity is the binding constraint, and the pins spend it badly. Decay-exempt pins plus a hand-designed salience writer subtract learnable state capacity from the general memory and hand it to a controller that, receiving only the language-modelling gradient, never learns to use it for retention. You pay for the protected slots and the controller in capacity, and you get a protected region that holds the wrong things. Retry conditions: a much larger scale, where raw state capacity stops being the binding constraint and reserving some of it costs less; pins trained with an explicit auxiliary recall loss, so the controller receives a gradient that actually rewards putting a retrievable fact into a pin, which it currently never does; or a streaming-trained regime, where cross-window retention is a trainable objective rather than an accident, so that a protected slot has a job the loss can teach it.
The last case is not a failed idea, it is a well-founded idea colliding with the wrong hardware, and the honest verdict reflects that. A mixture-of-experts (MoE) feed-forward layer stores many "expert" sub-networks but activates only a couple of them per token, so it buys many more parameters at nearly fixed active calculation cost. The DeepSeekMoE design we built, 8 fine-grained routed experts at quarter width plus 1 always-on shared expert, top-2 routed, is in principle an excellent fit for this box, which is rich in memory but limited in arithmetic. Extra parameters are cheap here (there is plenty of memory to hold them), and extra active calculation is expensive (arithmetic is the scarce resource), so a design that trades parameters for held-constant compute is aimed straight at the machine's strength. We built and smoke-tested it but did not launch a full run, for a measured reason.
Realised throughput was 0.715 of the hybrid baseline, 32.0k versus 44.8k bytes per second, measured on MPS at the standard batch and sequence. That sits right on the pre-registered kill line of 70 percent, clearing it by only 1.5 points, a margin thin enough that Muon's per-step overhead might push it under. The cause is hardware, not design, and it is specific. MoE's "free parameters" premise relies on sparse routing: on a datacentre GPU, sparse kernels dispatch each token only to its chosen experts and skip the rest. MPS has no sparse-routing kernels, so the capacity-based dispatch instead computes expert outputs over padded and dropped slots, doing the arithmetic for experts a token did not choose and then throwing it away. On the arithmetic-poor side of a machine that is otherwise arithmetic-poor, that wasted compute is exactly the resource you cannot spare. Parameters came in at 1.52x the hybrid, which is cheap on 256 GB of memory and is precisely the trade the design promises, but the calculation side is at the edge of acceptable because the platform cannot realise the sparsity that would make the trade pay.
MoE's free-parameters premise is partly defeated by this specific machine, not by the idea. Because the constraint is hardware and the throughput margin is razor-thin rather than clearly failed, the ledger records a sequencing decision rather than a kill. MoE is not bet on the primary chat model, where a 1.5-point throughput margin is too fragile to build a multi-day run on; it is instead reserved as a controlled A/B at the 80M scale, where DeepSeekMoE's own quality evidence begins, so that the throughput tax can be weighed against a real, measured quality delta rather than against zero. The decision rule is pre-committed and symmetric: if at 80M it clears neither 70 percent throughput nor a quality win, then it is dead on this hardware regardless of scale, and the sequencing decision converts into a kill.
None of these six are failures of the underlying published ideas, and it would be a misreading to file them that way. They are failures at our scale, on our hardware, or with our data, and each is fenced by a retry condition that says exactly what would change the verdict. That precision is the product. A vague negative ("we tried MoE and it was slow") is worthless, because it neither stops the next attempt nor guides it. A scoped negative with a retry condition ("MoE clears the throughput line by only 1.5 points on MPS because there are no sparse-routing kernels, revisit as an 80M A/B") does both: it saves the compute of a blind retry and tells you the one thing that would make a retry worthwhile.
Three of the six left something we kept, and it is worth naming them together, because they are the concrete return on the compute these kills cost. First, the carried fast-weight memory (E4b) is a free context extender, and it lives on in the codebase and in the long-context work of Long-context memory for constant-state models as a way to widen a small model's effective horizon at no attention cost. Second, the delta-rule numerical fix (M1) is a general-purpose stabiliser, not a one-off patch: any chunkwise linear-attention state on this stack inherits it, and the block-recursive triangular inversion is now the default path. Third, the tail-averaged validation method, forced on us by the noisy looped and selective-loss arms, is now the standard verdict rule across every experiment, along with the habit of reporting a tail standard deviation beside a tail mean.
The delta forensics in particular are a reusable warning that we expect to pay off again, because the trap is generic to a whole class of efficient-attention kernels. An algorithm can be exact in real arithmetic and exact in double precision and still be untrainable in single precision, if its intermediates must cancel from magnitudes of order 1e17 down to order 1, because single precision has only about seven digits and cancellation of that depth consumes far more. And the failure need not announce itself: it can hide behind a downstream norm layer for thousands of steps, descending loss and flat gradients and all, until the intermediate crosses the fp32 ceiling and a single optimiser step turns one NaN into a dead model. The lesson we drew, and the check we now run, is to watch the raw magnitude of the intermediate rather than trust the loss curve, because the loss curve is exactly what a downstream norm layer is best at keeping honest-looking while the arithmetic underneath it is already gone.