A routing collapse at N = 3, and what replaced it

Routing was closed form and the ship gate was 85% top-1 route accuracy. At two specialists the router cleared it: 18 of 20 combined across bases, with both misroutes low-margin general-to-architecture confusions on the cross-base wrapper. At three and above, over capsules from similar domains, top-1 accuracy fell to 23.3% and never recovered past 46.7%. One specialist stopped being reachable at all. The code capsule was absorbed by the architecture capsule and sat at 0% from five specialists onward.

Three follow-ups were run to find the mechanism. Each sharpened the result rather than softening it, and the last one is the reason the line was stopped.

A mixture of experts computes w = gate(x). XE4-S computes w = compose(x, m), where m summarises where a beam descent through a runtime-written memory of user facts terminated. Everything else about the stack is kept.

The composer routes 18 of 19 task shapes correctly in distribution. The bank it does that in is thirty-one wide. Twelve of those thirty-one are knowledge specialists that were never trained, because XE4-S ran with no knowledge corpus, and the composer gives all twelve exactly 0.000 weight, mean and max, on every probe. No mechanism asks it to. There is no dead attractor and nothing absorbing the mass of a neighbour.

That is fifteen times the specialist count at which the previous architecture stopped working. It is also a different quantity from the one in Figure 01, and the paper says so before anyone else can. Panel (a) of its own figure measures top-1 selection accuracy over a sparse active set chosen by capsule geometry; the N = 31 result measures whether a dense composer that selects nothing puts its mass on the right specialist. What transfers between them is the direction of the curve, not the axis.

A companion run of the same architecture did have a knowledge corpus, and its composer routed 12 of 12 knowledge branches to one specialist, know:Home. One destination for every input, which is the zero-bit failure of the NCN v2 router in different notation.

The cause was found by arithmetic rather than inferred. The batch sampler walked a sequential cursor from block zero. Stage E runs 2,500 steps, half of them on knowledge at batch 24, so its knowledge stream consumes 30,000 blocks of 512 tokens, 15.4 M tokens, starting at block zero. The knowledge corpus is laid out branch by branch and Home occupies blocks 0 through 34,921. The stream never left Home. Eleven of twelve labels were never once shown.

Measured directly against the real corpus over the exact 1,250 steps at batch 24 the stage uses: a sequential cursor shows 1 of 12 labels with Home at 100.00%. Random block draws show 12 of 12 with Home at 9.00%, and after the fix every branch appears in proportion to its size, from Person at 3.10% to Nature at 9.42%.

One parallel cuts against both architectures. In the NCN line the obvious remedy made things worse: a corpus nearly twenty times larger dropped N = 6 accuracy. In the XE4-S line the obvious remedy made things worse too, and three separate attempts to train the composer harder are the record of it. Neither system's routing problem was ever short of the thing intuition said it was short of.

Tokens enter a trunk of six pre-norm decoder blocks at d = 384, six heads of width 64, a 4d GELU feed-forward and a head weight-tied to the token embedding. The trunk's hidden states go three places: they are queried against the memory tree, they drive the composer, and they are the input to every specialist. The composer emits thirty-one weights, the specialist outputs are scaled and summed, the sum passes through the retrieval stage, and the head decodes it.

Position enters as a learned embedding table with one row per index, which is why the context window is exactly 512 and cannot be moved without retraining. There is no row for position 513, and asking for one raises an index error rather than degrading.

The memory is part of the network

Node keys are the l2-normalised mean of the trunk's token embeddings for the words of a node's breadcrumb. No trained parameter appears in that formula, so inserting a fact cannot move the key of any existing fact. A beam descent with a margin gate keeps two candidates per level and produces both the context vector m and a printable path; each hop contributes a log-probability and the hops compose along the route, so on a five-fact tree a survivor's weight is 0.868 against 0.132 where a flat mean gave 0.500 against 0.500. Retrieved nodes are prepended to the keys and values every block attends over, so a fact stays attendable after the turn that produced it has left the window.

Writing is gradient-free and sub-millisecond, addressed by breadcrumb, rewriteable at that address and retractable. Writing moves the output by 7.7×10⁻², rewriting the same address a further 1.17×10⁻¹, and forgetting returns the model to baseline at exactly 0.000×10⁺⁰⁰ with the symbolic path intact. Memory columns are gated rather than merely zeroed, because a zero value still takes attention mass out of the softmax: a zero value perturbs the output by 0.3, an additive logit bias of −10 on the memory columns by 2.6×10⁻⁵. The gate opens to 0.34 once trained.

The composer is the smallest thing in the system

Two linear maps and a softmax, 24,638 parameters, an order of magnitude smaller than anything else here. The claim is not that it has capacity, it is that m is in its argument list. W_m is zero-initialised, so an untrained model is exactly a router and the memory term has to earn its contribution rather than arriving as noise. It earned it: norm 11.22 against W_x's 12.14, which is 48% of the routing signal. A use bias is written by use and never by gradient, one scalar per specialist, step 0.0085 and cap 0.20, so one interaction moves a specialist's weight by under 0.2%, a hundred consistent ones move it about 4%, and then it stops.

Specialist sampling was got wrong twice before it was got right. Interleaved masking gives full input coverage and roughly 10% gradient density, which is 14.7 M effective tokens per specialist, 4.2 tokens per parameter against the trunk's 123. Contiguous slicing gives roughly 100% density and only ever shows a specialist hidden states produced by its own shape, which scored worse end to end, 36.5% against 57.9%. Stage C reads the grouped corpus and stage E reads the interleaved one, and that is not a convenience: the composer's routing loss needs several shapes inside one window to discriminate between, and on a grouped corpus the stage collapses at loss 10.08 against 0.5153 with identical specialists.

178 held-out probes, tool-choice exact match, greedy decoding. The dense baseline is a 17 M monolith trained on the same 700,000 sessions under an 8,192-entry BPE fitted on the task corpus, so the content is matched and the tokenizers are not.

The deficit is one failure mode rather than a spread. XE4-S emits no call at all on 50 probes and the wrong tool on 25; the baseline emits no call on 0 and the wrong tool on 31. On wrong-tool confusions XE4-S is ahead of the dense baseline. The entire deficit is failing to act.

Four causes were tested and rejected before the real one surfaced: not the tokenizer, not the prompt format, not the frozen trunk, and not dilution by the abstaining tail. What remains is where the composer puts its mass. On the 128 probes it gets right, 0.883 of the mixture sits on specialists that would open a call. On the 50 it fails, 0.285. Of those 50, seventeen put top weight on blunt_answer and fifteen put it on memory_read, which is the right shape and fails anyway.

The corpus then routes the four held-out syntactic frames straight at the specialists that abstain. Reported is 100% memory_read, oblique is 37% calendar_query and 28% memory_read, cleft is 38% memory_read. Routing a probe correctly hands it to a specialist that ends the turn six times in ten, which is why 15 of the 50 failures route to the right shape and fail regardless.

A benchmark that measured a tokenizer

Every accuracy figure in the project before this one was tool-choice exact match, and nothing had ever scored the arguments. Two live outputs make the gap concrete. Asked for a bin day, the checkpoint produces memory_get with the key user.allotments. Asked for the opening hours of a leisure centre, it produces web_search with the query "the opening hours of the the stonework centre". Right tool both times, useless both times, and counted as successes by every number the project had published.

The cause is subword length and whose tokenizer had seen the probe generator. Our BPE was fitted on 2.11 B tokens of real English, which contains no "Quillwick", so the generator's inventions shatter into five rare pieces where the baseline's own BPE encodes them in two. Copying two tokens is a different task from reproducing five rare subwords in exact sequence. On real text our tokenizer is the better of the two, 36 tokens against 44 for ten real entities, and worse only on the invented ones, 37 against 28. The paper calls this its most important methodological finding: a benchmark generated by the same machinery that produced one model's tokenizer measured that tokenizer. The tokenizer is not being rebuilt to match the generator, because that trades real-world quality for a benchmark number.

Stored memory does not change behaviour

The memory-contingent benchmark exists to separate a model with state from a model without one. Two of its arms have the fact outside the 512-token window and one never stored it at all, so the visible context is identical. A model whose only state is its context must answer both alike, right on one and wrong on the other, which is 50% by construction at any size and any training budget.

The likely cause is not a defect in the mechanism. Every memory_read session in the corpus has the fact present in context and the tool result hands the value back, so calling memory is never wrong and never informative. The model learned that a question about a personal fact calls memory_get, as a property of the phrasing. It was never given a case where the same phrasing needed a different answer depending on what was stored. The fact is written, the tree is navigable, the key-value path is live, and none of it reaches the decision.

Growth by use moves nothing

The use loop is the architecture's central claim, that serving a user improves the model. Two arms ran over an identical interaction stream under the same seed, one writing stated facts to memory and nudging the specialist that served each turn, the other doing neither. No gradient, no optimiser, no training step in either. The mechanism does run and had never run before: the use bias was 0 of 31 nonzero in every checkpoint in the project and after 20 interactions it was 17 of 31, with memory growing from 6,752 to 6,753 nodes. Over 300 interactions, held-out accuracy in both arms sat at 0.5167 at every checkpoint measured, at 0, 75, 150, 225 and 300. The live arm moved +0.0 against the frozen one.

That is a real null on the thing the architecture is for. It is also the one test a dense transformer cannot produce a difference on, since its behaviour on turn one and turn ten thousand is identical by construction, so the test is worth keeping and re-running against a model whose read path works.

The acceptance test: three of seven

Composition is worth something on the opening decision. Over the 178 probes, composed across all 31 specialists opens a tool call 71.9% of the time, the mean trained single path 43.1%, and an untrained specialist 36.9%. An untrained specialist is a residual block at initialisation and therefore approximately the trunk's own prediction, which makes that last row a trunk control rather than a weak expert. Where routing is correct, composition also beats the best single specialist: on Home, the one knowledge branch the companion run's composer could route, composed scores 5.0303 against best-single 5.1764, a gain of +0.1461 nats that a top-1 gate cannot reach by construction.

The bank is differentiated. Across twelve knowledge branches, own-branch mean loss is 5.4786 against 5.8999 on every other branch, a gap of +0.4212 nats, and 12 of 12 branches are best served by their own specialist. The task-axis equivalents, a +1.81 nat own-against-other gap and composition beating the best single specialist on all 19 domains at a mean +0.91 nats, appear only in a summary with no detail table or cited artifact behind them, and the paper flags them as needing re-derivation before anyone leans on them.

The truncation defect was found because an adversarial prior-art search went looking for papers that would embarrass us. It surfaced a 2026 result arguing that most reported top-k truncation cost is a renormalisation artefact, and the search found our own bug instead.

Fixes that reliably make things worse

Most of one long day went on the assumption that the composer was at fault. Three independent attempts to improve routing each raised training routing accuracy and each lowered the benchmark. Longer plain training went 71.9 to 73.6 to 70.2 with no trend while training routing accuracy rose 0.760 to 0.795. Frame-weighted sampling went 71.9 to 69.1 to 69.1 to 67.4 to 66.9, monotonically down, while training accuracy rose to 0.791. A hidden path with real capacity took training accuracy to 0.863 and held-out opening down to 62.9%, the worst of the three. A fix that reliably makes things worse is a fix to something that was not broken. The hidden path stays in the code defaulting to absent, because a negative result that is reachable is worth more than one taken on trust.

Weighting the one position where a specialist decides whether to act 8× for 1,500 steps took the four read specialists from 23.6–40.4% opening to 84.3–98.3% in 10.7 minutes of training, and memory_read crossed 90% within 300 steps and 31 seconds. End to end that is a regression from 57.3% to 36.5%. No-call misses fell from 50 to 31 and wrong-tool misses rose from 26 to 82. The specialists learned to open without learning what to open with, because the weight sat on the opening token and left the tool-name token immediately after it at 1. Silent abstention was converted into confident wrong answers.

Volume was measured directly rather than assumed, at eight times the stage C allocation on one read specialist. Right-tool rate goes 28.3% at stage C only, 53.3% after 3,000 additional steps, then oscillates between 35.0% and 53.3% for the next 21,000 while training loss moves 0.1954 to 0.1831. Doubling stage C is worth about +25 points and everything past that is noise. The specialists were undertrained by roughly 2×, not by the 10× the token-per-parameter ratio implies. Volume is not the reason held-out phrasing fails: a module that has converged on its own distribution and still cannot handle unseen phrasing is reporting that the distribution lacks the phrasing.

Loading a checkpoint without strict key matching drops an entry that has no destination in a freshly built model and says nothing. Python-side collections never enter the state dict at all. Nothing errors, and the numbers quietly mean something else. The same class produced four separate bugs in one system.

Every accuracy figure in the paper other than the argument scores is tool-choice exact match. Models were compared like for like, so the gap to the dense baseline is real, but the absolute numbers overstate how usable the model is by an amount nobody has established. The context window is 512 tokens and cannot be widened without retraining. Multi-turn conversation has never been run against a trained checkpoint. Serving is 4.6 tok/s.

The bare contrast between a gate and a composer is not a novelty statement. Retrieval-conditioned routing already exists in the literature; what survives here is that the retrieved object is runtime-written user content rather than a cached routing decision, and that the output is dense rather than sparse. Corpus-label routing with dense mixing and no balancing loss is not ours either. Dense mixtures beating sparse ones is established across at least three papers, and our own measurement of it turned out to be a bug. Gradient-free retraction of one named memory has a two-year-old answer in Larimar. Admitting a new expert into a trained bank by training only the router is a promotion stage in everything but name and is published; what is ours there is the 10⁻⁵ measurement that says what happens without one.

Two measurements come from neighbouring runs rather than from the evaluated checkpoint and are labelled where they appear: every knowledge-axis number comes from a companion run with a knowledge corpus, and the volume sweep was run on a later checkpoint of the same specialist geometry. The memory-contingent margin is measured through a read path that opens on 40.4% of held-out phrasings and has to be rescored once that path works. The curation result buys structure and not yet retrieval cost, because descent visits moved by under one node and both arms saturate the probe at 100% recall.

The training corpus is template-generated, the specialists fit it closely, and what they learned was the templates rather than the behaviours. Four of the five remaining failures are things the training data never required. Copying, because every slot has a typical filler and learning the filler scores as well as copying the entity. Consulting, because every read session sets the fact earlier in the same session and the tool result hands the value back. The hard frames, which sit at 0.09 to 0.30% of user turns and are among the construction families that do not transfer from any other. And saying more, because template answers are short and repetitive, so repetition is what the loss rewards. The fifth failure, 4.6 tok/s, is not a data problem at all: it is a verified 11.5× speed-up that nothing imports.

The architecture has not been shown to fail. The memory mechanism is built and separately verified, the growth points are measurably safe, and the specialists are measurably differentiated. What is missing is training that makes using any of it necessary, and the evidence for that is the same measurement in three places: behaviour identical across memory states, a use loop that moves nothing, and arguments that are never copied because the corpus never required copying.

So the next corpus is built so the easy strategy fails, not larger. Stage C goes to 6,000 steps, because doubling it is worth about +25 points on right-tool rate and the current 1,200 to 3,000 was never anything but a default. The batched bank gets imported. Arguments become a standing metric, because the distance between the right tool and an answer that works was invisible for the life of the project. The knowledge axis gets a stage E budget that matches the decision it makes, since it is the one place the collapse has not been cleared. And nothing gets claimed before it is replicated; the ordering result is the cautionary case, four seeds and a mean of −0.93 quoted for months as +5.3.

One structural item on that list is not a corpus fix. Nothing shared sits between the sum and the retrieval stage: thirty-one hidden states are added and decoded with no layer in between to reconcile them, so a weighted sum of vectors meaning "call a tool" and vectors meaning "end the turn" is a point between them rather than a vote. Blending after the head instead, scored as a rescore with the same weights and never as a trained configuration, recovers 8 of the 50 failures and loses 2 of the 128 successes, a net +3.4 points. Attaching such a stack to a trained checkpoint is verified to be exactly the identity, so it can be added under the same growth discipline as everything else. It has never been trained here, and nothing in the paper depends on it.

Section 12 of the paper lists the script behind each measurement, one script per question, from the headline accuracy figures and the memory ablation through the truncation and temperature sweeps to the growth-by-use arms and the seed sweep. Of the systems in this neighbourhood that actually train a model rather than wrapping a commercial API, the paper notes that exactly one published weights.