Sophia-XE4

A from-scratch language model that grows its own experts.

SOPHIA XT ships systems that take payment today. A live grounded-search system, a commercial real estate platform priced in three tiers, and a field-service diagnostics product over a multi-million-part corpus all run now, and the search product is the largest consumer of the memory layer described on this page. Sophia-XE4 is the model that makes work like that cheap to keep current.

XE4 is our own weights, trained from zero. It is a block-diffusion language model whose compositional machinery is inside the network from the first pretraining token rather than bolted onto somebody else's checkpoint. The router sits at four layers. Fold memory cells sit at three more. An anchor head predicts the positions that must survive generation intact. All of it trains together.

The result is a model that adds a capability by growing a new expert rather than by being retrained, and that holds its memory footprint flat no matter how much it knows.

This page carries two kinds of thing and they are not the same kind of thing. One is work that has run, on one consumer GPU, with the method that produced each number stated beside it. The other is a training plan with prices on it. Every figure, panel and mixed table row on this page is tagged with which it is, and the tags mean the same thing everywhere they appear.

Measured is ours and it ran. Published is ours, ran, and is externally citable. Computed is arithmetic from a measured input, shown so it can be checked. Cited is published work by other people, adopted as published. Proposed has not been built. Open is a gap we have measured and not closed. Retired is a mechanism we killed, with the evidence that killed it.

Every compositional system shipping today sits on top of a frozen model from someone else, with tuned weights and a retrieval layer stapled around it. That works, and we built one. It serves live traffic. But the model underneath was never trained to be composed, so the composition fights it. Routing has to infer structure the network was never asked to represent. Memory has to be injected through a prompt the network was never trained to read as memory.

XE4 is the version where that structure is native. It is a block-diffusion language model trained from zero on our own architecture, with four components that do not exist in any other pretrained model.

• A sparse expert router at layers 3, 7, 11 and 15, trained jointly with the backbone rather than fitted afterwards.
• Fold memory cells at layers 5, 10 and 15. Coordinate-addressed memory that is part of the language model rather than a database it queries.
• An anchor head, a separate classifier predicting which positions carry identifiers, codes and rare terms that must survive generation exactly.
• A confidence-hysteresis sampler, which commits a token above 0.85, carries it forward below 0.30, and iterates the rest.

Block diffusion is the substrate that makes those four possible. It generates a block at a time rather than a token at a time, some positions can be held fixed while others denoise, and decoding parallelises. An autoregressive model cannot hold an anchor fixed and refine around it. This one is built to.

Everything on this page was measured on this

XE4 is specified to the layer. The backbone is conventional and deliberately so, because the novelty budget is spent on the compositional layers rather than on re-litigating transformer basics.

Parameter count is the wrong measure, and we are not going to use it

Asking how many parameters XE4 has is like asking how many pages a library has. The backbone is fixed. Experts accumulate over the system's life and are not present at training time. Memory is on disk and unbounded. A single number cannot describe that, and the number the industry uses, total parameters as a proxy for capability, describes a dense autoregressive model and nothing else.

XE4 is specified by three numbers instead of one, plus the only one that governs cost.

So the model line is not named by size. It is written as backbone, experts, memory. A production system might read 1.7B backbone, 512 experts, 100M records, and the number that governs its serving cost is 1.72B whatever the other two say. Where a comparison against a dense model is unavoidable, we compare on active parameters and on cost per query, because those are the quantities a buyer is billed for.

The component list says what is in the model. This says what happens when you ask it something, because that is where the difference lives.

The three things that are genuinely different

Memory arrives through attention, not through the prompt. Retrieval-augmented systems paste retrieved text into the context window, which means retrieved knowledge competes with the conversation for the same scarce budget and the model reads it as ordinary input. XE4's fold cells address the store by coordinate at three depths and inject what they find as auxiliary key and value slots. The model attends over retrieved knowledge without spending context on it, and because the cells were trained alongside the backbone, the network learned what to do with that channel rather than having it introduced at inference.

Routing happens at depth, and more than once. A mixture-of-experts gate routes tokens between feed-forward blocks inside a single monolithic model. XE4 routes the whole query across separately trained, separately shippable experts, and it re-evaluates at layers 7, 11 and 15. A question whose subject only becomes clear halfway through the stack gets the right expert by layer 11 even if layer 3 guessed wrong. That is why top-3 accuracy of 0.983 matters so much: the right expert is nearly always in contention, and depth gives the model three more chances to land on it.

Positions can be held fixed while the rest of the answer is refined. An autoregressive model commits tokens left to right and cannot revise. Block diffusion generates a block at a time over roughly a dozen denoising steps, so a part number, a case citation, a dosage or an account identifier can be locked in place while the language around it is rewritten. Anchor preservation measured exactly 1.0 on our diffusion backend. The anchor head was trained to predict which positions deserve locking, which is why the model knows a part number when it sees one.

We compete on operating-model economics, not leaderboard position. The table below is built on the axes that decide whether a deployed model can be maintained, because that is where the category is contestable.

The row that decides it is the last one. Every other approach needs the provider in the loop when capability changes, because something learned has to be updated: a checkpoint, a gate, a routing model. XE4's router is closed-form, so a new expert is additive by construction. That turns model maintenance from a release event into a service cadence, and it is the difference between shipping a fix in an afternoon and shipping one next quarter.

The honest row

Our own first attempt belongs in that table. We framed the compositional vision as retrieval of documents and it did not produce a deployable architecture, because routing to documents is not routing to capability. The runtime piece survived as an open-source library whose suite collects 259 tests, and the lesson produced XE4. Investors should know we have already been wrong about this once, at our own expense, and corrected.

Every language model shipping today is frozen at its training cutoff. It cannot learn from what it does. Every agent framework built on top is scaffolding around a static mind: memory bolted on as a database, tools bolted on as function calls, and underneath, weights that will never change in response to anything the system experiences. That is the gap between a very capable function and something that gets better by running.

Closing it needs three things, and XE4 is built around exactly those three.

Put together, they describe a model that accumulates capability across its operational lifetime instead of being fixed at a cutoff date. It reads something it has never seen, finds no expert covers it, accumulates enough of that ground to be worth a specialist, trains one in about a minute, attaches it, and is measurably better at that thing than it was that morning, with every step of that recorded in an audit log and nothing it already knew put at risk.

That would be a real milestone, and we have not identified another system demonstrating this complete combination under the constraints described here. That sentence is deliberately narrow. Continual learning, parameter isolation, mixture-of-experts growth and memory-augmented transformers are all active literatures with real results in them, and every component here stands on one of them. The prior-art table above says which, and where this departs. What we have not found is the whole combination in one system, and that is a claim about our search rather than about the literature.

It is also not general intelligence, and we will not tell a room that it is. XE4 does not address reasoning depth, world modelling, or goal-directedness. Those are separate problems and a routing architecture does not solve them. What it addresses is the substrate, the part that makes continual, auditable, non-destructive learning mechanically possible at all. Every system that eventually gets to general capability will need something that does this job, and we would rather be measured on that narrow claim than on a bigger one we cannot defend.

The hard limit on every deployed language model is that memory costs grow with use. Stuff the history back into the prompt and tokens grow every turn until you hit the wall. Load a vector index into RAM and the machine grows with the corpus. Both are the same failure: state that scales with what you know.

XE4 breaks that coupling inside the network. Fold memory cells pull compact representations at three layers, cache them, and re-inject them into the next layer's attention as auxiliary key and value slots, so the growing tail of the KV cache is replaced by a fixed-size content-addressable buffer. The retrieval substrate underneath is already proven RAM-bounded, and it is the cleanest result we own.

Above the network, the same layer retrieves compressed grounded evidence rather than concatenating passages. Measured over 47 grounded probes at matched depth: 1.82 times fewer context tokens at equal grounding, and a retrieval hit becomes a correct answer 90.9% of the time against 63.2% for passage stuffing.

A compositional model is only as good as its ability to pick the right expert, and the received wisdom is that a fixed lookup cannot keep up as experts multiply, so you need a learned gate. We tested that across 13 expert counts from 4 to 53, on 14 methods, with 500-resample confidence intervals on every point.

There is no gate to retrain, which is what makes a growth loop affordable, because adding an expert costs one extraction pass and nothing else. The accuracy ceiling also belongs to the method rather than to the backbone: we swept five frozen bases across a 9 times size range and three model families, and routing sat inside a 1.7-point band. Scale does not buy routing. That result cancelled a funded larger-backbone run, and it is why the budget in section 15 spends where it does.

A mixture-of-experts model fixes its expert count at training time and routes tokens among them with a learned gate. XE4's expert count is a runtime variable and the router is closed-form, so the graph can change shape while the model is serving.

The structural property this needs is already built and tested. An expert node may itself contain a child graph, with the same routing mathematics at every level. In the stacked test, a two-level graph routed correctly at the coarse level and again at the fine level, matching the flat graph's score on identical probes. Recursion costs nothing in accuracy. What XE4 adds is the policy that decides when to recurse, and that policy is designed rather than running.

Figure 07. Three operations on the expert graph, driven by telemetry the router already emits at 0.17 ms per query. Every split, merge and germination is written to a hash-chained audit log, so the model's capability history stays reconstructable, which is what makes a self-modifying model auditable enough to deploy where it matters. The recursion these run on is measured. The operations themselves are the policy stage 3 funds.

Why this is a different object from mixture-of-experts

Experts inside a mixture model are trained jointly and cannot be shipped, audited or revoked one at a time. XE4 experts are separate 22.4 MB artifacts, each with its own scope, exclusion list and validation set, so one can be sold to a customer, shown to a regulator, or pulled without touching the rest. A mixture gate is learned parameters that must be retrained whenever the expert set changes, which is exactly the cost that makes continuous growth impossible. Ours is closed-form and costs 0.17 ms, so growth is affordable by construction.

Capital discipline: what we killed, and what killing it saved

Four of these were pre-registered, so the decision rule was committed in writing before the run and the outcome could not be argued with afterwards. This habit is why the budget in section 15 is a fraction of what a from-scratch model line normally costs.

Open, and this is what the money buys

The five questions below are unresolved. Each has a gate written before the work starts, and each gate can stop the program. Listing them is not modesty. A reader who checks this page against the sources should find the same list.

Three data tracks, each serving a different part of the architecture. The pretraining corpus is open and standard. We are not making a data-moat claim, and pretending otherwise would not survive diligence. The moat is in the second and third tracks, which nobody else builds because nobody else has the architecture that consumes them.

The second track is the unusual one

Ordinary pretraining teaches a model the statistics of word sequences. The attribute track teaches it the statistics of property adjacency. Each sample opens with a class, then walks its typed properties in an order determined by structural relatedness rather than by how the text happened to be written. The model learns which properties imply which other properties, which is the representation the router and the fold cells both need.

A working corpus exists: real records, a closed property vocabulary of 115 definitions rather than open subword tokens, and validation confirming the ordering is genuinely structural and not chronological order in disguise. Stage 2 scales it and tests it head to head against the same facts written as ordinary prose at matched compute. That comparison is the cheapest decisive experiment in the program, and two of our strongest confirmed results are already effects of how the sample was built rather than of how much compute was bought.

Per-expert data is the commercial asset

An expert needs 50 to 150 examples. That is small enough that a customer's own documents, tickets, manuals or case files are sufficient, and small enough that a vertical can be covered in an afternoon. Every expert trained for a customer stays theirs. The compounding asset is not a giant proprietary corpus. It is the pipeline that turns any customer's messy domain material into a routed expert for about fifty cents.

We do not currently publish standard benchmark scores, because the models trained to date were research substrates at 360M and 1.3B and a score there would tell a reader nothing useful. Rather than leave that silence, here is the commitment.

Two things we will report that nobody currently measures, because the standard suites were built for static models and cannot see what XE4 is for. Capability retention: after adding twenty experts, how much of the original score survives, a number that should be identical to where it started since the backbone is never touched, and any deviation is a finding. Cost of acquisition: dollars and wall-clock to take the model from failing a domain to passing it, which is the number an enterprise buyer actually cares about and which has no entry on any leaderboard.

The gates in the plan are stricter than any public benchmark, because they are pre-registered and each one can stop the program. A benchmark score is a claim. A pre-registered gate is a commitment made before the result was known.

The buyer is not someone shopping for a better chatbot. It is an operator who needs a model to know their domain, keep knowing it as it changes, run where their data already is, and be explainable when something goes wrong. Frontier APIs lose on at least two of those four, every time.

XE4 does not arrive without a business around it

Most research-stage model companies have a paper and a plan. This one sits inside a shipped portfolio that already takes payment, which matters for two reasons. The verticals the model will specialise into are ones we already serve, and the company does not depend on the model shipping in order to survive the year.

Eleven systems in the portfolio are sellable today. The independently assessed replacement cost of the shipped estate is $1.1M to $2.2M, which is what someone would spend to have what already exists, ignoring the four years it took. That is the floor under this raise, before any value is assigned to the published research, the released model, or the live customer bases.

How XE4 is sold

Three motions, in the order they become available. Experts as product: a customer buys a model that already knows their vertical, priced against what a fine-tune would have cost them. The pipeline as a service: they bring documents, we return a routed expert, repeatedly, on a service cadence rather than a release cadence. The model itself: licensed for on-premise and edge deployment where nothing may leave the building. The first two open the moment routing clears its production threshold at six experts. The third arrives with the production backbone.

The honest read on revenue: the portfolio takes payment but is not yet at scale, and getting there is a selling problem rather than an engineering one. The assessment that produced the figures above is blunt about the consequence. Going from nothing to half a million in recurring revenue would roughly triple the company's valuation, and no technology built in the same period would do that. XE4 is funded on its own merits, not on the promise that revenue arrives first.

Not the backbone, which is deliberately conventional and anyone can build one. Four things are hard to copy, in increasing order of durability.

The fourth is the real one, and it is a systems moat rather than a patent moat. Growth is only affordable because routing costs 0.17 milliseconds with no parameters to retrain, because an expert costs about fifty cents, and because memory does not grow in RAM. A competitor with a learned gate cannot add an expert without a retraining pass. One with prompt-based retrieval pays context for every fact. One with co-trained experts cannot change the count at all. Each of those is a reasonable design choice that happens to foreclose the growth loop, and undoing it means rebuilding from the objective up.

What is not defensible, stated plainly: the backbone, the tokenizer, the pretraining data and block diffusion itself are all public. We use them as published and cite them. Concentrating the originality where we have evidence is a deliberate choice, because it makes the claims checkable, and checkable claims are what a diligence process can actually confirm.

Thomas Garren is the founder and principal architect of SOPHIA XT LLC, founded 2024. The company is founder-funded with no outside investors and no instruments outstanding. Every result on this page was produced by a one-person research operation on a single consumer GPU, run with pre-registration, hardware telemetry and a logged experiment harness, which is the relevant signal about how the money would be spent.

The raise funds four to six people around that: a pretraining engineer who has taken a model past a billion parameters, a data engineer for the corpus pipeline, an evaluation engineer who owns the gates, and applied engineers for the vertical experts. Hiring is staged against the plan rather than front-loaded. The decision tranche adds nobody, and the full team is only needed once the production backbone is in training.

This is not a first attempt at training a model. The line already includes a 1.3B diffusion base released under Apache-2.0 with a clean five-epoch loss curve, a 360M compositional model serving live traffic on CPU hardware today, a training objective published with a DOI, four open-source repositories, and two manuscripts with built submission packages.

Stages are named by what they settle rather than by a model size, because size is the industry's proxy metric and it is the wrong one here. Research stages are interleaved so the cheap decisive work always runs before the expensive work it would change. Hours are A100-equivalents and include a block-diffusion cost multiplier over autoregressive training. Dollars assume $2.00 per GPU-hour. Nothing in this table has been run.

Compute is the only thing standing between this architecture and a trained model, and there is a route to clearing it that does not cost equity. NSF SBIR Phase I awards up to $305,000, non-dilutive, on a dedicated AI track, for exactly this: technical feasibility work too early for venture capital and too applied for academia. The next full-proposal deadline is 4 November 2026, with a project pitch and an invitation required first.

Our profile is unusually strong there, because SBIR reviewers are academics and they reward precisely what venture capital discounts: a published preprint with a DOI, an openly released model anyone can download, pre-registered experiments with decision rules committed before the run, negative controls, and a documented record of retiring our own hypotheses on the evidence. Most applicants manufacture that credibility for the application. We have four years of it, timestamped and public. That award alone funds the decision tranche and the validation backbone outright.

Alongside it, compute-credit programmes at the major clouds and the GPU-native providers run from $5,000 to $100,000 each and take an afternoon to apply for. Compute is the binding constraint on every research question in this document, and credits are the one route where effort substitutes for capital.

What the money is buying

A model, and deliberately not a bigger one. The architecture is specified to the layer, the hardest from-scratch failure mode is diagnosed and fixed, the memory layer is measured, the routing is measured to 48 experts, the training objective is published, the recursion is tested, and a smaller model from the same line serves live traffic today at roughly a fifth of a cent per million tokens.

The reason this is cheap is the reason it is interesting. Everyone else buys capability with parameters, which is why their costs rise with every domain they cover and why a frontier run is a nine-figure commitment. We buy capability with experts at about fifty cents each, attached to a backbone we never retrain, over memory that does not grow in RAM. One to two million dollars is not a scaled-down version of somebody else's plan. It is what this architecture costs.

The sequencing matters as much as the number. A grant and a stack of compute credits get the architecture to a trained, validated model without touching the cap table. Equity is then priced against a working model rather than a design document, which is a materially better trade for everyone in the room. The next SBIR deadline is 4 November 2026, which makes the decision tranche the immediate work rather than the eventual work.