SOPHIA XT
In build

Read the whole document
on the phone. Copy from it.

SRW Pocket is the Sophia Reader-Writer. A large reader takes in up to 35K tokens of context once. A small writer produces the answer and can copy tokens only from spans of that context. The whole model is 431M parameters and fits in 291 MB on a phone with int4 experts.

Figure 1 A writer that can only copy what it was given Illustrative
Context, up to 35K tokens

The lease starts on 1 March 2027 and runs for 36 months. Rent is $4,850 a month, due on the first. The deposit is held by Harlow Property Group.

Answer

The lease runs

last token: generated
The document and answer are an invented example. The structure is the design: a reader pass over the context, a writer with cross-attention to it, and a copy head that can point only at context spans.
431Mtotal parameters
16reader layers, mixture of experts
4writer layers, dense
291 MBon a phone, with int4 experts
35Ktokens of context
The design

Most of the model
is the reader.

The reader is a 16-layer mixture of experts, and its experts are stored at int4, which is how the phone footprint reaches 291 MB. The writer is four dense layers with cross-attention into the reader's output, so it writes with the document in view and does not carry it.

The copy head is restricted to context spans. A name, a date or an amount in the answer is either generated from the vocabulary or copied, and a copy can only point at a span that exists in the context.

SRW Pocket is in build. There are no benchmark numbers for it on this site yet, and none will be printed before there is a run to point at.