Byte-level models, with no tokenizer in the way
Most models cut text into pieces from a dictionary they learned once, and whatever is not in that dictionary comes out mangled. Ours read the raw bytes instead, the same 256 values every computer already uses, so any language, any code, and any file arrive the same way.
Why the dictionary gets in the way
A tokenizer is a dictionary of common chunks, learned once from one pile of text, and everything the model reads gets cut up with it first. It works well, which is why nearly everything uses one.
The trouble shows at the edges. A name it never saw, a line of code, a language that was thin in the original pile, all of that gets shredded into scraps and comes out worse for it.
It is also why so many models cannot count the letters in a word. They never see letters, only the chunks the dictionary handed them.
- No dictionary to build, ship, or keep in sync
- Nothing is out of vocabulary, every byte has a symbol
- Any language, pointed at directly
- Code, logs, and files read the same as prose
The trade
A dictionary packs several letters into one symbol, bytes do not, so the same paragraph is a longer stretch to read. The trunk of our model is recurrent, which means a longer stretch costs far less here than it would in a stack of attention blocks.
At this size, not having a dictionary to train, ship, and keep in sync is worth more than the extra length. Being able to point the model at any pile of text without touching anything is worth more still.
Where this goes
Reading bytes means the model can be pointed at anything. A codebase, a language nobody trained for, a file format it has never seen, it all arrives as the same stream and the model reads it the same way.
Nothing has to be built for your text first, you hand it what you have and it reads it.
The engine and the training scripts are published, so you can point one at your own material on your own machine and see how it reads.
- Any written language, including ones it was never tuned for
- Source code, configuration, and logs
- Names, addresses, and anything else a dictionary would mangle
- Its own files, byte for byte
Try it on your own text
Point it at whatever you have, code, mixed languages, messy files, it reads all of it the same way.