Byte-Level Language Modeling Without a Tokenizer
Why our Veritate models read raw bytes instead of tokens, what the 256-symbol vocabulary costs, and what our 85M and 800M models tell us about the trade.
Why our Veritate models read raw bytes instead of tokens, what the 256-symbol vocabulary costs, and what our 85M and 800M models tell us about the trade.
On saturated consumer hardware, you do not buy small-model quality with a faster framework. You buy it with better data and a better training objective, with numbers to back it.
You cannot retrofit activation sparsity onto a finished model, but a short, cheap retrain installs it, and the savings compose with weight pruning.
Years of building infrastructure taught me that trust is earned quietly, honesty about limits beats big promises, and locking people in is a slow death.
The AI energy cost is treated as an inconsequential while the power demand is at the size of some countries and why efficiency, not scale, is the future.
Speculative decoding and multi-token prediction cut byte-level generation latency without touching output quality. Here is what we measured on Veritate's models
Veritate is a research program building energy-efficient language models that train and run on consumer hardware, specifically a single Apple M3 Ultra workstation. The most consequential architectural decision we made is also the most contrarian: we threw away the tokenizer. Veritate models read and write raw bytes. The vocabulary is exactly 256 symbols, one per possible byte value. There is no byte-pair-encoding table, no subword merge rules, and no out-of-vocabulary problem to manage. This article lays out why we made that choice, what it costs, and what the measured behavior of our 85M and 800M models tells us about whether the trade is worth it.
A tokenizer compresses text into a smaller number of subword units before the model ever sees it. A byte-pair-encoding vocabulary of, say, fifty thousand tokens lets a model cover an average English word in roughly one to two steps instead of five or six. That compression is the whole point: fewer steps per sentence means shorter sequences, and self-attention cost grows with the square of sequence length, so shorter sequences are cheaper.
The liability is that the tokenizer is a separate, frozen, hand-tuned artifact that sits between the model and reality. It bakes in assumptions about which languages matter, fragments rare words and code in awkward ways, and is brittle to noise and spelling. The public literature has been circling this problem for years. ByT5 (Xue et al.) showed token-free byte-to-byte models are markedly more robust to noise and to spelling-sensitive tasks. MambaByte (Wang et al.) and the Byte Latent Transformer (Pagnoni et al.) are recent attempts to make byte-level modeling competitive at scale. The fixed 256-symbol vocabulary is language-agnostic by construction: every UTF-8 stream on earth is in distribution.
The honest downside is sequence length. Bytes are roughly four times as many positions per character of meaning as a typical subword token. A byte model pays that factor everywhere context is consumed: prefill compute, attention cost, and key-value cache size all scale with it. We call this the context tax, and we do not pretend it away. It is the single biggest known cost of the byte-level bet, and several of our active research directions exist specifically to claw it back (prompt compression and prefix-injection retrieval among them).
First, the per-byte uncertainty distribution is heavy-tailed and very low on average. On our trained 85M model, median next-byte entropy is 0.67 bits. About 58.5 percent of bytes carry less than one bit of uncertainty, 78.3 percent carry less than two bits, and 98.8 percent carry less than four bits. The top 36 percent of byte positions hold roughly 80 percent of the total uncertainty mass. The practical reading: most bytes are nearly free to predict, and the expensive work concentrates in a minority of positions. That structure is exactly what you want if you intend to spend compute proportionally to difficulty, which is a separate line of our work.
Below roughly four bits per weight, post-hoc QAT on byte-level models is catastrophic, and quantization-aware training recovers nearly all of the lost quality
Second, byte-level models at small scale have failure modes we had to find the hard way. A streaming-attention-sink scheme (sliding window plus a sink token) collapsed into repetitive byte loops at sequence lengths of 256 to 512 bytes on our 85M, under both rotary and absolute position encodings. The published intuition that attention sinks emerge cleanly only appears to hold at much larger scale; at 85M it did not, and we logged it as a refuted approach pending a retrained billion-parameter model. We also found that a naive retrieval scheme that mixed an n-gram corpus prior into the model logits at decode time degraded quality (a roughly 36 percent perplexity regression on the 85M) and induced looping, because a static corpus prior overrides the model's own plan. Retrieval has to be fused as input context, not as a logit bias.
These results are at small scale. The 85M model is a research vehicle, not a product, and single-scale findings can flip with width and depth. The attention-sink failure in particular has an explicit retry condition (a trained model at one billion parameters or more) and should not be read as a permanent verdict against the technique. The entropy distribution is measured on one checkpoint and one data distribution; it is suggestive of an exploitable structure, not a proof that adaptive-compute schemes will pay off. And the context tax remains an open, quantified liability rather than a solved one. What we can say with confidence is that a 256-symbol byte vocabulary trains stably, produces coherent English, and exposes a uniquely clean substrate for the energy-efficiency techniques the rest of this series describes.
The companion pieces on quantization-aware training and activation sparsity take up those techniques in turn. To see the engine itself and the live training and inference stats behind these numbers, visit Veritate.
Xue et al., ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models, 2021.
Wang et al., MambaByte: Token-free Selective State Space Model, 2024.
Pagnoni et al., Byte Latent Transformer: Patches Scale Better Than Tokens, 2024.