Byte-Level Language Modeling Without a Tokenizer

Thesis

Veritate is a research program building energy-efficient language models that train and run on consumer hardware, specifically a single Apple M3 Ultra workstation. The most consequential architectural decision we made is also the most contrarian: we threw away the tokenizer. Veritate models read and write raw bytes. The vocabulary is exactly 256 symbols, one per possible byte value. There is no byte-pair-encoding table, no subword merge rules, and no out-of-vocabulary problem to manage. This article lays out why we made that choice, what it costs, and what the measured behavior of our 85M and 800M models tells us about whether the trade is worth it.

Background: Why Tokenizers Exist, and Why They Are a Liability

A tokenizer compresses text into a smaller number of subword units before the model ever sees it. A byte-pair-encoding vocabulary of, say, fifty thousand tokens lets a model cover an average English word in roughly one to two steps instead of five or six. That compression is the whole point: fewer steps per sentence means shorter sequences, and self-attention cost grows with the square of sequence length, so shorter sequences are cheaper.

The liability is that the tokenizer is a separate, frozen, hand-tuned artifact that sits between the model and reality. It bakes in assumptions about which languages matter, fragments rare words and code in awkward ways, and is brittle to noise and spelling. The public literature has been circling this problem for years. ByT5 (Xue et al.) showed token-free byte-to-byte models are markedly more robust to noise and to spelling-sensitive tasks. MambaByte (Wang et al.) and the Byte Latent Transformer (Pagnoni et al.) are recent attempts to make byte-level modeling competitive at scale. The fixed 256-symbol vocabulary is language-agnostic by construction: every UTF-8 stream on earth is in distribution.

The Cost: The Context Tax

The honest downside is sequence length. Bytes are roughly four times as many positions per character of meaning as a typical subword token. A byte model pays that factor everywhere context is consumed: prefill compute, attention cost, and key-value cache size all scale with it. We call this the context tax, and we do not pretend it away. It is the single biggest known cost of the byte-level bet, and several of our active research directions exist specifically to claw it back (prompt compression and prefix-injection retrieval among them).

What We Measured

First, the per-byte uncertainty distribution is heavy-tailed and very low on average. On our trained 85M model, median next-byte entropy is 0.67 bits. About 58.5 percent of bytes carry less than one bit of uncertainty, 78.3 percent carry less than two bits, and 98.8 percent carry less than four bits. The top 36 percent of byte positions hold roughly 80 percent of the total uncertainty mass. The practical reading: most bytes are nearly free to predict, and the expensive work concentrates in a minority of positions. That structure is exactly what you want if you intend to spend compute proportionally to difficulty, which is a separate line of our work.