The Tokenization Trap: Domain-Adaptive BPE Restores Language-Like Token Statistics for Protein and Genome Language Models

Abstract

Protein and genome language models (pLMs/gLMs) have not shown the smooth, predictable scaling that drives progress in natural-language models. We argue a central, under-examined cause is tokenization. The near-universal choice of single-residue tokenization (one token per amino acid or nucleotide) collapses the token-frequency distribution onto the marginal residue frequencies, producing a flat, non-Zipfian spectrum (fitted exponent α ≈ 0.4-0.6). Natural language is Zipfian (α ≈ 1), and that heavy tail is exactly what neural scaling laws exploit. We formalize this as the tokenization trap and test it on real Swiss-Prot.

Domain-adaptive byte-pair encoding (BPE) moves the token-frequency exponent into the language-like band (α ≈ 1.0-1.2) with high goodness-of-fit, while single-residue tokenization and an English-trained GPT-2 BPE do not. At a fixed parameter budget, domain BPE reduces bits-per-residue by ~9% relative to single-residue tokenization, whereas GPT-2's English BPE is worst. A model-size sweep makes the trap quantitative: single-residue tokenization is nearly flat with scale (bits-per-residue slope b ≈ 0.0025), whereas domain BPE turns parameters into compression about twice as fast (b ≈ 0.0058). A frozen-feature linear probe on a composition-controlled family task shows single-residue features are near chance (0.23 vs 0.17) while domain BPE recovers motif structure (0.86).

Moving from synthetic probes to a real benchmark, we couple each tokenizer to an external 21-trait microbial-phenotype task over 364 native genomes: domain BPE restores the Zipfian exponent on real genomic DNA (α: 0.31 → 1.12), wins the matched single-nucleotide contrast at every taxonomic level, and, at ~3 M parameters, matches a 7-billion-parameter single-nucleotide genome model (Evo2). We conclude domain-specific subword tokenization is a necessary but not sufficient condition for language-like scaling: it fixes the input distribution, but the transformer still learns correlation, not mechanism, motivating a spectral, mechanism-aware architecture.

The tokenization trap

Scaling laws predict test loss as a smooth power law in parameters, data, and compute. Human language is Zipfian: a few very common tokens and a long tail of rare, information-dense ones. A protein is a string over 20 amino acids; a genome over 4 nucleotides. Under single-residue tokenization the token distribution is the marginal residue distribution: nearly flat, with no heavy tail. Our hypothesis: pLMs and gLMs scale poorly in part because single-residue tokenization yields non-Zipfian token statistics, and domain-adaptive BPE, learning merges on biological corpora so frequent motifs/k-mers become tokens, restores a language-like distribution. We contrast against the natural baseline of reusing a powerful existing LLM tokenizer (GPT-2's English BPE).

We stress that fixing tokenization is necessary but not sufficient. Even with a language-like input distribution, the transformer learns correlational structure (which residues co-occur), not the underlying biophysical mechanism (why they co-occur). The full path is a chain: (a) current pLMs/gLMs scale poorly; (b) single-residue tokenization is too flat; (c) BPE could fix this; (d) but generic (English) BPE learns no biological units; (e) so a domain-derived tokenizer is required; and (f) even then, attention is the wrong inductive bias, motivating (g) a spectral, mechanism-aware architecture. This work gives evidence for (a)-(e) and a concrete design and falsifiable test for (f)-(g).

Domain BPE restores Zipfian statistics

On real Swiss-Prot, single-residue tokenization sits at α ≈ 0.5-0.6 with a poor power-law fit. Across a vocabulary sweep, domain BPE moves the exponent into the language-like ≈ 1.0-1.2 band with high r² (≥ 0.9), a genuinely Zipfian distribution. The rank-frequency tail is flat for single residues and heavy-tailed for BPE.

Better statistics, better modelling (at fixed size)

Distribution is necessary but not sufficient, so we test modelling efficiency directly. We train an identical small Transformer ("TinyGPT") under each tokenizer, holding parameters, optimizer, steps, and data fixed, and evaluate with bits per residue, total next-token negative log-likelihood divided by the raw amino-acid characters scored, a tokenizer-agnostic compression metric. On a synthetic motif corpus with strong embedded structure, domain BPE attains the lowest bits-per-residue (3.535 at vocab 256 vs 3.869 for single residues, ≈ 9% lower), while GPT-2's English BPE is worst by a wide margin despite its large embedding table.

Why GPT-2 English BPE is special, and unstable

GPT-2's merges are frozen English byte-pairs; applied to biology they fall on arbitrary residue boundaries. The resulting token distribution is not a power law: its global fit has r² of only 0.75 (protein) and 0.49 (genome), and its local per-chunk slope (~0.9 protein, ~1.4 genome) differs sharply from its cliff-inflated global slope (~2.0). Any single "exponent" for GPT-2 is therefore fragile. We always report the fit r² so misfit rows are visible, and use a robust chunk-median estimator when r² < 0.80. The decisive evidence against GPT-2 is the modelling result, which needs no curve fit and ranks it worst: reusing a strong LLM tokenizer does not work; the merges must be learned on the target domain.

The scaling curve: single-residue tokenization barely benefits from scale

The single-point proof fixes model size; the central claim is about scaling. We train each tokenizer at four TinyGPT sizes on real Swiss-Prot and fit a power law bpr = a·N^−b over non-embedding parameters N. Single-residue tokenization is essentially flat (b = 0.0025, r² = 0.99): growing the model 63× reduces bits-per-residue by barely 1%. Domain BPE scales about twice as fast (b = 0.0058 at vocab 256) and narrows its gap to single-AA monotonically. GPT-2's English BPE has the steepest slope only because it starts catastrophically high and is digging out of a hole; it stays worst at every size.

We are honest about the caveat rather than overclaiming a crossover: at this small-scale, step-matched budget (< 2M params, 150 steps), single-residue tokenization still has the lowest absolute bpr, because shorter BPE token streams get less gradient at fixed steps. The load-bearing signal is the slope: single residues convert scale into compression ~2× more slowly than domain BPE, and the curves are converging. A compute/token-matched sweep at larger scale is the natural confirmation.

The downstream probe: single-residue features lose motif structure

Compression is one axis; we also want a functional signal. We build a controlled protein-family task where each of six families is defined by motifs that are anagrams of a shared residue multiset embedded in a common random background, so every family has a statistically identical amino-acid composition. Only an order-sensitive (subword) representation can separate them. We train each tokenizer's TinyGPT unsupervised, freeze it, mean-pool the final hidden states, and fit a logistic-regression probe. Single-residue features are near chance (0.233 vs chance 0.167); every subword tokenizer recovers the structure, and domain BPE does so with a 256-token vocabulary (0.863).

Interestingly, GPT-2's English BPE scores highest on this probe (0.937) even though it is the worst generative compressor. The results are complementary, not contradictory: GPT-2's 50k-token vocabulary exposes arbitrary orderings to a linear classifier (presence detection), but those misaligned merges make it a poor next-token model of biology. Domain BPE is the only tokenizer strong on both axes with a tiny vocabulary, exactly the profile a scalable foundation model needs.

Real genomic benchmark: microbial trait prediction

The composition-controlled probe isolates order/motif sensitivity by construction, but does the tokenization advantage survive on real genomes and a real functional task? We couple our tokenizers to an external benchmark, microbe-foundation, which predicts 21 heterogeneous microbial traits (gram stain, oxygen tolerance, motility, pathogenicity, cultivation medium, and more) from a single frozen feature vector per genome, under strict taxonomic holdouts at the species, genus, and family levels.

We fetch 364 bacterial genomes (BacDive/NCBI) and cache one normalized DNA string per genome, so every tokenizer sees byte-identical input. Under each tokenizer we train the identical TinyGPT, freeze it, and mean-pool final hidden states into one feature vector per genome. These vectors are scored by microbe-foundation's 21-head trait model across species/genus/family splits and seeds {0, 1, 2}. As an upper reference we add Evo2-7B (a single-nucleotide StripedHyena-2 gLM pretrained on 8.8 T tokens).

On native genomic DNA (no reverse translation), the token-distribution exponent reproduces cleanly: single-nucleotide tokenization is flat (α = 0.31), fixed 4-mers sit in between (α = 0.50), and domain BPE lands in the language-like band (α = 1.12 at vocabulary 1024, 4.29 nt/token). Domain BPE attains the best or tied-best score at every taxonomic level and beats the single-nucleotide control on all three splits. Notably, a ~3 M-parameter domain-BPE TinyGPT matches or exceeds the 7-billion-parameter Evo2 reference (0.568 vs 0.560 at the species split): how the genome is tokenized can outweigh three orders of magnitude of scale spent on single nucleotides.

We deliberately do not overclaim: the downstream gaps are small relative to seed variance, and pooling Evo2's per-nucleotide states along BPE spans does not help a model already trained per nucleotide; tokenization must be present at training time, not retrofitted at read-out. The load-bearing signals are qualitative: (i) Zipfian restoration holds on real genomic DNA, (ii) domain BPE wins the matched contrast at every taxonomic level, and (iii) a tiny domain-BPE model is competitive with a 7B single-nucleotide foundation model.

The second bottleneck: attention is the wrong inductive bias

Fixing tokenization removes the input-side trap but leaves a deeper problem. A transformer trained by next-token prediction learns which tokens co-occur; it has no built-in notion of why. In biology the "why" is mechanistic: residues are coupled through a sparse, low-rank physical interaction operator (the same structure that Potts/direct-coupling models and pLM attention maps partially recover). Vanilla softmax attention can approximate this operator but is not biased toward it, so it spends capacity memorizing surface correlations and scales inefficiently even with good tokens.

We propose Spectral Composition Attention as a follow-up: let C be the empirical token-coupling (pointwise mutual information) operator over the domain BPE vocabulary; its spectrum is sharply concentrated. The design (i) biases attention logits toward the leading eigenmodes of C (a mechanism prior rather than a free QKᵀ surface), and (ii) maintains a multi-scale residual stream pooled along BPE merge boundaries so the residue→motif→domain hierarchy is explicit. This is testable in exactly this paper's harness: fix the domain BPE tokenizer, swap only the attention block, and compare bits-per-residue and probe accuracy at matched parameters, with the prediction that the spectral bias, not added parameters, drives the gain.

What it means

Single-residue tokenization imposes a non-Zipfian token distribution that starves the scaling pressure language models exploit: the tokenization trap. Domain-adaptive BPE escapes it: genuinely Zipfian, high-r² statistics; ~2× faster conversion of scale into compression; and features that recover motif structure single-residue features cannot. An English-trained BPE does none of this for modelling. These claims, established intrinsically and on a controlled probe, now also hold on a real 21-trait microbial benchmark over native genomes, where a ~3 M-parameter domain-BPE model rivals a 7B single-nucleotide gLM.

Tokenization is a first-class design axis for biological sequence models, but it is only the first of two bottlenecks. A correct input distribution still feeds an architecture that learns correlation, not mechanism. Unlocking LLM-style scaling for biology will require pairing a domain-derived tokenizer with a domain-derived, spectral, mechanism-aware architecture.

Limitations

All modelling experiments are deliberately small-scale (< 2M non-embedding params, a few hundred steps) to isolate the tokenizer variable; they establish slopes and orderings, not absolute scaling exponents at foundation-model scale. The scaling curve is step-matched rather than compute/token-matched, which disadvantages BPE's shorter token streams. The composition-controlled probe is synthetic, not a natural functional benchmark. The distributional genome corpus uses reverse-translated proteins; §4.6 removes that caveat but at 364 genomes the downstream gaps are small relative to seed variance. A compute-matched run over the full BacDive corpus is the natural confirmation.

Keywords: tokenization · byte-pair encoding · Zipf's law · scaling laws · protein language models · genome language models · microbial trait prediction · Evo2 · bits per residue · spectral attention

@article{horiuchi2026tokenization,
  title   = {The Tokenization Trap: Domain-Adaptive BPE Restores Language-Like
             Token Statistics for Protein and Genome Language Models},
  author  = {Horiuchi, Miyu},
  year    = {2026},
  note    = {replicater.xyz/writing/tokenization-trap}
}