Benchmarks
Fewer tokens at every vocabulary size
Bytes per token on 41 held-out test sets, against 31 published tokenizers. Akshara is shown at nine sizes, each the first merges of one training run, so every comparison can be made at a matched vocabulary.
Score against vocabulary size bytes per token, higher is better
- Akshara at each size
- Published tokenizers
Matched vocabulary sizes Akshara minus the published tokenizer, bytes per token
Each published tokenizer is paired with the largest Akshara size whose vocabulary is no more than 2% larger; tokenizers with no such size are left out. Green: Akshara is ahead; red: Akshara is behind.
| Published tokenizer | Vocabulary | Akshara | Overall | English | Code avg | Languages avg |
|---|---|---|---|---|---|---|
| Phi-3.5-mini-instruct † | 32,011 | Akshara 32k | +0.76 | +0.05 | +0.38 | +1.85 |
| SmolLM2-135M † | 49,152 | Akshara 49k | +1.00 | −0.24 | +0.46 | +2.77 |
| gpt2 † | 50,257 | Akshara 49k | +1.25 | −0.35 | +1.19 | +2.92 |
| OpenAI p50k_base (Codex) † | 50,281 | Akshara 49k | +0.99 | −0.35 | +0.40 | +2.91 |
| cl100k_base (GPT-4, OLMo-2, Olmo-3, phi-4) | 100,277 | Akshara 92k | +0.85 | −0.13 | −0.09 | +2.77 |
| ERNIE-4.5-21B-A3B-PT | 101,304 | Akshara 92k | +0.70 | +0.16 | +0.41 | +1.54 |
| Hy3 | 120,818 | Akshara 120k | +0.78 | −0.05 | +0.15 | +2.24 |
| LFM3-1.7B | 125,033 | Akshara 120k | +0.53 | +0.03 | +0.25 | +1.32 |
| Llama-3.2-1B | 128,256 | Akshara 120k | +0.67 | −0.04 | −0.04 | +2.08 |
| DeepSeek-V3 | 128,815 | Akshara 120k | +0.55 | −0.08 | +0.24 | +1.48 |
| Mistral-Nemo-Base-2407 | 131,072 | Akshara 120k | +0.43 | +0.10 | +0.27 | +0.93 |
| GLM-5.3-Flash | 154,856 | Akshara 151k | +0.92 | +0.03 | +0.01 | +2.71 |
| Kimi-K3 | 163,840 | Akshara 151k | +0.84 | +0.03 | +0.06 | +2.43 |
| OpenAI o200k_base (GPT-4o) | 200,019 | Akshara 200k | +0.39 | +0.05 | +0.10 | +1.02 |
| Mistral-Large-4-1T-A52B | 204,800 | Akshara 200k | +0.45 | +0.18 | +0.23 | +0.93 |
| North-Micro-Vision-Instruct | 255,033 | Akshara 248k | +0.50 | −0.05 | +0.07 | +1.48 |
| Falcon-H1-34B-Instruct | 261,120 | Akshara 248k | +1.01 | +0.31 | +0.74 | +1.97 |
| sarvam-105b † | 262,144 | Akshara 248k | +0.68 | +0.19 | +0.49 | +1.37 |
| Gemma 3 / Gemma 4 tokenizer | 262,144 | Akshara 248k | +0.43 | +0.18 | +0.48 | +0.63 |
At or above 151k, Akshara matches or beats every same-size published tokenizer on all three groups, except on English against North-Micro-Vision-Instruct at 248k (−0.05).
Best tokenizer per test set bytes per token
| Test set | Akshara, by vocabulary size | Best other tokenizer | OpenAI GPT-4o (o200k_base) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 448k | 248k | 200k | 151k | 120k | 92k | 59k | 49k | 32k | Tokenizer | Score | Score | 448k uses fewer | |
| Averages | |||||||||||||
| Overall score | 5.41 | 5.14 | 5.03 | 4.88 | 4.75 | 4.59 | 4.32 | 4.20 | 3.93 | gemma-3-1b-it | 4.71 | 4.64 | 14% |
| Code average | 4.08 | 3.99 | 3.95 | 3.89 | 3.85 | 3.78 | 3.66 | 3.60 | 3.45 | North-Micro-Vision-Instruct | 3.92 | 3.85 | 6% |
| Languages average | 7.21 | 6.59 | 6.35 | 6.02 | 5.74 | 5.42 | 4.91 | 4.70 | 4.24 | gemma-3-1b-it | 5.97 | 5.33 | 26% |
| English | |||||||||||||
| English web | 4.94 | 4.84 | 4.79 | 4.72 | 4.66 | 4.56 | 4.39 | 4.31 | 4.09 | North-Micro-Vision-Instruct | 4.89 | 4.74 | 4% |
| Code | |||||||||||||
| Python | 4.36 | 4.27 | 4.23 | 4.17 | 4.13 | 4.05 | 3.91 | 3.85 | 3.69 | North-Micro-Vision-Instruct | 4.22 | 4.16 | 5% |
| JavaScript | 4.13 | 4.06 | 4.03 | 3.98 | 3.95 | 3.88 | 3.76 | 3.70 | 3.57 | Llama-3.2-1B | 4.04 | 3.98 | 4% |
| TypeScript | 4.40 | 4.29 | 4.24 | 4.17 | 4.12 | 4.05 | 3.88 | 3.84 | 3.71 | North-Micro-Vision-Instruct | 4.21 | 4.20 | 5% |
| Java | 4.72 | 4.63 | 4.58 | 4.51 | 4.46 | 4.39 | 4.26 | 4.20 | 4.05 | Llama-3.2-1B | 4.58 | 4.49 | 5% |
| C# | 5.04 | 4.94 | 4.88 | 4.81 | 4.76 | 4.68 | 4.55 | 4.47 | 4.29 | Llama-3.2-1B | 4.92 | 4.80 | 5% |
| C++ | 4.00 | 3.93 | 3.89 | 3.84 | 3.81 | 3.75 | 3.63 | 3.58 | 3.42 | North-Micro-Vision-Instruct | 3.92 | 3.88 | 3% |
| C | 3.57 | 3.50 | 3.47 | 3.43 | 3.39 | 3.34 | 3.25 | 3.20 | 3.08 | Llama-3.2-1B | 3.39 | 3.36 | 6% |
| Go | 3.73 | 3.65 | 3.61 | 3.57 | 3.53 | 3.48 | 3.37 | 3.32 | 3.20 | North-Micro-Vision-Instruct | 3.59 | 3.52 | 6% |
| Rust | 4.38 | 4.32 | 4.29 | 4.24 | 4.19 | 4.12 | 4.00 | 3.94 | 3.79 | North-Micro-Vision-Instruct | 4.24 | 4.16 | 5% |
| PHP | 4.04 | 3.95 | 3.92 | 3.88 | 3.84 | 3.78 | 3.68 | 3.63 | 3.50 | Llama-3.2-1B | 3.97 | 3.95 | 2% |
| Ruby | 4.21 | 4.15 | 4.11 | 4.06 | 4.01 | 3.95 | 3.83 | 3.77 | 3.62 | North-Micro-Vision-Instruct | 4.15 | 4.05 | 4% |
| Swift | 4.82 | 4.71 | 4.67 | 4.60 | 4.54 | 4.46 | 4.31 | 4.23 | 4.07 | Llama-3.2-1B | 4.60 | 4.46 | 7% |
| Kotlin | 4.77 | 4.63 | 4.57 | 4.49 | 4.43 | 4.36 | 4.22 | 4.16 | 3.97 | Llama-3.2-1B | 4.63 | 4.52 | 5% |
| Dart | 4.29 | 4.22 | 4.19 | 4.14 | 4.11 | 4.04 | 3.93 | 3.88 | 3.74 | GLM-5.3-Flash | 4.21 | 4.15 | 3% |
| Lua | 3.86 | 3.75 | 3.71 | 3.65 | 3.58 | 3.52 | 3.40 | 3.35 | 3.22 | North-Micro-Vision-Instruct | 3.66 | 3.62 | 6% |
| R | 3.22 | 3.14 | 3.11 | 3.04 | 3.01 | 2.96 | 2.86 | 2.82 | 2.72 | North-Micro-Vision-Instruct | 3.14 | 3.06 | 5% |
| SQL | 3.62 | 3.54 | 3.50 | 3.42 | 3.37 | 3.29 | 3.12 | 3.02 | 2.83 | North-Micro-Vision-Instruct | 3.50 | 3.36 | 7% |
| Bash/Shell | 3.63 | 3.53 | 3.49 | 3.43 | 3.37 | 3.31 | 3.18 | 3.11 | 2.98 | North-Micro-Vision-Instruct | 3.51 | 3.36 | 8% |
| PowerShell | 4.25 | 4.13 | 4.07 | 3.99 | 3.94 | 3.86 | 3.72 | 3.65 | 3.50 | North-Micro-Vision-Instruct | 3.93 | 3.75 | 12% |
| HTML/CSS | 3.84 | 3.79 | 3.76 | 3.72 | 3.68 | 3.63 | 3.51 | 3.43 | 3.25 | North-Micro-Vision-Instruct | 3.70 | 3.58 | 7% |
| Assembly | 2.74 | 2.68 | 2.64 | 2.60 | 2.57 | 2.53 | 2.45 | 2.40 | 2.29 | North-Micro-Vision-Instruct | 2.56 | 2.50 | 9% |
| Languages | |||||||||||||
| German | 4.20 | 3.94 | 3.84 | 3.71 | 3.60 | 3.45 | 3.21 | 3.12 | 2.90 | North-Micro-Vision-Instruct | 3.87 | 3.70 | 12% |
| French | 4.60 | 4.35 | 4.25 | 4.12 | 3.99 | 3.83 | 3.59 | 3.50 | 3.27 | MiniMax-M2.5 | 4.38 | 4.10 | 11% |
| Spanish | 4.83 | 4.58 | 4.47 | 4.32 | 4.19 | 4.03 | 3.76 | 3.66 | 3.41 | MiniMax-M2.5 | 4.66 | 4.37 | 10% |
| Russian | 7.38 | 6.76 | 6.51 | 6.19 | 5.93 | 5.62 | 5.12 | 4.90 | 4.42 | Falcon-H1-34B-Instruct | 6.48 | 6.08 | 18% |
| Chinese | 5.10 | 4.73 | 4.61 | 4.44 | 4.29 | 4.12 | 3.78 | 3.67 | 3.37 | Qwen3.8-27B | 4.62 | 3.44 | 32% |
| Japanese | 5.93 | 5.47 | 5.31 | 5.09 | 4.90 | 4.69 | 4.35 | 4.18 | 3.85 | MiniMax-M2.5 | 5.33 | 3.53 | 40% |
| Korean | 4.55 | 4.17 | 4.04 | 3.87 | 3.73 | 3.57 | 3.28 | 3.17 | 2.90 | North-Micro-Vision-Instruct | 4.30 | 3.59 | 21% |
| Arabic | 7.21 | 6.58 | 6.33 | 6.03 | 5.75 | 5.45 | 4.95 | 4.76 | 4.33 | Falcon-H1-34B-Instruct | 6.73 | 5.73 | 21% |
| Urdu | 6.37 | 5.97 | 5.79 | 5.55 | 5.34 | 5.07 | 4.59 | 4.36 | 3.95 | gemma-3-1b-it | 5.59 | 5.25 | 18% |
| Hindi | 9.20 | 8.49 | 8.17 | 7.79 | 7.39 | 6.96 | 6.31 | 6.09 | 5.45 | gemma-3-1b-it | 9.06 | 7.62 | 17% |
| Marathi | 9.93 | 8.96 | 8.58 | 8.04 | 7.59 | 7.08 | 6.36 | 6.08 | 5.43 | sarvam-1 | 9.14 | 7.01 | 29% |
| Gujarati | 8.62 | 7.75 | 7.44 | 6.95 | 6.60 | 6.19 | 5.59 | 5.33 | 4.73 | sarvam-105b | 7.46 | 6.49 | 25% |
| Bengali | 11.69 | 10.56 | 10.07 | 9.39 | 8.81 | 8.23 | 7.18 | 6.78 | 6.02 | gemma-3-1b-it | 11.37 | 7.66 | 34% |
| Tamil | 10.97 | 9.82 | 9.38 | 8.84 | 8.33 | 7.81 | 6.98 | 6.64 | 5.90 | sarvam-1 | 9.75 | 7.37 | 33% |
| Telugu | 9.23 | 8.20 | 7.84 | 7.33 | 6.96 | 6.47 | 5.77 | 5.51 | 4.91 | sarvam-1 | 7.79 | 6.27 | 32% |
| Thai | 12.36 | 10.92 | 10.43 | 9.66 | 9.03 | 8.36 | 7.34 | 6.96 | 6.09 | Qwen3.8-27B | 10.68 | 6.86 | 44% |
| Vietnamese | 4.99 | 4.87 | 4.80 | 4.70 | 4.58 | 4.44 | 4.16 | 4.00 | 3.59 | North-Micro-Vision-Instruct | 4.79 | 4.25 | 15% |
| Turkish | 4.92 | 4.47 | 4.31 | 4.07 | 3.90 | 3.70 | 3.33 | 3.20 | 2.92 | Mistral-Large-4-1T-A52B | 3.96 | 3.78 | 23% |
| Indonesian | 4.99 | 4.64 | 4.49 | 4.29 | 4.11 | 3.90 | 3.57 | 3.43 | 3.12 | North-Micro-Vision-Instruct | 4.73 | 4.08 | 18% |
Same vocabulary size Akshara cut to each group's smallest vocabulary
The distinct benchmarked tokenizers grouped by vocabulary size. In each group Akshara is cut to the group's smallest vocabulary and compared on every measure. Select a group to see its chart.
| Vocabulary | Akshara size | Compared with | Akshara | Best other | Overall | Ahead by | Measures led |
|---|---|---|---|---|---|---|---|
| 32,000 | 4 tokenizers | 3.93 | Mamba-Codestral-7B-v0.1 | 3.26 | +20% | 40 of 44 | |
| 49,000 | 6 tokenizers | 4.20 | pythia-160m | 3.49 | +20% | 41 of 44 | |
| 59,000 | 8 tokenizers | 4.32 | Falcon-H1-3B-Instruct | 3.63 | +19% | 30 of 44 | |
| 92,000 | 7 tokenizers | 4.59 | ERNIE-4.5-21B-A3B-PT | 3.89 | +18% | 24 of 44 | |
| 120,000 | 13 tokenizers | 4.75 | Mistral-Nemo-Base-2407 | 4.31 | +10% | 20 of 44 | |
| 151,000 | 7 tokenizers | 4.88 | Seed-OSS-36B-Instruct | 4.11 | +19% | 34 of 44 | |
| 200,000 | 4 tokenizers | 5.03 | OpenAI o200k_base (GPT-4o) | 4.64 | +8% | 37 of 44 | |
| 248,000 | 8 tokenizers | 5.14 | gemma-3-1b-it | 4.71 | +9% | 36 of 44 |