Evaluation

Speed, memory, correctness and cost

Bytes per token says how well a tokenizer compresses text, not whether it is fast, small, correct or cheap for a given job. 40 tokenizers measured on six separate dimensions, every number tagged as measured, estimated or unavailable.

Best tokenizer for a workload

Workload

Multilingual and mixed-language chat, parity with English, no byte fallback. Each column is a raw value, with ↑ or ↓ for the better direction; the score is the mean of their percentile ranks (1 = best).

Worst in the columnBest in the column

#	Tokenizer	Score	Tokens, multilingual chat ↓	Tokens, mixed-language chat ↓	Bytes/token, languages ↑	Parity gap vs English ↓	Byte-fallback share ↓
1	Akshara 448k	0.99	51,424	22,958	7.213	0.254	0.15%
2	Akshara 248k	0.93	54,663	24,303	6.592	0.320	0.25%
3	Akshara 200k	0.88	56,012	24,928	6.350	0.347	0.31%
4	gemma-3-1b-it	0.88	53,716	25,167	5.966	0.330	0.37%
5	gemma-4-31B-it	0.86	53,821	25,167	5.966	0.330	0.37%
6	Akshara 151k	0.79	57,827	25,793	6.019	0.386	0.40%
7	OpenAI o200k_base	0.76	52,950	26,250	5.326	0.444	0.80%
8	Mistral-Large-4-1T-A52B	0.75	53,059	26,596	5.417	0.402	1.17%
9	Akshara 120k	0.71	59,769	26,805	5.739	0.417	0.49%
10	Akshara 92k	0.65	62,101	27,919	5.419	0.451	0.64%
11	North-Micro-Vision-Instruct	0.65	51,377	28,350	5.111	0.512	5.09%
12	Mistral-Nemo-Base-2407	0.57	57,645	28,794	4.805	0.514	1.74%
13	LFM3-1.7B	0.54	57,513	30,569	4.418	0.595	1.02%
14	Akshara 59k	0.54	66,162	29,961	4.906	0.502	1.07%
15	DeepSeek-V3	0.48	57,084	31,914	4.263	0.686	3.42%
16	Akshara 49k	0.46	68,002	31,005	4.703	0.520	1.30%
17	Falcon-H1-34B-Instruct	0.44	58,120	32,199	4.620	0.556	10.18%
18	Akshara 32k	0.37	72,421	33,562	4.240	0.564	2.15%
19	sarvam-105b	0.36	61,691	36,378	5.223	0.628	44.85%
20	ERNIE-4.5-21B-A3B-PT	0.34	66,085	33,845	3.881	0.692	3.55%
21	Kimi-K3	0.31	65,407	35,210	3.593	0.838	5.86%
22	GLM-5.3-Flash	0.29	57,233	42,375	3.309	0.933	17.66%
23	Hy3	0.28	64,276	36,821	3.494	0.856	8.92%
24	Llama-3.2-1B	0.28	60,394	40,321	3.657	0.864	18.43%
25	cl100k_base	0.16	69,668	47,553	2.644	1.150	22.39%
26	Phi-3.5-mini-instruct	0.11	76,201	56,772	2.388	1.160	28.87%
27	SmolLM2-135M	0.08	98,488	64,809	1.933	1.479	42.37%
28	OpenAI p50k_base	0.03	110,194	71,477	1.788	1.605	53.08%
29	gpt2	0.01	111,625	71,484	1.787	1.605	53.03%

Measurements ↑ higher is better, ↓ lower is better

Speed and memory are measured on the same library for every tokenizer. Italic values are estimates (arithmetic on stated assumptions); n/a values were not measured. Hover or focus a value with a dotted underline to see its note.

Section

Tokenizer	Vocabulary	Bytes/token, overall ↑	Bytes/token, English ↑	Bytes/token, code ↑	Bytes/token, languages ↑	Parity gap vs English ↓
Akshara 448k	448,000	5.409	4.936	4.078	7.213	0.254
Akshara 248k	248,000	5.141	4.839	3.991	6.592	0.320
Akshara 200k	200,000	5.032	4.792	3.952	6.350	0.347
Akshara 151k	151,000	4.877	4.719	3.894	6.019	0.386
Akshara 120k	120,000	4.747	4.655	3.847	5.739	0.417
Akshara 92k	92,000	4.589	4.565	3.782	5.419	0.451
Akshara 59k	59,000	4.319	4.393	3.658	4.906	0.502
Akshara 49k	49,000	4.202	4.306	3.598	4.703	0.520
Akshara 32k	32,000	3.928	4.093	3.452	4.240	0.564
Phi-3.5-mini-instruct	32,011	3.170	4.044	3.077	2.388	1.160
SmolLM2-135M	49,152	3.204	4.543	3.136	1.933	1.479
gpt2	50,257	2.950	4.659	2.405	1.787	1.605
pythia-160m	50,277	3.493	4.666	3.369	2.444	1.230
OpenAI p50k_base (Codex)	50,281	3.216	4.659	3.202	1.788	1.605
sarvam-1	68,096	3.498	4.020	2.766	3.709	1.024
OpenAI cl100k_base (GPT-4)	100,277	3.736	4.691	3.872	2.644	1.150
OLMo-2-0425-1B	100,278	3.736	4.691	3.872	2.644	1.150
Olmo-3-1025-7B	100,278	3.736	4.691	3.872	2.644	1.150
phi-4	100,352	3.736	4.691	3.872	2.644	1.150
ERNIE-4.5-21B-A3B-PT	101,304	3.885	4.401	3.373	3.881	0.692
Hy3	120,818	3.965	4.706	3.696	3.494	0.856
LFM3-1.7B	125,033	4.213	4.628	3.594	4.418	0.595
Llama-3.2-1B	128,256	4.080	4.693	3.891	3.657	0.864
DeepSeek-V3	128,815	4.202	4.732	3.610	4.263	0.686
Mistral-Nemo-Base-2407	131,072	4.312	4.552	3.579	4.805	0.514
Qwen2.5-0.5B	151,665	3.903	4.573	3.793	3.343	0.894
Qwen3-0.6B	151,669	3.903	4.573	3.793	3.343	0.894
GLM-5.3-Flash	154,856	3.958	4.685	3.880	3.309	0.933
Seed-OSS-36B-Instruct	155,121	4.109	4.589	3.617	4.122	0.658
Ling-3.0-tiny	157,153	3.920	4.617	3.669	3.473	0.870
Kimi-K3	163,584	4.041	4.693	3.837	3.593	0.838
OpenAI o200k_base (GPT-4o)	200,019	4.640	4.742	3.852	5.326	0.444
MiniMax-M2.5	200,054	4.161	4.821	3.780	3.882	0.800
Mistral-Large-4-1T-A52B	204,800	4.584	4.609	3.727	5.417	0.402
Qwen3.8-27B	248,077	4.456	4.589	3.809	4.970	0.475
North-Micro-Vision-Instruct	255,033	4.640	4.888	3.920	5.111	0.512
Falcon-H1-34B-Instruct	261,120	4.134	4.531	3.250	4.620	0.556
gemma-4-31B-it	262,144	4.710	4.659	3.505	5.966	0.330
sarvam-105b	262,144	4.458	4.653	3.498	5.223	0.628
gemma-3-1b-it	262,145	4.713	4.659	3.512	5.966	0.330

Encode speed by input size

MB encoded per second on one thread, on the same library for every tokenizer. Higher is better.

Showing 6 of 36. Select a model to add or remove it.

Tokens per language, relative to English same FLORES-200 sentences; lower is better

Each value is a language's token count divided by English's for the same sentences: 1.00 means as many tokens as English, 2.00 twice as many. Deeper red means more tokens, so worse.

1.00 or less: as many as English4× English or more

Tokenizer	de	fr	es	ru	zh	ja	ko	ar	ur	hi	mr	gu	bn	ta	te	th	vi	tr	id
Akshara 448k	1.21	1.30	1.25	1.32	0.96	1.03	1.25	1.19	1.48	1.51	1.40	1.46	1.38	1.46	1.50	1.24	1.37	1.19	1.08
Akshara 248k	1.28	1.34	1.30	1.44	1.02	1.10	1.36	1.28	1.57	1.62	1.52	1.59	1.48	1.61	1.65	1.37	1.39	1.29	1.16
Akshara 200k	1.31	1.37	1.32	1.48	1.04	1.13	1.41	1.32	1.61	1.67	1.58	1.64	1.53	1.67	1.71	1.41	1.40	1.32	1.19
Akshara 151k	1.35	1.40	1.36	1.55	1.08	1.18	1.48	1.37	1.66	1.73	1.67	1.73	1.59	1.74	1.82	1.50	1.42	1.37	1.24
Akshara 120k	1.39	1.43	1.38	1.60	1.10	1.22	1.52	1.41	1.70	1.79	1.73	1.79	1.65	1.82	1.89	1.58	1.45	1.41	1.28
Akshara 92k	1.42	1.46	1.40	1.66	1.13	1.25	1.57	1.46	1.74	1.86	1.82	1.87	1.73	1.89	1.98	1.66	1.48	1.45	1.32
Akshara 59k	1.48	1.50	1.45	1.75	1.18	1.31	1.66	1.53	1.83	1.94	1.94	1.98	1.82	2.03	2.14	1.80	1.52	1.55	1.38
Akshara 49k	1.49	1.51	1.46	1.79	1.19	1.33	1.68	1.55	1.87	1.96	1.99	2.03	1.87	2.09	2.18	1.85	1.56	1.58	1.40
Akshara 32k	1.52	1.53	1.49	1.89	1.23	1.39	1.76	1.60	1.93	2.06	2.11	2.15	1.95	2.23	2.31	1.96	1.66	1.64	1.45
Phi-3.5-mini-instruct	1.41	1.46	1.44	1.66	2.02	2.25	3.20	3.47	4.40	4.61	4.59	10.00	5.39	5.91	10.78	4.37	2.92	2.09	1.76
SmolLM2-135M	2.07	1.92	1.93	3.94	2.68	3.65	4.80	4.04	5.55	5.18	5.47	11.88	8.54	9.53	12.81	8.27	3.83	2.39	1.92
gpt2	2.13	1.98	1.97	5.77	3.20	2.98	5.05	4.43	6.29	7.47	7.86	12.19	9.62	15.53	13.14	8.99	4.53	2.43	1.97
pythia-160m	1.81	1.74	1.71	2.69	2.10	2.21	3.58	3.02	4.49	4.77	5.13	6.95	6.57	7.81	7.36	5.11	3.21	2.19	1.87
OpenAI p50k_base	2.13	1.98	1.97	5.77	3.20	2.98	5.05	4.43	6.29	7.47	7.86	12.19	9.62	15.53	13.14	8.99	4.53	2.43	1.97
sarvam-1	2.16	2.12	2.07	8.38	3.85	5.38	5.00	6.79	7.41	1.15	1.07	1.21	1.28	1.16	1.17	11.54	4.36	2.68	1.95
cl100k_base	1.58	1.59	1.54	2.51	1.90	2.29	2.39	3.08	4.41	4.80	5.07	7.69	5.85	7.66	8.43	4.36	2.45	1.92	1.55
ERNIE-4.5-21B-A3B-PT	1.57	1.59	1.52	1.98	0.95	1.70	2.15	1.90	2.10	2.35	2.51	2.94	2.59	2.93	3.31	2.68	1.70	1.62	1.37
Hy3	1.69	1.66	1.67	2.13	0.93	1.63	1.71	2.03	3.00	3.40	3.72	5.16	3.93	3.88	5.45	2.47	1.76	1.79	1.52
LFM3-1.7B	1.36	1.39	1.38	1.66	1.23	1.31	1.39	1.36	2.06	2.53	2.72	2.70	2.87	3.20	3.05	2.07	1.41	1.52	1.31
Llama-3.2-1B	1.58	1.59	1.54	1.65	1.30	1.53	1.50	1.68	3.12	2.55	2.80	7.70	5.81	7.67	8.43	2.20	1.39	1.41	1.54
DeepSeek-V3	1.51	1.53	1.49	1.62	0.96	1.44	1.67	1.64	2.40	2.83	3.02	3.88	2.04	3.10	3.83	1.79	2.11	2.03	1.35
Mistral-Nemo-Base-2407	1.31	1.31	1.31	1.54	1.43	1.58	1.30	1.27	1.79	1.82	2.16	2.75	2.07	2.20	2.31	2.37	1.43	1.49	1.31
Qwen2.5-0.5B	1.56	1.56	1.52	1.79	1.02	1.42	1.65	1.66	3.20	4.46	4.63	6.76	5.06	6.14	7.12	2.56	1.43	1.63	1.54
GLM-5.3-Flash	1.44	1.50	1.43	1.46	0.97	1.51	1.95	1.80	2.78	4.35	4.59	7.69	4.75	7.66	8.42	4.18	1.46	1.65	1.54
Seed-OSS-36B-Instruct	1.40	1.45	1.43	1.52	1.00	2.15	1.72	1.70	2.27	2.70	2.77	3.08	2.77	3.00	2.96	2.61	1.46	1.53	1.31
Ling-3.0-tiny	1.56	1.58	1.58	1.76	0.88	1.51	1.50	1.82	3.60	3.83	4.27	6.98	2.57	7.01	6.81	2.29	1.60	1.65	1.53
Kimi-K3	1.72	1.72	1.69	2.08	0.89	1.66	2.00	2.09	2.20	2.77	2.90	3.78	2.99	4.19	4.00	3.55	2.80	1.95	1.58
OpenAI o200k_base	1.31	1.36	1.31	1.45	1.27	1.67	1.48	1.41	1.69	1.59	1.83	1.81	1.72	2.00	1.96	1.95	1.49	1.43	1.24
MiniMax-M2.5	1.49	1.29	1.24	1.77	0.93	1.03	1.33	1.38	3.82	2.97	3.31	6.73	3.78	4.05	4.25	4.75	2.79	1.96	1.08
Mistral-Large-4-1T-A52B	1.24	1.28	1.25	1.41	1.25	1.65	1.40	1.32	1.56	1.61	1.56	1.77	1.57	1.73	1.80	2.40	1.41	1.32	1.26
Qwen3.8-27B	1.31	1.38	1.31	1.39	0.93	1.16	1.23	1.33	2.01	2.04	2.32	3.10	2.21	2.62	2.91	1.20	1.31	1.43	1.09
North-Micro-Vision-Instruct	1.31	1.37	1.33	1.51	1.11	1.19	1.22	1.37	2.03	1.67	2.38	5.85	2.04	2.31	2.49	1.70	1.36	1.55	1.08
Falcon-H1-34B-Instruct	1.41	1.49	1.37	1.27	0.92	1.11	1.16	1.13	1.56	1.69	1.69	2.40	1.87	7.27	6.52	2.47	1.68	1.55	1.08
gemma-4-31B-it	1.32	1.38	1.28	1.39	1.09	1.21	1.37	1.49	1.55	1.32	1.40	1.87	1.22	1.48	1.80	1.60	1.37	1.37	1.14
sarvam-105b	1.32	1.38	1.28	1.39	1.09	4.50	5.77	1.49	1.55	1.32	1.40	1.51	1.22	1.48	1.47	13.38	3.91	1.61	1.14
gemma-3-1b-it	1.32	1.38	1.28	1.39	1.09	1.21	1.37	1.49	1.55	1.32	1.40	1.87	1.22	1.48	1.80	1.60	1.37	1.37	1.14

About this evaluation