Tokenizer · version 0.5.0
Akshara
A tokenizer built for Indian languages, code and English. It runs in your browser: type below and see every token.
Try it
Pick Akshara at any of its nine sizes or an OpenAI encoding. Everything runs locally; your text never leaves the page.
The measurements
40 tokenizers on 41 held-out test sets: English web text, 21 programming languages and 19 Wikipedia languages.
Benchmarks
Score against vocabulary size, matched-size pairings with every published tokenizer, and where Akshara trails.
Evaluation
Speed, memory, correctness and cost on one shared library, ranked per workload.
Rankings
Every tokenizer ranked on any of the 41 test sets.
Full results
Bytes per token for every tokenizer on every test set, sortable.