Aksharav0.5.0

Rankings

Every tokenizer, ranked on any test set

UTF-8 bytes per token on held-out text; higher means fewer tokens for the same text. Filter by provider or vocabulary size, or switch from single lines to whole documents.
  • Akshara
  • Google
  • OpenAI
  • Cohere
  • Mistral
  • Sarvam
  • Alibaba (Qwen)
  • Liquid AI
  • DeepSeek
  • MiniMax
  • TII (Falcon)
  • ByteDance
  • Meta
  • Moonshot AI (Kimi)
  • Tencent
  • Z.ai (GLM)
  • inclusionAI (Ant Group)
  • Baidu
  • Ai2
  • Microsoft
  • EleutherAI
  • Hugging Face