Current LLM tokenizer directory

LLM Tokenizer Support by Model

Find the correct token counter for current language models. This directory separates exact local tokenizers, compatible encoding views, and provider API counting methods so you can choose an authoritative result.

Accuracy note

Support labels describe the counting method, not model quality. Exact local means the published tokenizer is loaded directly; API counting means the model provider remains authoritative.

Facts checked against official sources on .

Which LLM Token Counter Should You Use?

Use the tokenizer or counting endpoint tied to the exact model and request structure you plan to send. A raw encoding can explain text boundaries, but chat roles, tool schemas, images, documents, cached content, and provider-side formatting can change the number reported by an API.

Tiktokenizer labels every model as exact local, compatible, or API counting required. That distinction is the quickest way to decide whether the interactive browser result can be used directly or should be treated only as a comparison.

Model or familySupport levelAuthoritative counting method
GPT-5.6Compatibleo200k_base view; confirm complete request usage with OpenAI
DeepSeek V4 FlashExact base textPublished Hugging Face tokenizer; DeepSeek usage for the final request
Kimi K3Exact base textPublished K3 vocabulary; Kimi API usage for complete XTML and multimodal requests
Grok 4.5Exact via APIxAI TokenizeText endpoint for model grok-4.5
Gemini 3.6 FlashExact via APIGemini models.countTokens
Claude Sonnet 5Exact via APIAnthropic Messages count_tokens
DeepSeek R1Exact base textPublished Hugging Face tokenizer
GPT-OSS 20B & 120BCompatible base texto200k_base view; Harmony wrapper excluded

How Tiktokenizer Defines Accuracy

  • Exact local tokenizer: the page loads the tokenizer artifacts published for the named open model and counts base text in the browser.
  • Compatible encoding: the page uses a known base encoding that is useful for inspection but does not claim to reproduce every provider-side wrapper.
  • Exact via provider API: the model owner exposes a model-aware counting or tokenization endpoint, so the page documents that endpoint and treats the browser explorer as comparison-only.
  • Reference only: no verified browser tokenizer or dedicated count endpoint is available; the final API usage is the source of truth.

Why Model-Specific Token Counts Differ

Token IDs belong to a vocabulary. The same text can produce different boundaries and counts across OpenAI, DeepSeek, Kimi, Grok, Gemini, Claude, Qwen, Mistral, and Llama models. A tokenizer update can also change the amount of text represented by a fixed context window.

Structured requests introduce another layer. System instructions, prior messages, function definitions, media inputs, reasoning, and special tokens may not appear in the text editor even though they contribute to request usage. For hard limits, cost estimates, or migrations, compare the local analysis with the usage returned by the target provider.

Tiktokenizer Reference Sources

Tokenizer behavior and model limits change. Verify production decisions with current provider documentation:

Frequently asked questions

Can one tokenizer count tokens for every LLM?

No. Token vocabularies, normalization rules, special tokens, and chat templates vary by model family.

What does exact base-text counting exclude?

It can exclude chat templates, tools, media inputs, provider-added formatting, and other request components outside the pasted text.

Which number should I use for billing?

Use the usage returned by the provider for the real request. A preflight tokenizer or count endpoint is planning data and may differ slightly from final billed usage.

Does Tiktokenizer upload prompts?

Supported local tokenizers run in the browser. Provider API examples are separate methods and follow the provider's own data policies.