How the LLM token counter works
Start with the same plain-text prompt and choose a raw encoding from the model selector. The counter splits the input into token pieces and displays the corresponding IDs. Keep punctuation, whitespace and line breaks unchanged when comparing two tokenizers, so a change in input does not get mistaken for a difference between models.
For a useful multilingual check, paste Hello world! 你好，世界！. The o200k_base raw encoding produces 8 tokens and cl100k_base produces 11. Switching between these two raw encodings keeps the text available for comparison. Selecting a chat-model workflow can introduce message formatting, so compare like-for-like inputs when interpreting the result.
LLM token counter vs word counter
Words are not universal model input units. A token may be a word, a word fragment, punctuation or another character sequence. The vocabulary determines the mapping, which is why the same text can produce different token boundaries and totals under different encodings. Numeric IDs are meaningful only alongside the tokenizer that produced them.
Use a word counter for an editorial length requirement and an LLM token counter to inspect a model input. A short code sample or compact JSON object can have a different token density from ordinary prose. The practical comparison is the actual text under the intended tokenizer, rather than a fixed rule such as one token per word.
LLM token count by model
This table uses the same support data as the model directory. Published base-text tokenizers can run locally, compatible views are labeled as such, and provider API methods remain separate. An API-only row documents a counting method; it does not mean that the provider's tokenizer has been loaded into this browser.
When migrating an application, record the text, selected tokenizer and count for each test case. Then check a complete request with the target provider. System instructions, conversation templates, tool definitions and media can change the input beyond the visible text. Use the linked model guides to understand where each local measurement stops.
| Model | Encoding / tokenizer | Is the count exact? |
|---|---|---|
| GPT-5.6 | o200k_base (compatible view) | Compatible |
| DeepSeek V4 Flash | Published DeepSeek V4 tokenizer | Exact base text |
| Kimi K3 | Published Kimi K3 vocabulary | Exact base text |
| Grok 4.5 | Provider tokenizer via xAI TokenizeText | Exact via API |
| Gemini 3.6 Flash | Provider tokenizer via Gemini countTokens | Exact via API |
| Claude Sonnet 5 | Provider tokenizer via Anthropic count_tokens | Exact via API |
| DeepSeek R1 | Published DeepSeek R1 tokenizer | Exact base text |
| GPT-OSS 20B & 120B | o200k_base (Harmony wrapper excluded) | Compatible base text |