Text to tokens · No installation

Online Tokenizer

Turn text into token pieces and numeric IDs in your browser. Paste a sample, switch encodings and inspect spaces, punctuation and multilingual text.

Accuracy note

Raw encodings count the supplied text exactly for that encoding. Compatible model views exclude some request formatting; provider API usage remains authoritative for complete requests.

How the online tokenizer works

Paste text into the editor to see its token count, colored segments and numeric IDs. Each visual segment connects a readable piece of the input to the IDs needed to represent it. Hover over a segment or an ID to highlight the matching part, then enable Show whitespace when you need to inspect spaces or line breaks.

The default o200k_base selection analyzes plain text without adding a chat wrapper. Switch to cl100k_base to compare another raw vocabulary. For example, Hello world! 你好，世界！ produces 8 tokens with o200k_base and 11 with cl100k_base. Keep the sample unchanged to see how the tokenizer, rather than an edit, changes the result.

Online tokenizer vs word counter

An online tokenizer shows the units used by a selected encoding; a word counter counts words according to its own text rules. One visual word may become several token IDs, and punctuation or spacing can also affect the split. This makes token inspection useful for debugging prompts that appear short but take more input space than expected.

Try a sentence, a code snippet and a JSON object separately. Compare their token totals and inspect the separators around numbers, quotes and newlines. The exercise shows why character length and word length are useful descriptions of a document but cannot guarantee its token length under every model vocabulary.

Online token count by encoding and model

Raw encodings provide reproducible tokenization for the text entered. Model presets may use a compatible base encoding or a published model tokenizer, and their request formatting can have a different scope. Use the support note and model directory to select the counting method that matches your purpose.

When exporting IDs into your own script, record which encoding produced them. Token IDs are vocabulary-specific, so an ID list alone is not a portable text format. If your goal is a complete API input count, include the same messages, tools and other supported inputs in the provider's counting workflow.

SelectionWhat is countedAccuracy boundary
o200k_baseRaw text in the selected vocabularyExact raw encoding; no complete request wrapper
cl100k_baseRaw text in a different vocabularyExact raw encoding; choose it deliberately
GPT-5.6 compatible viewText under the compatible o200k_base viewCheck complete request usage with OpenAI

Frequently asked questions

Is this online tokenizer free?

Yes. Paste text, inspect tokens and switch supported encodings without payment or signup.

Does an online tokenizer send my text to a server?

This tool performs supported tokenization locally in your browser. It may download tokenizer files, but it does not upload your pasted prompt to an application server.

What does Show whitespace do?

It makes spaces and line breaks easier to inspect in the visual token display. It changes the presentation, not the original text or token count.

Why can a colored segment contain several token IDs?

Some readable character sequences, including certain Unicode characters, require several IDs. The visual display groups them so you can still read the text.

Can I tokenize code and JSON online?

Yes. Paste them as text and inspect punctuation, quotes, indentation and newlines with the selected encoding.

Does the online tokenizer work without JavaScript?

The instructions, encoding table and FAQs are readable in the initial HTML. Live tokenization requires JavaScript; no local Python installation is needed.