Text Encoding Tokenizer

What is an encoding?

An encoding defines how text is split into tokens and which numeric ID is assigned to each token. It is a vocabulary and a set of rules, rather than a language model. Multiple models can share an encoding.

This tool lets you choose those rules directly. The model tokenizer tool instead selects a supported tokenizer or reference encoding for the model you choose. You do not need to combine the two selectors.

How to use this tool

Choose an encoding, paste your text and inspect the count, colored token boundaries and numeric IDs. Enable Show whitespace to reveal spaces and line endings. Switch encodings to compare the same text without changing the input.

Use this view for plain-text analysis and debugging. For a particular model, use its model page and check the stated support level. Complete API usage can include chat formatting and other inputs that are outside a raw text count.

Available text splitting rules

Tiktokenizer Reference Sources

Tokenizer behavior and model limits change. Verify production decisions with current provider documentation:

• OpenAI tiktoken source and encoding documentation