OpenAI Tokenizer

Use Tiktoken Online in the Browser

Choose a GPT model for a compatible view or select a raw encoding such as o200k_base or cl100k_base. Paste text into the editor and the page updates the token count, colored boundaries, and IDs immediately. Show whitespace is useful when indentation, line endings, or leading spaces explain an unexpected result.

A raw-encoding selection measures exactly what that encoding does to the pasted text. A model selection is labeled compatible because a complete API request can contain message structure, tool definitions, images, and provider-managed tokens that are not present in a plain text field.

OpenAI Tokenizer Counts by Model

Compare the exact same text without quotation marks or a trailing newline: “Hello world! 你好，世界！”. The table counts only that raw text. Select the named raw encoding in the browser to reproduce the result; a chat-model selection may add message formatting.

Current tiktoken maps gpt-5-mini, gpt-4o, and gpt-4.1 to o200k_base, so their base-text counts match here. Its gpt-5 prefix rule also resolves gpt-5.6 to o200k_base; that is a compatibility mapping, not independent verification of every GPT-5.6 request format. The cl100k_base row is a raw-encoding comparison and is not assigned to any of these four models.

Count Tokens with Tiktoken in Python

Install the tiktoken package in your Python environment, load the same raw encoding selected in the browser, and encode the same string. The length of the returned ID list is the raw-text token count, and the list itself can be compared with the IDs displayed by Tiktokenizer.

Choose an encoding deliberately when you need a reproducible comparison. Do not assume that one encoding reproduces every model or every request wrapper. For production limits, keep the model ID and complete input structure tied to the provider's current documentation.

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")
text = "Count this prompt with tiktoken."
token_ids = encoding.encode(text)

print(len(token_ids))
print(token_ids)
print(encoding.decode(token_ids))

Choose the OpenAI Tokenizer by Model Name in Python

Use tiktoken.encoding_for_model when your application knows the model ID. For gpt-4o, the lookup returns o200k_base. This example encodes the same text as the model table and prints its encoding name, eight raw-text tokens, and token IDs.

Model lookup selects a vocabulary; it does not add chat roles or message boundaries. Keep tiktoken up to date, handle an unrecognized model explicitly, and check whether a result comes from a named mapping or a broad prefix rule before treating it as verified model support.

import tiktoken

text = "Hello world! 你好，世界！"
encoding = tiktoken.encoding_for_model("gpt-4o")
token_ids = encoding.encode(text)

print(encoding.name)  # o200k_base
print(len(token_ids))  # 8 raw-text tokens
print(token_ids)

Estimate Tokens for a Multi-Turn Conversation in Python

A multi-turn prompt includes the earlier user and assistant messages you send again, plus message framing. This text-only example follows the OpenAI Cookbook estimate for the pinned gpt-4o-2024-08-06 Chat Completions snapshot: three tokens per message, one extra token for an optional name field, and three tokens to prime the next assistant reply.

These overhead constants are specific to the documented estimate. Do not reuse them as an exact GPT-5.6 count or for tool calls, images, files, or Responses API inputs. Count the complete supported Responses input with the official input_tokens endpoint, or compare the estimate with usage.prompt_tokens after a Chat Completions request. The next generated reply is output usage and is not included below.

import tiktoken

# Text-only estimate for this pinned Chat Completions snapshot.
model = "gpt-4o-2024-08-06"
encoding = tiktoken.encoding_for_model(model)
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help?"},
    {"role": "user", "content": "Explain tokens in one sentence."},
]

estimated_tokens = 3  # Assistant reply priming.
for message in messages:
    estimated_tokens += 3  # Message framing for this snapshot.
    for field in ("role", "content", "name"):
        if field in message:
            estimated_tokens += len(encoding.encode(message[field]))
            if field == "name":
                estimated_tokens += 1

print("Estimated input tokens:", estimated_tokens)
# Excludes tools, images, files, and the next generated reply.
# Confirm actual input usage with response.usage.prompt_tokens.

Match Python and Browser Token IDs

Use the exact same Unicode text and encoding name on both sides. Invisible differences such as a trailing newline, non-breaking space, smart quote, or normalization change can produce different token IDs even when two strings look similar on screen.

If the totals differ, turn on whitespace visualization, copy the raw text again, and compare the first token position where the ID lists diverge. This approach narrows the problem to the input or encoding instead of treating tokenization as a black box.

Tiktoken Count vs OpenAI API Input Tokens

Tiktoken is useful for local raw-text inspection. OpenAI's Responses API also exposes a request-level input-token counting operation at POST /responses/input_tokens. It accepts request components such as model, instructions, conversation context, text, image, or file inputs and returns an input_tokens total.

Use the API operation when you need a count for the structured request you plan to send. Use the browser or Python tokenizer when you need to understand token boundaries, inspect IDs, compare encodings, or debug why a particular string is expensive.

Tiktokenizer Reference Sources

Tokenizer behavior and model limits change. Verify production decisions with current provider documentation:

• OpenAI API: Count input tokens
• OpenAI tiktoken repository
• OpenAI model comparison
• OpenAI tiktoken model-to-encoding mappings
• OpenAI Cookbook: Count tokens with tiktoken
• OpenAI: Understanding and counting tokens