OpenAI Tokenizer

What is an OpenAI tokenizer?

An OpenAI tokenizer converts text into the tokens that an OpenAI model reads. Each token has a numeric ID in a particular vocabulary, and its readable piece may be a word, part of a word, punctuation, or bytes from a character. Tokenization lets you inspect the size and representation of a prompt before building a request around it.

Tokens are different from words and characters. Leading spaces, line breaks, source code, JSON keys, and multilingual text can all change the boundaries. Two passages with the same word count can therefore consume different numbers of tokens. Paste the actual text rather than relying on a fixed conversion, then inspect the colored segments and IDs to understand unexpected splits.

This page is an independent browser tool, not an official OpenAI service. Its raw encodings provide exact local results for the selected encoding. A compatible model view is useful for prompt inspection but does not establish complete API usage. A reference-only view has no verified mapping to the named model. Keep that support label alongside any count you save or share.

OpenAI tokenizer vs tiktoken: what's the difference?

OpenAI tokenizer describes the job of turning text into model tokens, or a tool that exposes that process. Tiktoken is OpenAI's open-source tokenizer library, with named encodings and functions that turn strings into token IDs. The library is useful when you want reproducible counts in code, while a browser interface makes boundaries and changes easier to inspect as you edit a prompt.

Use this tiktoken online playground when you want to paste text, switch an encoding, and see the result immediately. Tiktokenizer runs the supported encodings locally in your browser; it does not run the Python package on a remote server. To reproduce a browser result in Python, choose the same encoding and preserve the exact input, including whitespace and Unicode characters.

Neither a library name nor an online interface guarantees a complete model request count. Exact local applies to the raw encoding being measured. GPT-5.6 uses the site's compatible view, while GPT-6 remains reference-only. For messages with tools, images, or files, use OpenAI's model-aware input counting method with the structured request. The Python and conversation examples below explain how local inspection fits that workflow.

How to count tokens for OpenAI models

Start with the model ID you intend to call and decide whether you are measuring plain text or a complete request. For plain text, choose the relevant raw encoding in the editor, paste your prompt, and read the live count and token IDs. Keep the text unchanged when comparing browser and Python results. An extra newline or leading space can change the IDs without changing what the prompt appears to say.

The official OpenAI Cookbook lists o200k_base for GPT-4o and GPT-4o mini. It lists cl100k_base for GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, text-embedding-ada-002, text-embedding-3-small, and text-embedding-3-large. These mappings describe the text vocabulary. Here, GPT-5.6 keeps its compatible o200k_base inspection view; GPT-6 uses that encoding only as a reference-only comparison. Selecting a newer model name does not prove that an older encoding is its tokenizer.

For production input limits, send the same model ID, instructions, messages, tools, and supported media to OpenAI's Responses input-token counting operation. Its request count includes formatting that is absent from a pasted string. After generation, inspect actual response usage for output, reasoning, and cache categories. Encoding guidance checked on 2026-10-09 against the official OpenAI Cookbook and token-counting guide linked in Reference Sources.

Which OpenAI models does this tokenizer support?

This site provides model guides for the GPT-6 and GPT-5.6 families, but the local support level differs between them. The GPT-5.6 family page covers Sol, Terra, and Luna with a compatible o200k_base view. You can use it to inspect prompt text, compare revisions, and reproduce raw encoding IDs. It does not certify the complete input format of every API snapshot or include everything billed by OpenAI.

GPT-6 Sol and GPT-6 Luna have separate guides and a shared family page. They offer a no-key text-size estimate using OpenAI's rough English rule of about 4 characters per token. No GPT-6 tokenizer mapping, token boundaries or IDs are claimed. Use OpenAI's model-aware counting operation for the actual Sol or Luna request.

For direct raw-encoding work, choose o200k_base or cl100k_base to get an exact local count of the supplied text under that encoding. A model's guide and its support label explain how far that result can be taken. Follow the links below for each family member, or use the model directory to compare providers. When switching models, retain the original prompt and compare equivalent structured inputs before drawing a cost or context-limit conclusion.

Use Tiktoken Online in the Browser

Choose a GPT model for a compatible view or select a raw encoding such as o200k_base or cl100k_base. Paste text into the editor and the page updates the token count, colored boundaries, and IDs immediately. Show whitespace is useful when indentation, line endings, or leading spaces explain an unexpected result.

A raw-encoding selection measures exactly what that encoding does to the pasted text. A model selection is labeled compatible because a complete API request can contain message structure, tool definitions, images, and provider-managed tokens that are not present in a plain text field.

OpenAI Tokenizer Counts by Model

Compare the exact same text without quotation marks or a trailing newline: “Hello world! 你好，世界！”. The table counts only that raw text. Use Plain text mode with the named encoding to reproduce the result. Chat messages mode counts each message body separately and does not add provider message formatting.

Current tiktoken maps gpt-5-mini, gpt-4o, and gpt-4.1 to o200k_base, so their base-text counts match here. Its gpt-5 prefix rule also resolves gpt-5.6 to o200k_base; that is a compatibility mapping, not independent verification of every GPT-5.6 request format. The cl100k_base row is a raw-encoding comparison and is not assigned to any of these four models.

Count Tokens with Tiktoken in Python

Install the tiktoken package in your Python environment, load the same raw encoding selected in the browser, and encode the same string. The length of the returned ID list is the raw-text token count, and the list itself can be compared with the IDs displayed by Tiktokenizer.

Choose an encoding deliberately when you need a reproducible comparison. Do not assume that one encoding reproduces every model or every request wrapper. For production limits, keep the model ID and complete input structure tied to the provider's current documentation.

import tiktoken

encoding = tiktoken.get_encoding("o200k_base")
text = "Count this prompt with tiktoken."
token_ids = encoding.encode(text)

print(len(token_ids))
print(token_ids)
print(encoding.decode(token_ids))

Choose the OpenAI Tokenizer by Model Name in Python

Use tiktoken.encoding_for_model when your application knows the model ID. For gpt-4o, the lookup returns o200k_base. This example encodes the same text as the model table and prints its encoding name, eight raw-text tokens, and token IDs.

Model lookup selects a vocabulary; it does not add chat roles or message boundaries. Keep tiktoken up to date, handle an unrecognized model explicitly, and check whether a result comes from a named mapping or a broad prefix rule before treating it as verified model support.

import tiktoken

text = "Hello world! 你好，世界！"
encoding = tiktoken.encoding_for_model("gpt-4o")
token_ids = encoding.encode(text)

print(encoding.name)  # o200k_base
print(len(token_ids))  # 8 raw-text tokens
print(token_ids)

Estimate Tokens for a Multi-Turn Conversation in Python

A multi-turn prompt includes the earlier user and assistant messages you send again, plus message framing. This text-only example follows the OpenAI Cookbook estimate for the pinned gpt-4o-2024-08-06 Chat Completions snapshot: three tokens per message, one extra token for an optional name field, and three tokens to prime the next assistant reply.

These overhead constants are specific to the documented estimate. Do not reuse them as an exact GPT-5.6 count or for tool calls, images, files, or Responses API inputs. Count the complete supported Responses input with the official input_tokens endpoint, or compare the estimate with usage.prompt_tokens after a Chat Completions request. The next generated reply is output usage and is not included below.

import tiktoken

# Text-only estimate for this pinned Chat Completions snapshot.
model = "gpt-4o-2024-08-06"
encoding = tiktoken.encoding_for_model(model)
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help?"},
    {"role": "user", "content": "Explain tokens in one sentence."},
]

estimated_tokens = 3  # Assistant reply priming.
for message in messages:
    estimated_tokens += 3  # Message framing for this snapshot.
    for field in ("role", "content", "name"):
        if field in message:
            estimated_tokens += len(encoding.encode(message[field]))
            if field == "name":
                estimated_tokens += 1

print("Estimated input tokens:", estimated_tokens)
# Excludes tools, images, files, and the next generated reply.
# Confirm actual input usage with response.usage.prompt_tokens.

Match Python and Browser Token IDs

Use the exact same Unicode text and encoding name on both sides. Invisible differences such as a trailing newline, non-breaking space, smart quote, or normalization change can produce different token IDs even when two strings look similar on screen.

If the totals differ, turn on whitespace visualization, copy the raw text again, and compare the first token position where the ID lists diverge. This approach narrows the problem to the input or encoding instead of treating tokenization as a black box.

Tiktoken Count vs OpenAI API Input Tokens

Tiktoken is useful for local raw-text inspection. OpenAI's Responses API also exposes a request-level input-token counting operation at POST /responses/input_tokens. It accepts request components such as model, instructions, conversation context, text, image, or file inputs and returns an input_tokens total.

Use the API operation when you need a count for the structured request you plan to send. Use the browser or Python tokenizer when you need to understand token boundaries, inspect IDs, compare encodings, or debug why a particular string is expensive.

Tiktokenizer Reference Sources

Tokenizer behavior and model limits change. Verify production decisions with current provider documentation:

• OpenAI API: Count input tokens
• OpenAI: Count tokens for structured requests
• OpenAI tiktoken repository
• OpenAI model comparison
• OpenAI tiktoken model-to-encoding mappings
• OpenAI Cookbook: Count tokens with tiktoken
• OpenAI: Understanding and counting tokens