TiktokenizerHow to Count Claude Sonnet 5 Tokens
For a private estimate without an API key, paste text into the editor. The local rule uses Anthropic's approximate 3.5 English characters per token for earlier tokenizers, with about 30% added for the newer tokenizer. This is a planning heuristic and does not reconstruct token boundaries or IDs.
For a model-aware estimate, call Anthropic's Messages token counting endpoint with model claude-sonnet-5 and the same supported system prompt, messages, client tools, and base64 images or PDFs you plan to send. The response returns an estimated input_tokens value before message generation.
Anthropic describes this as a preflight estimate: actual input usage can differ slightly, and system-added optimization tokens in the estimate are not billed. Use it for prompt planning, then read the Messages response usage for the actual request. The local character estimate is separate from this model-aware endpoint and should not be used as an exact context or billing boundary.
import anthropic
client = anthropic.Anthropic()
result = client.messages.count_tokens(
model="claude-sonnet-5",
system="You are a concise assistant.",
messages=[{"role": "user", "content": "Count this prompt."}],
)
print(result.input_tokens)Claude Sonnet 5 Uses a Newer Tokenizer
Anthropic states that Claude Sonnet 5 uses the newer tokenizer also used by Claude Opus 4.7 and later Opus models, Claude Fable 5, and related current models. The same text produces approximately 30 percent more tokens than on earlier Claude models, although the exact change depends on the content.
That does not mean Sonnet 5 is always more expensive or less efficient; it means token-based limits and prices must be evaluated with Sonnet 5 counts. Recount real prompts rather than copying a Sonnet 4.6 token total into a new budget.
What count_tokens Includes
| Request component | Supported by preflight count | Caveat |
|---|---|---|
| System and messages | Yes | Use the exact conversation structure |
| Client tools and advisor tool | Yes | For the advisor tool, only the executor's first sampling call is counted |
| Other server tools and MCP servers | No | Read actual usage from the Messages response |
| Images and PDFs | Base64 sources | URL and file sources are not supported by count_tokens |
| Generated answer | No | Read output usage after generation |
| System-added optimization tokens | May appear | Anthropic says these are not billed to you |
Claude Sonnet 5 Context Planning
Claude Sonnet 5 supports a one-million-token context window and up to 128,000 output tokens. Because the newer tokenizer represents the same source text with more tokens on average than Sonnet 4.6, the effective amount of text that fits in a fixed token budget can be lower.
Leave headroom for the answer and for request components that are easy to overlook, especially tool schemas, documents, and long multi-turn histories. Run count_tokens again whenever the model ID, system prompt, tool set, or document set changes.
Tiktokenizer Reference Sources
Tokenizer behavior and model limits change. Verify production decisions with current provider documentation: