Tiktoken encoding comparison
o200k_base vs cl100k_base
Compare two major OpenAI BPE encodings. Paste the same prompt, switch encodings, and inspect changes in token boundaries and IDs.
Accuracy note
Exact raw-encoding output. Model API requests can add message formatting and other tokens beyond raw text.
Frequently asked questions
What is o200k_base?
o200k_base is a newer OpenAI tiktoken encoding used by recent model families.
What is cl100k_base?
cl100k_base is an earlier OpenAI encoding used by several GPT and embedding model generations.
Can token IDs be compared directly?
Token IDs belong to a specific vocabulary. The same number does not necessarily represent the same text across different encodings.