How We Count OpenAI Tokens
TokenCounts counts OpenAI tokens by calling the official Responses API `input_tokens` endpoint. We send the exact prompt payload (including role and modal parameters) to OpenAI servers and return the true integer length calculated by OpenAI's active tokenizers.
OpenAI Tokenization Methodology details
We avoid relying strictly on local BPE libraries (like tiktoken) because they do not account for prompt framing overhead, which we details below.
| Factor / Area | How OpenAI Counts It | TokenCounts Handling |
|---|---|---|
| Text Inputs | Splits characters using BPE vocabulary bases (e.g. cl100k_base, o200k_base). | Sends text inside User messages via the Responses endpoint to fetch the official API count. |
| Chat Formatting Overhead | Adds system wrapper tokens (such as <|im_start|>) per message. Usually 3-4 buffer tokens per turn. | Automatically calculated by OpenAI and included in our returned API count total. |
| Image Inputs | Vision tokens scale depending on resolution (e.g. 85 tokens per tile for low, 170 per tile for high detail). | Passes the base64 image data inside the content array and returns OpenAI's actual visual token total. |
| PDF Attachments | Extracts plain text and processes visual pages. PDF pages yield fixed visual tokens. | Submits the document block as inline data to return the exact multimodal count. |