Guide

AI Tokens Explained

A plain-English guide to tokens in AI: what a token is, how many words a token is, how tokenisers split text, what a context window is, and why tokens cost money, with worked examples.

November 3, 2026 · The AI Rankings

Quick answer: A token is the small chunk of text an AI language model actually reads and writes: usually a whole common word, part of a longer word, a number or a punctuation mark. In English, one token is about four characters or three-quarters of a word, so 1,000 tokens is roughly 750 words, according to OpenAI. Tokens matter because everything is measured in them: the context window (how much a model can hold at once), the maximum answer length, rate limits and the API bill, which charges separately for input and output tokens. The one caveat: every company uses its own tokeniser, so the same text is a different number of tokens on different models, and Anthropic says its newer Claude models produce about 30% more tokens for the same text than its older ones.

This page is the explainer for tokens, context windows and token pricing. It does not rank models. For how language models work end to end, see what is an LLM. For the current API price comparison, see best LLM APIs, and for which model is strongest, see best AI models. For one-line definitions of the rest of the vocabulary, see the AI glossary.

What is a token in AI?

A token is the basic unit of text a language model processes. Before a model sees your prompt, a tokeniser splits it into tokens from a fixed vocabulary and turns each one into a number, called a token ID. The model reads token IDs, predicts the next token ID, and the app turns those IDs back into text.

A token is not the same as a word. Short, common words are usually one token each. Longer or rarer words are split into several pieces, so a word such as “unbelievable” might become “un”, “believ” and “able”. Spaces, punctuation, numbers and emojis are tokens too, and a leading space is often folded into the token that follows it.

Models work in tokens rather than words or letters because tokens are a compromise between the two. A vocabulary of whole words could never cover every name, typo and new coinage. Working letter by letter would make every sentence several times longer, which makes models slower and more expensive to run. Sub-word tokens keep text short while still being able to spell out anything.

Tokens vs words vs characters

The vendors publish slightly different rules of thumb for English, because each uses its own tokeniser.

The working conversion most people use is 1,000 tokens to about 750 English words, or 1 word to about 1.33 tokens.

How many tokens is that? Quick reference

Treat these as estimates. The exact count depends on the model, the language and the kind of text: code, tables, URLs and numbers usually take more tokens per character than ordinary prose.

How tokenisation works

Tokenisation is the step that converts text into tokens. Most modern language models use a method based on byte pair encoding (BPE).

Byte pair encoding

BPE started life as a data-compression technique. Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh adapted it for language models in “Neural Machine Translation of Rare Words with Subword Units”, handling rare words “by encoding rare and unknown words as sequences of subword units”.

The idea is simple:

1. Start with every individual character (or byte) as its own token.
2. Look through a huge amount of training text and find the pair of tokens that appears next to each other most often, such as “t” and “h”.
3. Merge that pair into a new token, “th”, and add it to the vocabulary.
4. Repeat, merging the next most common pair, until the vocabulary reaches the size the designers chose.

After enough merges, common words such as “the” and “and” become single tokens, common word pieces such as “ing” and “tion” become tokens, and anything unusual can still be spelled out from smaller pieces. Modern tokenisers work on raw bytes, so any text in any language, including emojis and code, can be encoded without an “unknown word” gap.

Why the same text gives different counts on different models

Each model family trains its own tokeniser on its own data, with its own vocabulary size. That means:

• Counts differ between companies. A prompt that is 10,000 tokens on an OpenAI model will not be exactly 10,000 tokens on a Claude or Gemini model.
• Counts can change between versions from the same company. Anthropic’s token-counting documentation says Claude 4.7 and later models use a newer tokeniser and that “the same input text produces approximately 30 percent more tokens than on earlier models”. Anthropic tells developers to recount prompts on the model they plan to use rather than reuse old counts.
• Price per token is not price per word. Because token counts differ, the cheapest price per million tokens is not automatically the cheapest price for the same job. Compare the cost of a real request, not just the headline rate.

Tokens in other languages

Tokenisers are trained mostly on English, so other languages often need more tokens to say the same thing. OpenAI notes that the Spanish phrase “Cómo estás” is 5 tokens for 10 characters. A study by Aleksandar Petrov and colleagues found that the same content can take up to 15 times more tokens in some languages than in English. The study’s authors point out three knock-on effects: higher cost on per-token APIs, slower responses, and less text fitting into the context window.

Why tokens cause odd failures

A model sees token IDs, not letters, so tasks that depend on individual characters are harder for it than they look. Counting the letters in a word, reversing a string, rhyming and some arithmetic on long numbers are classic weak spots, because a word or number may arrive as one or two opaque chunks. A model that handles hard reasoning well can still miscount letters in a single word. Giving the model a code tool to do the counting fixes most of these cases.

Do images, audio and video use tokens?

Yes. Multimodal models convert images, audio and video into tokens too, and those tokens count towards the context window and the bill in the same way as text.

At Claude Opus 5.5’s input price of $4 per million tokens, a 1,000 × 1,000 image costs about half a cent. Video is the big one: an hour of footage on Gemini fills most of a 1-million-token context window on its own.

What is a context window?

A context window is the maximum number of tokens a model can work with in a single request, covering everything it reads and everything it writes. Anthropic’s context-window documentation describes it as the model’s “working memory”, as distinct from the data the model was trained on.

What counts towards the context window

• The system prompt, the hidden instructions an app or developer gives the model.
• The whole conversation so far. Every earlier message and reply is sent again with each new turn, so a long chat fills the window as it goes.
• Attached files, images, audio and tool results, such as a PDF, a web search result or the output of a piece of code.
• Tool definitions, the descriptions of tools the model is allowed to call.
• The model’s answer, including any hidden reasoning tokens.

Caching does not free up space. Anthropic notes that cached content still occupies the window: caching “changes what you pay for those tokens, not whether they count”.

How big are context windows?

A 1-million-token window holds roughly 750,000 words by OpenAI’s rule of thumb, which is several long novels or a mid-sized codebase. Maximum output is a separate and much smaller limit, so a model can read a whole book in one go but cannot rewrite all of it in one reply.

What happens when you hit the limit

What happens depends on the app or API.

• In an API, an oversized prompt is rejected. Anthropic’s API returns a “prompt is too long” error if the input alone exceeds the window. If the input fits but the answer runs out of room, newer Claude models stop and report that the context window was exceeded (Anthropic).
• In chat apps, older material is summarised or dropped. Long conversations in consumer apps keep going by compressing or trimming earlier turns, which is why a chatbot can “forget” something you said hours earlier. Anthropic offers this to developers as compaction, which “automatically summarizes earlier parts of the conversation” so it can continue past the limit.
• On some models, a long prompt costs more per token. OpenAI bills GPT-6.1 Sol prompts over 272,000 input tokens at twice the input rate and 1.5 times the output rate for the whole request (OpenAI). Claude Haiku 5.5 charges five times its base rates once a prompt passes 100,000 tokens (Anthropic). Other current Claude models bill the full 1M window at one flat rate.

Bigger is not always better: context rot

A large context window is a capacity, not a guarantee that the model uses all of it equally well. Anthropic’s own documentation says that “as token count grows, accuracy and recall degrade, a phenomenon known as context rot” (Anthropic). Chroma’s study of 18 models found the same pattern across vendors. In practice, the most relevant few thousand tokens usually beat the whole archive: trimming what goes in, or retrieving only the passages that matter (see what is RAG), tends to give more accurate answers and cheaper requests.

Why tokens cost money

API access to a language model is priced per million tokens, with separate rates for input tokens (everything you send) and output tokens (everything the model writes). Chat subscriptions hide this behind a monthly fee and usage limits, but the same token maths sits underneath.

Why output costs more than input

Output tokens cost more because they are more work to produce. A model reads all your input tokens in one parallel pass, but it writes output one token at a time, with a full pass through the model for each new token. Every current model on the price table below charges five times as much per output token as per input token. The what is an LLM guide explains the prefill and decode phases behind this in more detail.

What tokens cost on current models

Standard API prices per million tokens, from each provider’s own pricing page.

For the full provider comparison, including gateways, open-weight hosts and free tiers, see best LLM APIs.

Worked example: summarising a report

Summarising a 15,000-word report (about 20,000 input tokens) into a 1,100-word summary (about 1,500 output tokens), using the prices above (the introductory prices for the two Gemini models) and assuming the same token count on every model:

Two things change the real number. First, token counts differ between tokenisers, so the same report will not be exactly 20,000 tokens on every model. Second, reasoning models add thinking tokens to the output side, covered next.

Reasoning tokens: the hidden output

Reasoning models write a chain of intermediate “thinking” before their visible answer, and that thinking is made of tokens you pay for. OpenAI’s reasoning guide says reasoning tokens are not visible through the API but “are billed as output tokens” and take up space in the context window. Google labels its output price “including thinking tokens” (Google), and Anthropic says Claude’s thinking tokens are billed as output (Anthropic).

In the report example, if GPT-6.1 Sol thinks for 8,000 tokens before writing its 1,500-token summary, output becomes 9,500 tokens and the cost per report rises from $0.055 to $0.135, about two and a half times as much. That is why cost per completed task is the fairer way to compare reasoning models than price per token. OpenAI recommends reserving at least 25,000 tokens for reasoning and output when starting out with its reasoning models.

Caching and batch discounts

Two discounts cut token bills more than switching models usually does.

• Prompt caching charges much less for input the provider has seen recently, such as a long system prompt or a document you ask several questions about. On Claude Opus 5.5, writing to Anthropic’s default 5-minute cache costs $5 per million tokens and reading from it costs $0.20, against $4 for normal input (Anthropic). OpenAI and Google list cached-input rates on their pricing pages, shown in the table above.
• Batch processing gives 50% off input and output on Anthropic’s Batch API for jobs that can wait for an answer (Anthropic).

A caching example: asking 100 questions about the same 50,000-token contract on Claude Opus 5.5 costs $20.00 in contract input tokens without caching (100 × 50,000 tokens at $4 per million). With caching, it costs $0.25 to write the contract to the cache once and $0.99 to read it 99 more times, $1.24 in total, as long as each question arrives within five minutes of the last.

Rate limits are counted in tokens too

API providers cap how fast you can send tokens as well as how much they cost. OpenAI’s rate limits include tokens per minute (TPM) and tokens per day alongside request limits, and the limit counts the larger of your max_tokens setting and the estimated tokens in the request, so setting max_tokens far higher than you need uses up headroom.

How to count tokens

You do not need to guess. Each major provider offers a way to count tokens before you send a request.

After a request, every API response reports the input, output and (where relevant) reasoning tokens actually used, which is the number you are billed on.

How to use fewer tokens

• Send only what the model needs. Paste the relevant section, not the whole document, or use retrieval to pull in the passages that matter. Smaller inputs are cheaper and less exposed to context rot.
• Start fresh chats for new topics. Every message resends the conversation so far, so a long chat costs more per turn and pushes older material towards the edge of the window.
• Ask for the length you want. A one-paragraph answer costs a fraction of a one-page answer, and output tokens are the expensive ones.
• Match the model to the job. Routine work on a $0.10 / $0.50 model costs 1/40th of the same job on a $4 / $20 model at the same token count.
• Turn reasoning down for easy tasks. Many models let you set a lower thinking effort, which cuts hidden output tokens.
• Cache repeated context and batch work that can wait. On Anthropic’s API, those two discounts can stack.
• Resize images before sending them. On Claude, image tokens scale with pixel count, so a smaller image is a cheaper image.

Frequently asked questions

What are tokens in AI?

Tokens are the small chunks of text an AI language model reads and writes, usually a common word, part of a longer word, a number or a punctuation mark. A tokeniser splits your prompt into tokens before the model sees it. Context windows, answer length limits, rate limits and API prices are all measured in tokens, not words.

How many words is 1,000 tokens?

1,000 tokens is about 750 English words by OpenAI’s rule of thumb, and 600 to 800 words by Google’s. The figure varies by model and language: Anthropic says its Claude Opus 4.7 and later models produce about 30% more tokens for the same text than earlier Claude models, and many non-English languages need more tokens per word than English.

How much text fits in 1 million tokens?

One million tokens is roughly 750,000 English words by OpenAI’s rule of thumb, which is several long novels or a mid-sized codebase. On a tokeniser that produces more tokens per word, such as the one in newer Claude models, the same window holds less text. One hour of video on Google’s Gemini uses close to 950,000 tokens.

What is a context window?

A context window is the maximum number of tokens an AI model can handle in one request, including the system prompt, the conversation so far, attached files, tool results and the model’s own answer. Current frontier models from Anthropic, OpenAI and Google have context windows of about 1 million tokens. The maximum length of a single answer is a separate, smaller limit, such as 128,000 tokens on Claude Opus 5.5 and GPT-6.1 Sol.

What happens when you exceed the context window?

An API rejects a prompt that is larger than the model’s context window, or stops the answer early if the output runs out of room. Chat apps usually keep long conversations going by summarising or dropping older messages, which is why a chatbot can lose track of something said much earlier in a long chat.

Is a bigger context window always better?

A bigger context window lets a model take in more at once, but accuracy can fall as the input grows, an effect called context rot. Anthropic’s documentation says accuracy and recall degrade as token count grows. Sending the most relevant material, rather than everything, usually gives better and cheaper answers.

Why are output tokens more expensive than input tokens?

Output tokens cost more because a model reads all input tokens together in one parallel pass but has to generate output one token at a time, with a full pass through the model for each. On every model in this guide’s price table, output tokens cost five times as much as input tokens.

Do reasoning tokens cost money?

Reasoning tokens cost money and are billed as output tokens, even when the thinking is hidden from you. OpenAI, Google and Anthropic all bill thinking as output. A reasoning model that thinks for thousands of tokens before answering can cost several times as much per task as its price per token suggests.

How much does 1 million tokens cost?

One million tokens costs anywhere from $0.10 to $10 for input and from $0.50 to $50 for output on current models from the major providers. GPT-6 Luna and Claude Haiku 5.5 sit at the cheap end at $0.10 input and $0.50 output, Claude Opus 5.5 charges $4 and $20, and GPT-6 Astra and Claude Fable 5.1 charge $10 and $50. Cached input and batch processing cost less.

Why do different AI models count tokens differently?

Each AI company trains its own tokeniser with its own vocabulary, so the same sentence splits into a different number of tokens on different models. Counts can even change between versions from one company: Anthropic’s tokeniser for Claude Opus 4.7 and later produces about 30% more tokens for the same text than the one before it. Compare the cost of a real request, not just the price per token.

Do images use tokens?

Images use tokens and count towards both the context window and the bill. On Claude, an image costs about its width times its height in pixels divided by 750, so a 1,000 by 1,000 image is about 1,334 tokens. On Gemini, a small image is 258 tokens and larger images are split into 258-token tiles; audio is 32 tokens per second.

How do I count tokens?

You can count tokens with OpenAI’s Tokenizer tool or its tiktoken library, Anthropic’s free count_tokens endpoint, or the countTokens method in Google’s Gemini API. Each counts with its own company’s tokeniser, so use the tool that matches the model you plan to use. Every API response also reports the exact tokens used, which is what you are billed on.

← All guides