Tokens: how a model reads text

Models do not read letters or whole words. They read tokens — small chunks of text, often a word or part of a word. Before your message reaches the model, a tokenizer splits it into tokens and turns each one into a number.

What tokens look like

Common words are often one token. Longer or rarer words are split into several. Spaces and punctuation usually belong to tokens too. An illustration:

Copy code"Android engineers love Kotlin"  →  [Android] [ engineers] [ love] [ Kot] [lin]

The exact split depends on the model. Two models can split the same sentence differently, so the same text can be a different number of tokens.

As a rough rule of thumb for English, one token is about four characters, or about three-quarters of a word. That is only an estimate. Code, numbers, and many non-English languages often use more tokens for the same meaning.

Why tokens matter to you

Tokens are the unit everything is measured in:

• Cost: providers usually charge per token — separately for the tokens you send (input) and the tokens the model writes (output).
• Speed: the model produces output one token at a time, so longer answers take longer.
• Limits: each model can only handle a certain number of tokens at once. That limit is the context window — the next lesson.

A quick estimate in code

Copy code# academy-check
def estimate_tokens(text):
    """Rough English estimate only. Use the model's real tokenizer for accurate counts."""
    return max(1, round(len(text) / 4))

prompt = "Summarise this support ticket in two sentences."
print(estimate_tokens(prompt))           # about 12
assert estimate_tokens("Hi") == 1
assert estimate_tokens("a" * 400) == 100

This is fine for a back-of-the-envelope budget. When you need exact numbers — for billing or hard limits — use the tokenizer or token-counting tool your model provider supplies, and read the token counts the API returns with each response.

Try it: paste a paragraph into your provider's token counter (most offer one). Compare an English paragraph with the same paragraph translated into another language.

Quick check: why might the same question cost more in one language than another? (It can split into more tokens, and you pay per token.)