Tokens and Context Windows Explained

TL;DR

• A token is a small chunk of text, often a word or part of a word, that an AI model reads and writes instead of whole words [1].
• The context window is the most text, counted in tokens, that the model can consider at one time. It acts as the model's working memory [3][4].
• Your messages, the chatbot's replies, hidden instructions and attached documents all count towards it [3][4].
• When a chat gets too long, older content may be cut or summarised, which is why a chatbot can seem to forget early details.

What a token is #

Language models do not read letters or whole words. They split text into tokens first. A token can be a single character, a fragment of a word or a whole word, and many models use sub-word pieces, so "unwatched" can become "un", "watch" and "ed" [1]. As a rough guide for English, one token is about four characters, or about three quarters of a word [1]. Another source suggests roughly 1.5 tokens per word, so treat any rule of thumb as approximate, since it varies by model [4]. Some languages need many more tokens for the same sentence than English does [4].

What a context window is #

The context window is all the text the model can look at while it writes a response, including the response itself. It is different from the huge body of text the model was trained on: it is a working memory for the current conversation [3]. It is measured in tokens, not words or pages [2].

Window sizes differ between models and have grown quickly. One vendor's documentation says some of its current models have a window of up to 1 million tokens, while older models had 200 thousand [3]. Check the figure for the specific model you use.

What uses up the window #

Each new turn carries the earlier conversation with it, so a long chat gets heavier every message. Chat apps may drop the oldest content first, or summarise earlier parts, to keep going [3].

Bigger is not always better #

A larger window lets a model handle longer documents and conversations, but there are trade-offs. Processing more text takes more computing power and can be slower, and models may recall details in the middle of very long inputs less well than details at the start or end [4]. Vendor guidance also notes that accuracy and recall can decline as the amount of text grows, a problem sometimes called context rot [3]. What is in the window matters as much as how big it is.

Practical tips #

• Start a new chat when you change topic.
• Put key instructions and facts near the start or repeat them when a chat is long.
• Share only the relevant part of a big document.
• Ask for a short summary of a long chat and paste it into a fresh one.

FAQ #

How many words is 1,000 tokens? #

Roughly 750 English words by the three-quarters rule of thumb, though it varies by model and language [1].

Why does my chatbot forget what I said earlier? #

If the conversation outgrows the context window, older text may be cut or summarised, so the model can no longer see it [3][4].

Do tokens affect cost? #

For many AI services, usage is priced by tokens, and long conversations use more because earlier text is sent again each turn. Check the pricing of the service you use [3].

Is the context window the same as the model's knowledge? #

No. Knowledge comes from training. The window holds only what is in the current conversation [3].

Sources #