Tokens: The Pieces a Text Model Uses

Work through the explanation, apply it to the example, and complete the quick check before continuing.

Models do not always read whole words

Before a text model works with language, a tokenizer breaks the text into smaller pieces called tokens.

A token may be:

• A short word.
• Part of a word.
• A number.
• A punctuation mark.
• A space joined with letters.
• Another small coded piece.

One token does not always equal one word.

Text description of the visual

The figure shows the sentence The meeting starts at 10:30. One possible token split is shown as separate pieces: The, meeting, starts, at, 10, colon, 30, and period. A second example shows the word unhelpful split into un, help, and ful. The figure notes that this is only an example, another tokenizer may split the same text differently, and tokenization does not check facts.

A simple example

Take the sentence:

The meeting starts at 10:30.

One tokenizer might split it into pieces such as:

The | meeting | starts | at | 10 | : | 30 | .

This is only an example. Another model may split the same sentence in a different way.

Now take a longer word such as unhelpful.

A tokenizer might treat it as one token. Another might split it into pieces such as un, help, and ful.

Why use pieces?

Using word pieces helps a model handle many words without needing one separate entry for every possible word.

It can reuse common parts across:

• New words.
• Different word endings.
• Names.
• Numbers.
• Several languages.

Tokenisation is a way to represent text. It does not check if the text is true.

Tokens and limits

Models have a limit on how many tokens they can process together. This is called the context window.

A token limit is not the same as a word limit. A page with code, numbers, or unusual words may use tokens differently from a page of simple prose.

We will study context windows in a later unit.

Pause and decide

Which statement is best?

A. Every token is one word. B. Tokens are pieces chosen by a tokenizer, and the pieces can differ between models. C. A token checks whether a word is correct. D. Long words always use one token.

Answer: B. Tokens are model-specific pieces. They are not always words.

Do not over-focus on counting

Token counts can matter for cost and context limits in some products. But this course is not asking you to predict the exact token count of a sentence.

The useful idea is simple:

A text model receives and creates a sequence of tokens, not a perfect list of complete words.

Remember

• Tokens are the pieces a text model uses.
• One token may be a word, part of a word, punctuation, or another piece.
• Different tokenizers can split the same text differently.
• Tokenisation represents text. It does not check facts.

Next, we will see how a model uses tokens to build a response one step at a time.