Tokens: The Pieces a Text Model Uses

Work through the explanation, apply it to the example, and complete the quick check before continuing.

Models do not always read whole words

Before a text model works with language, a tokenizer breaks the text into smaller pieces called tokens.

A token may be:

• A short word.
• Part of a word.
• A number.
• A punctuation mark.
• A space joined with letters.
• Another small coded piece.

One token does not always equal one word.

A simple example

Take the sentence:

The meeting starts at 10:30.

One tokenizer might split it into pieces such as:

The | meeting | starts | at | 10 | : | 30 | .

This is only an example. Another model may split the same sentence in a different way.

Now take a longer word such as unhelpful.

A tokenizer might treat it as one token. Another might split it into pieces such as un, help, and ful.

Why use pieces?

Using word pieces helps a model handle many words without needing one separate entry for every possible word.

It can reuse common parts across:

• New words.
• Different word endings.
• Names.
• Numbers.
• Several languages.

Tokenisation is a way to represent text. It does not check if the text is true.

Tokens and limits

Models have a limit on how many tokens they can process together. This is called the context window.

A token limit is not the same as a word limit. A page with code, numbers, or unusual words may use tokens differently from a page of simple prose.

We will study context windows in a later unit.

Pause and decide

Which statement is best?

A. Every token is one word. B. Tokens are pieces chosen by a tokenizer, and the pieces can differ between models. C. A token checks whether a word is correct. D. Long words always use one token.

Answer: B. Tokens are model-specific pieces. They are not always words.

Do not over-focus on counting

Token counts can matter for cost and context limits in some products. But this course is not asking you to predict the exact token count of a sentence.

The useful idea is simple:

A text model receives and creates a sequence of tokens, not a perfect list of complete words.

Remember

• Tokens are the pieces a text model uses.
• One token may be a word, part of a word, punctuation, or another piece.
• Different tokenizers can split the same text differently.
• Tokenisation represents text. It does not check facts.

Next, we will see how a model uses tokens to build a response one step at a time.