Understand what a generator does

The library team has found the correct renewal passage. Someone now suggests replacing the plain quotation with “a language model that writes nicely.” Before we accept that change, we need to know what new decision enters the system. Choosing the next piece of text is a different operation from selecting a source.

Let us begin with a generator small enough to account for by hand. It will not answer library questions. It will help us see why a likely continuation is not the same as a supported statement.

Learn a tiny continuation table

Our invented training collection has three sentences: “desk opens weekdays,” “desk opens weekdays,” and “desk closes holidays.” Repetition is intentional. After the word desk, we observed opens twice and closes once. A count-based estimate assigns the next word probabilities 2/3 and 1/3. After opens, only weekdays was observed, so its estimated probability is 1. A special end token records when a sentence stops.

The program builds these counts in train_bigram. A bigram model uses just the previous word to predict the next. It deliberately discards earlier history. This is a small actual language model, because it estimates a next-word distribution from data; it is not a neural model, an LLM or a RAG answerer. The foundational terminology is explained by Jurafsky and Martin's n-gram chapter.

The learning step is training: sentences change the table. Reading the table to produce a continuation is inference. If we add ten new examples of “desk closes holidays,” retraining changes the distribution. Asking the same existing table a new question does not retrain it. This distinction will matter when a source changes: updating a retrieval collection and changing a model's parameters are different operations.

From project/, run:

python3 -B methods.py decoding

Read greedy.trace one row at a time. Starting after desk, the model chooses opens, then weekdays, then the end token. The returned tokens contain opens and weekdays; the supplied starting history is not repeated. The output is a continuation, not a claim that the desk is open on every weekday in a real library. Our three training sentences never established such a policy.

Decide how to choose from a distribution

Greedy decoding chooses the largest next-token probability at each step. In our code, equal counts resolve alphabetically, making ties reproducible. Greedy choices are local: choosing the most likely next word does not establish the most likely complete sentence, and neither criterion verifies a sentence against a handbook.

Sampling draws a token using weights from the distribution. The seeded sample in the demo selects closes holidays instead. The code uses a local random-number generator, so another part of the program does not disturb its seed. A seed makes this local experiment replayable in the tested environment; it is not a promise about reproducibility of a separate hosted neural service.

Temperature changes the relative weights before sampling. We compute softmax(log(count) / T). Equivalently, positive counts are raised to 1/T and normalized. For counts 2 and 1, T=1 gives 2/3 and 1/3. At T=0.5, the weights become 4 and 1, giving 0.8 and 0.2. At T=2, they become approximately 1.414 and 1, giving about 0.586 and 0.414. Lower positive temperatures sharpen this distribution; higher values flatten it. Temperature zero is not accepted by this arithmetic function; use the separate greedy path.

Changing temperature cannot add a missing fact to the training table. Both available continuations came from the same tiny collection. The unseen history laptop returns unseen_history; no fallback or smoothing is implemented. Every run also has a step limit so a cycle cannot run forever. A stopping limit controls execution length, not factual quality.

Move carefully from words to tokens

Our toy model extracts lowercase English words. Modern language models generally operate on tokens produced by their own tokenizer, which can divide words into smaller units and represent punctuation or other symbols. A model's context window bounds the token sequence it can process; the exact limit and accounting depend on that model and interface. We do not assume a character-to-token conversion or a current provider limit in this course.

Consider the string renewals-v1. Our toy word extractor loses the hyphen and number. A different tokenizer may preserve or split them differently. If a later experiment adds a real model, record its actual tokenizer, count the real serialized input and reserve space for output. Our current character budgets are intentionally labeled character budgets. They cannot prove an input fits an unknown model.

See the arithmetic behind an attention connection

The Transformer introduced a way to combine information using attention rather than recurrence as its central sequence mechanism. Its learned representations and multiple layers are much richer than a word-count table. Vaswani et al., 2017 is the primary architectural reference; we are not implementing a Transformer here.

The demo includes a deliberately small attention-like weighted average. A scalar query is 1; two keys are 0 and log(3). Their scores are 0 and log(3). Exponentiation gives weights proportional to 1 and 3, so normalization gives 0.25 and 0.75. Applied to values 10 and 20, the weighted result is 0.25×10 + 0.75×20 = 17.5.

This makes one mechanism concrete: information from several positions can contribute to a new representation with different weights. Our chosen numbers are manual inputs, not learned parameters or meaningful language embeddings. A large attention weight is not a source citation, a causal explanation or proof that a generated claim follows from a passage. We have shown arithmetic, not trained reasoning.

Return to the renewal problem

If a later model receives both a question and the renewal passage, the supplied context can affect its continuation. That does not force it to keep every exception. It might omit a condition, add a common policy remembered from training, or produce a correct citation next to an unsupported claim. Those are hypotheses to test with actual outputs when a real model experiment is authorized, not results from this toy program.

For this local course, the executable route remains quote copying and, later, an explicitly human-authored template. We can study source selection, context assembly and review without buying access or pretending we ran a neural model. Complete the practice, then compare the explained solutions. Continue when you can distinguish training, inference, decoding and evidence support in ordinary language.