Unit 07.01: Trimming before the context window decides

If you do not trim, the context window trims for you, and it starts with your instructions.

Keep the system message, start on a user turn

A trim that respects both rules.

The code trims thirteen messages to a budget.

messages = [{"role": "system", "content": "Answer from the policy only."}]
for i in range(1, 7):
    messages.append({"role": "user", "content": f"question {i}"})
    messages.append({"role": "assistant", "content": f"answer {i}"})

BUDGET = 5


def trim(msgs, budget):
    system = [m for m in msgs if m["role"] == "system"]
    rest = [m for m in msgs if m["role"] != "system"][-budget:]
    while rest and rest[0]["role"] != "user":
        rest.pop(0)
    return system + rest


trimmed = trim(messages, BUDGET)
print(f"before: {len(messages)} messages")
print(f"after : {len(trimmed)} messages")
for m in trimmed:
    print(f"  [{m['role']:9}] {m['content']}")

# Two rules the trim has to respect: keep the system message, and start on a
# user turn. Letting the context window truncate for you drops the system
# message first, because it is oldest.

Keeping the system message is the first rule because it is oldest, and a naive trim drops it first - removing every rule the app depends on while keeping six turns of small talk.

Starting on a user turn keeps the sequence valid. A history that begins with an assistant message reads as the model having spoken unprompted, and produces odd continuations.

The mistake this prevents

The mistake is trimming by message count rather than by tokens. Messages vary enormously in length, so ten messages might be two hundred tokens or twenty thousand - and the count-based version passes every test until someone pastes a document.

Takeaway

Trim deliberately by tokens, always keeping the system message and starting on a user turn. Letting the window truncate drops your instructions first.

Mark complete