Unit 08.01: Trimming history before the window decides

If you do not trim history, the context window trims it for you, and it does so without regard for what mattered.

Keep the system message, start on a human turn

trim_messages drops old turns under a budget. Two of its options matter more than the budget itself.

The code trims thirteen messages down under a token budget.

from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, trim_messages

messages = [SystemMessage("You answer from the policy text only.")]
for i in range(1, 7):
    messages.append(HumanMessage(f"question {i} about billing and refunds"))
    messages.append(AIMessage(f"answer {i} referencing the policy document"))

trimmed = trim_messages(
    messages, max_tokens=6, strategy="last",
    token_counter=len, include_system=True, start_on="human")

print(f"before: {len(messages)} messages")
print(f"after : {len(trimmed)} messages")
for m in trimmed:
    print(f"   [{m.type:6}] {m.content[:44]}")

# Trimming deliberately beats letting the context window truncate for you.
# `include_system=True` keeps the instructions, and `start_on="human"` keeps
# the sequence valid -- dropping to an AI message first confuses the model.

include_system=True keeps your instructions. Without it, the first thing dropped is the system message - the one piece of context that should never go - because it is the oldest.

start_on="human" keeps the sequence valid. A history beginning with an AI message reads as the assistant having spoken unprompted, which confuses models and produces odd continuations.

The mistake this prevents

Summarising older turns instead of dropping them is the usual next idea, and it trades one loss for another. A summary keeps the gist and loses the specific: the account number, the exact date, the figure the user expects the assistant to still know. That is precisely the detail extraction into fields preserves - so summarise the narrative if you like, and keep the established facts as fields alongside it rather than inside it.

The mistake is trimming by message count rather than by tokens. Messages vary enormously in length, so ten messages might be two hundred tokens or twenty thousand - and the count-based version passes every test until someone pastes a document.

Takeaway

Trim deliberately by tokens, keep the system message, and start on a human turn. Letting the window truncate drops your instructions first.

Mark complete