Unit 10.04: What to keep from a trace, and for how long

Traces are a second copy of your most sensitive data, in a system built for debugging and usually reviewed less carefully than the primary one.

Retention per field, not per trace

Metrics and versions can be kept indefinitely. Content cannot.

The code sorts seven trace fields by retention period.

FIELDS = [
    ("run_id, timestamps, durations", "indefinite", "no personal data"),
    ("token counts and costs",        "indefinite", "aggregate metrics"),
    ("prompt/model/policy versions",  "indefinite", "needed to explain old runs"),
    ("retrieved chunk ids and scores", "90 days",   "debugging window"),
    ("resolved prompt text",          "30 days",    "contains user content"),
    ("full model output",             "30 days",    "contains user content"),
    ("raw user question",             "30 days",    "personal data"),
]

print(f"{'field':34} {'keep':12} why")
for field, keep, why in FIELDS:
    print(f"{field:34} {keep:12} {why}")

short = sum(1 for _, k, _ in FIELDS if k != "indefinite")
print(f"\n{short} of {len(FIELDS)} fields carry content and need a limit")

# Traces are a second copy of your most sensitive data, in a system built for
# debugging and usually reviewed less carefully than the primary one. Split the
# retention by field rather than keeping everything for the longest period any
# field needs.

Three of seven fields carry no personal data and are worth keeping forever: durations, token counts, and versions. Keeping versions indefinitely is what lets you explain a run from a year ago after the prompt has changed six times.

Four carry content and need a limit. Splitting retention by field means you keep the long-term metrics you want without keeping the user's question for three years - which is what a single trace-level retention policy forces you to choose between.

The mistake this prevents

The mistake is one retention period for the whole trace, set to whatever the longest-needed field requires. That keeps raw user questions for as long as you want cost metrics, which is almost never defensible.

Takeaway

Split trace retention by field. Metrics and versions indefinitely, content on a short limit - a single policy forces you to over-retain the sensitive part.

Mark complete