Unit 12.02: Retention that differs by field

Splitting retention by field lets you keep the metrics without keeping the content.

Three fields forever, three on a short limit

Six field groups with a retention period and a reason.

The code sorts them.

FIELDS = [
    ("request_id, timestamps, latency", "indefinite", "no personal data"),
    ("token counts and cost",           "indefinite", "aggregate metrics"),
    ("model and prompt version",        "indefinite", "needed to explain old runs"),
    ("redacted prompt",                 "30 days",    "still contains free text"),
    ("model response",                  "30 days",    "may repeat user content"),
    ("user id",                         "30 days",    "links everything to a person"),
]
print(f"{'field':36} {'keep':12} why")
for field, keep, why in FIELDS:
    print(f"{field:36} {keep:12} {why}")

short = sum(1 for _, k, _ in FIELDS if k != "indefinite")
print(f"\n{short} of {len(FIELDS)} need a limit")

# Splitting retention by field means you keep cost and latency metrics forever
# without keeping the user's question for three years. One policy for the whole
# record forces you to choose between those.

Request ids, timings, costs and versions carry no personal data and are worth keeping indefinitely - keeping the versions is what lets you explain a run from a year ago after the prompt has changed six times.

The content fields need a limit. One policy for the whole record forces you to choose between keeping cost metrics for a year and keeping the user's question for a year.

The mistake this prevents

The mistake is one retention period set to whatever the longest-needed field requires. That keeps raw user questions for as long as you want aggregate metrics, which is almost never defensible.

Takeaway

Set retention per field. Metrics and versions indefinitely, content on a short limit - a single policy forces you to over-retain the sensitive part.

Mark complete