Unit 12.00: Capstone problem framing

The framing decides what a good model even means. Write it before touching the data, or you will optimise a number nobody asked for.

Four questions, answered in writing

What decision does this support? What is the unit of prediction? What is the target, and what is genuinely known at prediction time? And what does success look like, compared against what?

The last pair is the one people skip — the asymmetry between error types:

import torch

# A framed problem answers four questions before any model exists.
frame = {
    "decision": "flag learners at risk of dropping out, for a human to review",
    "unit_of_prediction": "one learner, once per week",
    "target": "did not open a lesson in the following 14 days",
    "available_at_prediction_time": ["activity in the last 7 days", "progress so far"],
    "NOT_available": ["anything recorded after the prediction week"],
    "success_looks_like": "beats the majority baseline AND beats logistic regression",
    "cost_of_a_false_positive": "a learner is contacted unnecessarily -- cheap",
    "cost_of_a_false_negative": "a learner drops out unnoticed -- expensive",
}
for key, value in frame.items():
    print(f"{key:28}: {value}")

# The asymmetry in that last pair decides your threshold. If a miss costs more
# than a false alarm, 0.5 is the wrong cut-off, and choosing it by default is a
# decision you made without noticing.
print("\nWrite this down before touching the data. A model trained without it")
print("optimises a number nobody asked for.")

Look at the two cost lines. A false positive contacts someone unnecessarily; a false negative lets a learner drop out unnoticed. Those costs are not equal, so a 0.5 decision threshold is a choice, and usually the wrong one.

The NOT_available line is the leakage guard: anything recorded after the prediction moment cannot be a feature.

The mistake this prevents

Choosing a threshold of 0.5 by default. It is the right cut-off only when the two error types cost the same, which is rarely true and almost never checked.

Takeaway

Frame the problem in writing first. The costs of each error type decide the threshold, and the threshold decides what the model does.

Mark complete