Capstone A defensible operational decision brief
Your commission
Advise the fictional Northstar Dispatch manager on two linked decisions: what operational question to investigate next, and whether the checklist merits a real controlled field pilot. A mean task-time reduction of at least 2 minutes is the supplied hypothetical worthwhile threshold. The manager has not supplied implementation costs, safety outcomes or quality measures. Do not invent them as observed evidence.
Use the supplied 240-shipment fixture, its raw export and the separate 40-participant simulated trial. All evidence is synthetic. The goal is to show reasoning that would transfer to genuine work, without making any claim about an actual company or person.
Deliverables and stages
- Plan: one-page analysis contract specifying population, unit, period, outcomes, contrast, missingness policy, a 2-minute practical threshold and one exploratory question. Save it before opening the reference answer.
- Audit and calculation record: raw-to-clean reconciliation, spot-check of five records, main descriptive summaries, missing-rating counts by late status, group comparability, trial effect/CI/test and interpretation. Use calculator workings, a spreadsheet with visible formulas, or your own Python script. Preserve the original files.
- Two original charts: one directly showing the trial difference and 95% interval against zero and the worthwhile threshold; one answering your chosen shipment question. Include underlying plotted values and a written equivalent for each.
- Management brief: 250–400 words with recommendation, evidence, uncertainty, the most consequential limitation and a concrete next step. Include source filenames and scope without overwhelming the headline.
- Methods appendix and review: explain assumptions, rejected/duplicate handling, quartile and SD conventions if used, rounding, exploratory choices, software versions if used and how to reproduce the result. Include a self-review and one peer challenge or your own skeptical challenge.
The optional learner_analysis.py starter and check_learner.py verify a small subset of your calculations. They are not substitutes for these deliverables. You may use the supplied distribution-function results for the trial if working without a statistical package, but must reconstruct the summaries and explain the method.
Rubric 100 points
| Area | Strong evidence | Points |
|---|---|---|
| Question and design | Explicit populations, units, period, outcomes, contrast and distinction between description and simulated experiment | 15 |
| Data integrity | Preserved raw file, 248→240 reconciliation, correct exclusions and normalization, missingness/denominators visible | 15 |
| Descriptive analysis | Correct summaries with units, subgroup counts, route/service comparability and selected exploratory question | 15 |
| Inference and effect | Correct signed difference, compatible Welch CI/test, assumptions, correct interval/p interpretation and practical-threshold comparison | 20 |
| Visual communication | Two question-matched charts, honest scales, labels/counts, readable text equivalents and exact-value support | 10 |
| Decision brief | Audience-appropriate recommendation traceable to evidence, proportionate uncertainty and actionable next measurement | 15 |
| Reproducibility and review | Inspectable calculations/code, environment and rounding details, skeptical review and documented revisions | 10 |
| Total | 100 |
Suggested completion target: at least 80/100 with no critical integrity failure. Score each area by the proportion of criteria convincingly met, and write one concrete improvement for every deduction. Another reviewer may reasonably differ on style points; calculations and inferential meaning should reconcile.
Critical failures requiring revision before completion: altering raw evidence to obtain preferred results; silently omitting valid records or unknowns; using duplicated/clustered rows as independent participants; claiming the simulated trial proves a real effect; causal hub attribution from the observational comparison; treating a p-value as the probability a hypothesis is true; describing a mean-confidence interval as the range containing 95% of individuals; hiding a material limitation to make the recommendation stronger.
Review questions
- Can every percentage be traced to numerator and denominator?
- Are the two files' units and study designs kept distinct?
- Would the conclusion change if the true reduction were 0.6 minutes rather than 3.2?
- Did the chart show the comparison the sentence actually makes?
- Are missing ratings and group-mix differences visible?
- What new evidence could reverse your recommendation?
- Can another learner reproduce the important numbers without asking which records you used?
The proposed rubric has not been calibrated on a learner cohort. It supports feedback and revision, not certification or a claim of professional statistical competence.