Northstar Dispatch data dictionary and reproducibility

All organizations, shipment IDs, people and outcomes in this package are invented. These files contain no real customers, addresses or workforce data. Weeks 1–8 are numbered teaching periods, not actual calendar dates. A spreadsheet or calculator is sufficient for the core reading; the optional Python route assumes the basic lists, loops, CSV files and functions introduced in the Aabha Python course. SQL is not a prerequisite.

Files and observational units

The 40 trial tasks are separate from the 240 shipments. Never join them by row number, treat tasks as shipments, or interpret the task effect as a delivery-time effect. Each person appears once; the trial is not paired or repeated-measures data.

Shipment columns

Rates use all valid shipments in the stated group as the denominator. Rating summaries use responding shipments only and must report that denominator. A respondent mean is a summary of numeric category codes, not a population satisfaction estimate; also show the rating distribution.

Trial columns

The comparison is always checklist minus control. Negative differences mean shorter checklist times. The absolute-unit difference is expressed in minutes; the relative difference divides by the control mean. A 95% interval for a minute difference is not a 95% interval for a percentage reduction.

Deterministic cleaning policy

The complete policy is implemented in tools/generate_data.py, function clean_shipment_rows. It does not look at a hidden answer file or use the generator to repair an observation.

1. Read the CSV as text. Trim leading/trailing whitespace; normalize hub to title case and service to lowercase. Preserve shipment IDs.
2. Validate ID format, week 1–8, East/West, standard/priority, finite positive distance and durations, and satisfaction either blank or an integer 1–5. Quarantine any row with an invalid required field or a nonblank invalid rating. Do not invent its value.
3. Recompute late from the two validated durations. A stale exported late flag is repairable because its complete definition and inputs are available; log changes.
4. Collapse canonical exact duplicate records with the same ID. A conflicting duplicate ID quarantines every validated record for that ID, including the first record and any earlier or later identical copies, rather than arbitrarily keeping the first. Rows that failed field validation keep their invalid-row disposition. No conflicting duplicate occurs in the shipped raw fixture, but the cleaner and tests cover it.
5. Sort retained records by shipment_id and write the clean CSV. Retain missing ratings and valid extreme values. Record the source line, ID and reason for every quarantined row. Every raw source row also receives exactly one final disposition: accepted, invalid_quarantined, conflict_quarantined or exact_duplicate_removed. Duplicate classification is finalized only after all validated copies of an ID are examined.

The raw fixture has 240 valid underlying records, three exact duplicate copies and five invalid appended records. Six formatting cells across three rows normalize; two stale late flags recompute. The invalid extras demonstrate negative distance, nonnumeric actual time, unknown hub, missing promised time and rating outside 1–5. Therefore 248 raw rows produce 240 clean observations. The full machine-readable audit is outputs/cleaning_audit.json. Its counts reconcile as input_rows = accepted_rows + total_rows_quarantined + exact_duplicate_rows_removed; total_rows_quarantined is the sum of invalid_rows_quarantined and conflicting_rows_quarantined. Formatting and late-flag repair counters describe transformations and are not additional row dispositions. This rule is for this documented export; it is not a universal permission to discard unusual operational data.

How the fictional observations were generated

The generator uses Python random.Random with shipment seed 20261014 and independent trial seed 20261015. Seeds were fixed before inspecting results; no outcome rerolls were used to obtain a preferred p-value. Committed CSVs are the canonical fixtures. Python 3.12 is the tested reproduction environment; record versions if reproducing elsewhere because higher-level random algorithms can change.

Observational shipments

Each week contains 15 East and 15 West jobs. East receives 5 priority and 10 standard jobs per week; West receives 10 priority and 5 standard. Service order is shuffled within hub/week. East distances are uniform from 20 to 55 km; West distances are uniform from 5 to 35 km, rounded to one decimal. The promise is 25 + 1.7 × distance − 8 if priority. Actual time is 15 + 1.7 × distance − 5 if priority, plus a week condition and independent Normal(0, 7) noise. With probability 0.08, an additional uniform 15–30 minute delay is added. Durations are floored at 5 minutes, rounded to one decimal; late is derived afterward.

Week conditions are −2, 1.5, 0, 4, −1.5, 2.5, 0.5 and 3 minutes. Shared conditions create a reason to question independent-row inference. The generator has no direct hub term; the raw mean difference between hubs is associated with distance and service mix. Merely adjusting one variable would not establish causal identification in an actual operation. This snapshot is not a probability sample of a real delivery population.

A provisional rating is rounded from 4.4 − 1.4 if late + Normal(0, 0.7), then clipped to 1–5. A late shipment's rating is missing with probability 0.45; an on-time shipment's with probability 0.10. This is deliberately selective response, associated with an observed variable. It is not a demonstration that missingness necessarily depends on the unobserved rating after conditioning on lateness. Do not automatically label it “missing not at random.” Do not fill blanks with zero or the overall mean.

Separate randomized teaching fixture

Forty independent simulated people are generated from one fixed teaching process. A shuffled balanced list assigns 20 control and 20 checklist labels before outcomes are generated. For each person, the underlying control duration is 24 + independent Normal(0, 4.2) noise. Their checklist potential outcome is exactly three minutes less before rounding. Only the assigned outcome is observed and is rounded to 0.01 minute. There is no second task per person, attrition, crossover or outcome-based reassignment.

The data model supports the intended normal independent-errors teaching example. The observed group difference need not equal the built-in −3 minutes because the randomly assigned groups have different realized noise. Random assignment illustrates causal identification within this simulated experiment; these fictional people are not a representative sample of any real workforce. The exercise cannot establish a real checklist benefit. Failure to reject a null in a different realization would not establish no effect or equivalence.

Numerical conventions and reference outputs

• Standard deviation uses the sample denominator n−1 unless explicitly marked population SD. A singleton's sample SD is undefined, stored as null.
• Quartiles use NumPy's method="linear": sort values, locate index (n−1) × p with zero-based indexing and interpolate. Other conventions can give different answers on short lists. For [8,9,10,11,12,30], Q1=9.25, Q3=11.75, IQR=2.5.
• One-sample control-mean interval: mean ± t(.975, n−1) × sample SD / √n. This is a mean interval, not an interval for individual task duration.
• Two-group comparison: Welch's two-sided test, null mean difference 0, and a compatible 95% Welch–Satterthwaite difference interval. No equal-variance assumption is required. The script includes both explicit formulas and an independent SciPy consistency check.
• Pearson r, ordinary least-squares slope/intercept and R² summarize distance versus time in the observational shipments. No slope p-value or causal inference is supplied. Inspect linearity, influential observations, residual spread and dependence before inference; never assume a high R² establishes causation.
• Fractions are stored on a 0–1 scale unless the key explicitly says percent. Full precision is preserved; round only for presentation.
• outputs/reference_results.json contains instructor/reference answers, including group counts, rates and missingness. It is not a learner's submission or a test of their exercise implementation.

Optional probability and sampling illustrations

The probability toy is separately stipulated, not estimated from the CSV: among 1,000 jobs, 100 are late and 900 on time. An alert flags 80 late and 90 on-time jobs. Thus sensitivity is 80/100 and the false-positive rate is 90/900; 80 of the 170 alerts are late. The two conditional probabilities P(flag | late) and P(late | flag) have different denominators.

Optional tools/sampling_demo.py creates 5,000 new independent normal samples, each n=20, from a known mean 24 and SD 4 with NumPy seed 20261016. It reports the spread of sample means and how often Student-t intervals contain the known fixed mean. A finite run need not yield exactly 95% coverage. This simulation is independent of both CSV datasets and does not certify their assumptions. Results go to outputs/sampling_demo.json.

Reproduce locally

From this course folder, using the Python interpreter in your virtual environment:

python tools/generate_data.py
python tools/analyze.py
python tools/sampling_demo.py
python -m unittest discover -s tests -v

The generator overwrites only the named course fixtures and cleaning audit. The analysis overwrites reference_results.json; it does not change raw observations. No tool accesses the network or a real account. requirements.txt pins the tested NumPy, SciPy and Matplotlib versions; Matplotlib supports the parent course's charts and is not imported by the two core data scripts. See the main setup guide for virtual-environment commands.

Primary documentation checked 2026-10-14

• NIST: confidence limits for a mean supplies the Student-t interval and its interpretation.
• SciPy 1.17: independent t tests documents equal_var=False, test direction and confidence_interval.
• SciPy 1.17: linear regression documents slope, intercept and squared correlation for the simple linear fit.
• SciPy: Pearson correlation documents the linear-association coefficient and inference assumptions; this package reports the coefficient descriptively.
• NumPy 2.3: quantiles documents the explicitly chosen linear interpolation convention.
• Python 3.12: random documents seeded pseudo-random generation and reproducibility limitations.

Original data and code are supplied for teaching; linked documentation is not copied into this package.