Unit 08.03: Measuring whether it beat the manual process

The comparison that matters includes the review time the automation creates and the errors it adds.

Hours saved, and errors gained

Manual time against automated time plus review time, with both error rates stated.

The code compares 900 items a month both ways.

MANUAL = {"minutes_per_item": 6.0, "error_rate": 0.02, "items_per_month": 900}
AUTOMATED = {"minutes_per_item": 0.4, "error_rate": 0.05,
             "review_minutes_per_item": 1.5, "usd_per_item": 0.018,
             "items_per_month": 900}

manual_hours = MANUAL["minutes_per_item"] * MANUAL["items_per_month"] / 60
auto_hours = ((AUTOMATED["minutes_per_item"] + AUTOMATED["review_minutes_per_item"])
              * AUTOMATED["items_per_month"] / 60)

print(f"manual    : {manual_hours:>6.1f} human hours/month, "
      f"{MANUAL['error_rate']:.0%} error rate")
print(f"automated : {auto_hours:>6.1f} human hours/month, "
      f"{AUTOMATED['error_rate']:.0%} error rate, "
      f"${AUTOMATED['usd_per_item'] * AUTOMATED['items_per_month']:.2f} in tokens")
print(f"\nsaved: {manual_hours - auto_hours:.1f} hours/month")
print(f"extra errors: {(AUTOMATED['error_rate'] - MANUAL['error_rate']) * 900:.0f} per month")

# Review time is the line people leave out, and it usually dominates. So does
# the error rate going UP: 27 extra errors a month may be worth 66 saved hours,
# or may not, and that is a decision someone should make explicitly.

Review time is the line people leave out, and here it is nearly four times the processing time - it dominates the automated column entirely. An estimate that counts only the model's speed would overstate the saving by a wide margin.

The error rate going *up*, from 2% to 5%, is the honest part. That is 27 extra errors a month against 66 hours saved. It may well be a good trade; it is not a trade anyone should make accidentally, and it is certainly not one to leave out of the summary.

The mistake this prevents

One thing this comparison cannot tell you: whether the manual process was any good. Automating a broken process does not fix it - it produces the same broken results faster and at larger scale, and the hours-saved figure will look excellent while doing so. Before measuring the automation against the manual baseline, check that the baseline is one you would defend. If the manual process has a step everyone works around, automate the corrected process, not the observed one.

The mistake is reporting hours saved without reporting error rate. It is the number that makes the case look best and the one that will be discovered later by whoever handles the consequences of those 27 errors.

Takeaway

Compare manual against automated *including review time and both error rates*. An automation that saves time and adds errors is a trade to make explicitly.

Mark complete