Unit 11.04: A suite fast enough that people run it

A suite that takes minutes is a suite people stop running.

Offline is most of the tests and a fraction of the time

Five groups costed by count and duration.

The code totals them.

SUITE = [
    ("prompt rendering",   90, 0.003, "no model"),
    ("input validation",   60, 0.002, "no model"),
    ("schema validation",  70, 0.002, "no model"),
    ("endpoint tests",     40, 0.012, "no model"),
    ("live model calls",    6, 2.800, "paid, slow"),
]
print(f"{'group':22} {'tests':>6} {'sec each':>9} {'total':>8}  note")
total = 0.0
for name, count, each, note in SUITE:
    seconds = count * each
    total += seconds
    print(f"{name:22} {count:>6} {each:>9.3f} {seconds:>7.2f}s  {note}")

offline_tests = sum(c for _, c, _, n in SUITE if n == "no model")
offline_time = sum(c * e for _, c, e, n in SUITE if n == "no model")
all_tests = sum(c for _, c, _, _ in SUITE)
print(f"\noffline: {offline_tests}/{all_tests} tests in {offline_time:.2f}s")
print(f"the {all_tests - offline_tests} live tests are "
      f"{(total - offline_time) / total:.0%} of the wall clock")

# Keep the offline suite under a few seconds and it runs on every save. Put the
# live calls behind a separate command that runs before a release.

The overwhelming majority of tests need no model, and the handful that do take most of the wall clock. Splitting them means the fast suite runs on every save and the slow one runs before a release.

That split is what keeps the fast suite fast enough that nobody skips it, which matters more than any individual test in it.

The mistake this prevents

The mistake is putting the live calls in the same command because they belong to the same system. The suite then takes minutes, developers run it less, and the cheap tests that catch most regressions stop running on every change.

Takeaway

Split the offline suite from the paid one and keep it fast. The tests that catch most regressions are the ones that need no model.

Mark complete