Unit 04.04: Keeping the suite fast enough to run

A test suite that takes ten minutes is a test suite people stop running.

Split offline from paid

The offline tests are the overwhelming majority by count and a small fraction of the time. The model tests are the reverse.

The code costs out a five-group suite.

SUITE = [
    ("prompt rendering",     120, 0.004, "no model"),
    ("schema validation",     80, 0.002, "no model"),
    ("tool refusal branches", 45, 0.003, "no model"),
    ("retrieval filters",     30, 0.020, "no model"),
    ("end-to-end with model",  8, 3.100, "paid, slow"),
]
print(f"{'group':24} {'tests':>6} {'sec each':>9} {'total':>8}  note")
total = 0.0
for name, count, each, note in SUITE:
    seconds = count * each
    total += seconds
    print(f"{name:24} {count:>6} {each:>9.3f} {seconds:>7.1f}s  {note}")

offline = sum(c * e for n, c, e, note in SUITE if note == "no model")
offline_tests = sum(c for _, c, _, note in SUITE if note == "no model")
all_tests = sum(c for _, c, _, _ in SUITE)
print(f"\ntotal {total:.1f}s across {all_tests} tests")
print(f"offline: {offline_tests}/{all_tests} tests ({offline_tests / all_tests:.0%}) "
      f"in {offline:.1f}s ({offline / total:.0%} of the time)")
print(f"the {all_tests - offline_tests} model tests are "
      f"{(total - offline) / total:.0%} of the wall clock")

# Keep the offline suite under a minute and it runs on every commit. Put the
# eight slow model tests behind a separate command that runs before a release,
# and the fast suite stays fast enough that nobody skips it.

Two hundred and seventy-five of the two hundred and eighty-three tests need no model at all, and the eight that do take most of the wall clock. Splitting them means the fast suite runs on every commit and the slow one runs before a release.

That split is what keeps the fast suite fast enough that nobody skips it, which matters more than any individual test in it.

The mistake this prevents

The mistake is putting the model tests in the same command as everything else because they belong to the same system. The suite then takes minutes, developers run it less, and the cheap tests that catch most regressions stop running on every change.

Takeaway

Split the offline suite from the paid one. Keep the offline suite under a minute so it runs on every commit; run the model tests before a release.

Mark complete