Unit 03.04: The cost difference over a thousand runs

Cost and latency in a graph are almost entirely determined by how many nodes call a model. Everything else rounds to zero.

Two nodes out of five

The code prices a five-node workflow over a thousand runs, splitting each node into code or model and reporting cost and latency separately.

Read the proportions rather than the absolute figures - the rates will be wrong by the time you read this, and the shape will not.

RUNS = 1000
STEPS = [
    ("parse input",     "code",  0.0,    0.2),
    ("classify",        "model", 0.0015, 800),
    ("look up account", "code",  0.0,    15),
    ("summarise",       "model", 0.0030, 1200),
    ("format output",   "code",  0.0,    0.3),
]

print(f"{'step':18} {'kind':6} {'cost/1k':>9} {'latency':>9}")
cost = latency = 0.0
for step, kind, per_call, ms in STEPS:
    cost += per_call * RUNS
    latency += ms
    print(f"{step:18} {kind:6} {'$' + format(per_call * RUNS, '.2f'):>9} {ms:>7.1f}ms")

print(f"\ntotal over {RUNS} runs: ${cost:.2f}, {latency / 1000:.1f}s per run")

model_ms = sum(ms for _, k, _, ms in STEPS if k == "model")
print(f"model steps are {model_ms / latency:.0%} of latency and 100% of cost")

# Two model steps out of five account for essentially all of both. That is the
# argument for moving every decision you can into code -- and for asking, per
# model step, whether one call could do the work of two.

The two model steps account for effectively all of the cost and the overwhelming majority of the latency. The three code steps together contribute under a millisecond.

That proportion is the argument for the last four units, and it suggests a second question worth asking per model step: could one call do the work of two? Merging classify and summarise into a single structured call is often possible, and it halves the dominant term.

The mistake this prevents

The mistake is optimising the code steps. They are already free, and time spent making a parse faster is time not spent on the two nodes that actually cost something. Profile before optimising, and in a graph the profile is nearly always the same shape.

Takeaway

Model steps dominate cost and latency. Reduce their number - by moving decisions into code, and by asking whether two calls could be one - before optimising anything else.

Mark complete