Unit 01.03: What a script guarantees that an AI app does not

Six guarantees a script gives you, and what an AI app gives you instead.

Including one that changes without a commit

Determinism, offline operation, fixed cost, known output shape, loud failure, and stability over time.

The code compares them.

GUARANTEES = [
    ("same input, same output",      "script: yes", "AI app: no"),
    ("runs offline",                 "script: yes", "AI app: no"),
    ("fixed cost",                   "script: yes", "AI app: no"),
    ("output shape is known",        "script: yes", "AI app: only if validated"),
    ("failure is loud",              "script: yes", "AI app: often silent"),
    ("behaviour is stable over time", "script: yes",
     "AI app: no -- the provider updates the model"),
]
for prop, script, app in GUARANTEES:
    print(f"{prop:30} {script:14} {app}")

print("""
The last is the one people forget. A script you deploy behaves the same next
year; an app calling a hosted model can change behaviour without any commit,
because the model behind the endpoint changed.

That is why the eval set in Module 11 has to run on a schedule and not only
when you push.
""")

The last row is the one people forget. A script you deploy behaves the same next year. An app calling a hosted model can change behaviour with no commit and no deploy on your side, because the model behind the endpoint changed.

That is why the eval set has to run on a schedule rather than only when you push - the change you need to detect is not one of yours.

The mistake this prevents

The mistake is treating the provider's model as a fixed dependency because the endpoint URL did not change. Pin the version where the provider allows it, and re-run the evals when the version moves.

Takeaway

An AI app gives up determinism, offline operation, fixed cost and - most easily forgotten - stability over time. Schedule the evals rather than triggering them only on your own commits.

Mark complete