Unit 01.00: The same input, a different answer

Every habit you have from writing scripts assumes the same input produces the same output. That assumption is the first thing to give up.

Three correct answers, all different

A deterministic function called three times, and a model call made three times with the same prompt.

The code shows both.

def deterministic(name):
    return f"Hello, {name}!"


class Model:
    """Stand-in: same prompt, different completion each call."""
    def __init__(self, replies):
        self.replies = list(replies)

    def __call__(self, prompt):
        return self.replies.pop(0)


model = Model(["Hi there, Ada!", "Hello Ada, good to meet you.",
               "Hey Ada!"])
print("deterministic function, three calls:")
for _ in range(3):
    print("  ", deterministic("Ada"))

print("\nmodel call, three times, same prompt:")
for _ in range(3):
    print("  ", model("greet Ada"))

# Three different strings, all correct. Every test you write, every cache you
# add and every comparison you make has to cope with that -- which is the whole
# difference between this and a script.

The three model replies are all correct and none is the same string. Every test you write, every cache you add, every comparison between versions has to cope with that.

It is not noise to be eliminated. Asking for the same wording every time is asking for something the technology does not offer, and designing around variation is cheaper than fighting it.

The mistake this prevents

The mistake is writing assertions on exact output. They pass locally, pass in CI for a month, and fail on a provider-side model update - at which point someone disables the test rather than rewriting twenty assertions.

Takeaway

The same prompt produces different correct answers. Design tests, caches and comparisons around that rather than trying to suppress it.

Mark complete