Unit 10.00: Asking a vision-language model a checkable question

A prompt to a vision-language model should define what a correct answer looks like.

Checkable, enumerable, and with a way to say no

Five prompts, sorted by whether the answer can be checked.

The code lists them with a reason each.

PROMPTS = [
    ("Describe this image.", False, "no checkable answer"),
    ("Is there visible damage to the carton? Answer yes or no.", True,
     "binary, checkable"),
    ("What do you think about this?", False, "not a question about the image"),
    ("List every visible text string, verbatim. If none, say NONE.", True,
     "enumerable, and has an explicit empty answer"),
    ("How many cartons are in the image? If you cannot count them "
     "reliably, say UNSURE.", True, "has a refusal option"),
]
print(f"{'prompt':64} checkable?")
for prompt, checkable, why in PROMPTS:
    print(f"{prompt:64} {'yes' if checkable else 'NO':4} {why}")

# A prompt with no checkable answer produces output you can only judge by
# reading. The three good ones each define what a correct answer looks like,
# and two of them define what "I cannot tell" looks like.

"Describe this image" produces output you can only judge by reading. The three good prompts each define a correct answer, and two of them define what "I cannot tell" looks like.

The explicit UNSURE and NONE options are the part people leave out, and they are what turn a guess into a refusal you can count.

The mistake this prevents

The mistake is asking open questions and evaluating by reading a few outputs. It does not scale past the first fifty, and by then the drift you are looking for is too small to see by eye.

Takeaway

Write prompts whose answers can be checked mechanically, and give every one an explicit way to say the model cannot tell.

Mark complete