Unit 10.03: Grounding an answer in the image region

An answer that names a region can be checked in seconds.

The claim, and where in the image it is

A structured answer with a bounding region and a description of it.

The code prints one and validates its shape.

import json

answer = {
    "question": "Is there visible damage?",
    "answer": "yes",
    "region": {"x1": 120, "y1": 340, "x2": 260, "y2": 470},
    "region_description": "crushed corner, lower left of the carton",
    "confidence": "high",
}
print(json.dumps(answer, indent=2))

checks = [
    ("names a region", "region" in answer),
    ("region is inside the image", answer["region"]["x2"] <= 1920),
    ("describes what is in the region", bool(answer["region_description"])),
]
for check, ok in checks:
    print(f"  {'OK  ' if ok else 'FAIL'} {check}")

print("\na reviewer can crop that box and see whether the claim holds")

# An answer with a region is checkable in seconds. An answer that says
# "yes, there is damage" is checkable only by re-examining the whole image,
# which nobody does at volume.

A reviewer crops that box and sees whether the claim holds. An answer that says only 'yes, there is damage' can be checked only by re-examining the whole image, which nobody does at volume.

The region also constrains the model usefully: having to point at something makes an unfounded claim harder to produce than a free-text assertion does.

The mistake this prevents

The mistake is accepting a region without validating it against the image bounds. A model asked for coordinates will produce coordinates, including ones outside the image, and an unchecked region is not evidence.

Takeaway

Require a region with every visual claim and validate it against the image bounds. It makes the answer checkable in seconds instead of minutes.

Mark complete