Unit 03.04: Checking a pipeline on the images it will actually see

A pipeline tuned on the images you had is a pipeline that works on the images you had.

Test across the conditions that will actually occur

The same fixed pipeline run against three lighting levels.

The code shows where it stops working.

import cv2
import numpy as np

rng = np.random.default_rng(2)


def pipeline(image):
    blurred = cv2.GaussianBlur(image, (5, 5), 0)
    _, binary = cv2.threshold(blurred, 128, 255, cv2.THRESH_BINARY)
    return binary


CONDITIONS = {
    "bright, even":  120,
    "dim":            40,
    "very dim":       15,
}
for name, level in CONDITIONS.items():
    image = np.full((32, 32), level // 3, dtype="uint8")
    image[8:24, 8:24] = level
    found = pipeline(image)[8:24, 8:24].mean() > 128
    print(f"{name:14} object level {level:>3}  detected: {found}")

# The pipeline was tuned on bright images and silently fails on dim ones. Test
# on the range of conditions the images will actually arrive in, not on the
# ones you had to hand while developing.

The pipeline was implicitly tuned for bright images - the hard-coded threshold of 128 is the tell - and it fails silently on dim ones. No error, just an object that is not detected.

Collecting the range of conditions before building is what makes this visible. It is also cheap: a handful of images from each condition is enough to expose a hard-coded constant.

The mistake this prevents

The mistake is developing on a convenient sample and validating on the same one. Every hard-coded constant in the pipeline is a hidden assumption about the images, and only different images reveal them.

Takeaway

Test the pipeline across the real range of conditions. Hard-coded constants encode assumptions about lighting that only differently-lit images expose.

Mark complete