Unit 02.04: A loader that fails loudly on a bad file

A loader that skips bad files silently gives you a training set smaller than you think.

Reject with a reason, and count the reasons

Decode, size and channel checks, each returning a specific explanation.

The code runs three inputs through it.

import io
import numpy as np
from PIL import Image, UnidentifiedImageError


def load(data, expect_channels=3, min_size=32):
    """Fail loudly, with the reason, before anything downstream sees it."""
    try:
        image = Image.open(io.BytesIO(data))
        image.load()
    except (UnidentifiedImageError, OSError) as exc:
        return None, f"undecodable: {type(exc).__name__}"
    if min(image.size) < min_size:
        return None, f"too small: {image.size}"
    array = np.asarray(image.convert("RGB"))
    if array.shape[2] != expect_channels:
        return None, f"unexpected channels: {array.shape[2]}"
    return array, None


good = io.BytesIO()
Image.fromarray(np.zeros((64, 64, 3), dtype="uint8")).save(good, format="PNG")
tiny = io.BytesIO()
Image.fromarray(np.zeros((8, 8, 3), dtype="uint8")).save(tiny, format="PNG")

for label, data in [("valid", good.getvalue()), ("tiny", tiny.getvalue()),
                    ("not an image", b"hello world")]:
    array, error = load(data)
    print(f"{label:14} {'ok ' + str(array.shape) if error is None else 'REJECT: ' + error}")

# Silently skipping bad files means a training set that is quietly smaller than
# you think. Return the reason and count the rejections by kind.

Each rejection names its cause. That is what lets you count rejections by kind - and a sudden rise in one kind is a signal about an upstream change rather than a mysterious drop in dataset size.

The size check matters more than it looks. An 8×8 image decodes perfectly and contains nothing, and it will sit in the training set contributing noise.

The mistake this prevents

The mistake is a bare try/except: continue. It works, it is one line, and it means nobody ever finds out that 12% of one source's images fail to decode.

Takeaway

Return the reason for every rejection and count them by kind. Silent skipping hides both dataset shrinkage and upstream changes.

Mark complete