Unit 08.00: Reading a file you did not write

A file a user uploaded is untrusted input, and three of the four things that can be wrong with it produce no exception at the point you care.

Encoding, emptiness, and binary content

Four files through a decode-and-check sequence.

The code rejects three of them with distinct reasons.

import io

FILES = {
    "notes.txt": b"Refunds are allowed within 7 days.\n",
    "empty.txt": b"",
    "binary.bin": b"\x00\x01\x02\xff",
    "latin1.txt": "Refund fee: 5\u00a3\n".encode("latin-1"),
}
for name, data in FILES.items():
    try:
        text = data.decode("utf-8")
        if not text.strip():
            print(f"{name:12} REJECT: empty after decoding")
        elif "\x00" in text:
            print(f"{name:12} REJECT: contains null bytes, probably binary")
        else:
            print(f"{name:12} ok, {len(text)} chars")
    except UnicodeDecodeError as exc:
        print(f"{name:12} REJECT: not UTF-8 ({exc.reason})")

# Four files, three rejections, each with a different reason. A user upload is
# untrusted input: check the encoding, check it is not empty, and check it is
# not binary before any of it reaches a prompt.

The latin-1 file raises a UnicodeDecodeError, which is the easy case. The empty file decodes perfectly and contains nothing - it would produce a prompt with an empty document section and a confident answer about nothing.

The binary file may decode into something that looks like text and is not, and it will be sent to the provider and billed for.

The mistake this prevents

The mistake is calling .decode() with errors="ignore" to make the problem go away. It succeeds on every input, produces silently mangled text, and turns a loud failure into a quiet one.

Takeaway

Check encoding, emptiness and binary content before a file reaches a prompt. errors="ignore" converts a clear failure into mangled text you will pay to process.

Mark complete