Skip to course content
Free course

Python Foundations / Module 7 / Provenance and Non-Destructive Output

Module 7 lesson

Provenance and Non-Destructive Output

Unit ID: M07-U06 Estimated active time: 22-30 minutes

Preserve the raw source

The checkpoint never opens the source in write mode. Outputs use distinct names in a separate folder.

Before conversion, read the source bytes and calculate a checksum:

import hashlib

source_bytes = source_path.read_bytes()
source_sha256 = hashlib.sha256(source_bytes).hexdigest()

After conversion, calculate it again and assert equality. A checksum does not prove the source is correct, but it detects byte-level change.

Record transformation facts

A transformation log should include:

  • source file name;
  • source checksum;
  • required headers;
  • validation rules;
  • source row count;
  • accepted and rejected counts;
  • output file names; and
  • course code version or release identifier.

Avoid dynamic timestamps when a deterministic learning output is required, or document why time is included.

Keep rejection evidence

Rejected rows are not thrown away. Store source row, safe code, and reasons. Do not copy sensitive raw values into logs.

Separate raw, working, and output folders

A common layout is:

data/source_courses.csv
output/accepted_courses.json
output/rejected_courses.json
output/transformation_log.json

Version-control or backup rules depend on the real project. The course uses fictional data and reproducible output.

Practice

Write a transformation-log dictionary for the supplied file. Explain what the checksum can and cannot prove. Confirm that every output path differs from the source path.

Takeaway

Provenance connects outputs to source, rules, and code. Preserve raw bytes, separate outputs, and keep rejection evidence. You are now ready for the complete CSV-to-JSON conversion checkpoint.