Experimental Protocol for Algorithm Comparison

Unit ID: AMLA-M01-U04 Estimated active time: 30-45 minutes

Classroom explanation

Algorithm comparison is an experiment. If the experiment is messy, the result is not trustworthy.

Before comparing algorithms, write the protocol.

What the protocol must include

1. The dataset version.
2. The target.
3. The prediction time.
4. The train, validation, and test strategy.
5. The baseline.
6. The candidate algorithms.
7. The metric or metrics.
8. The hyperparameter search boundary.
9. The rule for choosing or rejecting a model.
10. The final test-set rule.

Why this matters

If you keep changing the metric, split, features, and candidate list after seeing results, you can accidentally tune your conclusion. The notebook may look scientific while the decision is actually unstable.

The test set should be saved for final checking. It should not guide every choice.

A simple protocol sentence

For this synthetic learner-support dataset, I will compare a dummy baseline, logistic regression, a pruned tree, and a random forest using cross-validation on the training data. I will choose based on recall and precision trade-off, then check the selected candidate once on the held-out test set.

Practice

Write a five-line protocol for a small regression or classification task. Include baseline, candidates, metric, split, and stop rule.

Takeaway

A comparison without a protocol is just exploration. Exploration is useful, but it should not be sold as a final model choice.