Baselines, Rules, Statistics, and the No-Model Option
Unit ID: ML-M01-U04 Estimated active time: 25-35 minutes
A model must beat a sensible alternative
Machine learning is not automatically the best solution.
Before building a model, compare it with simpler options.
The simplest useful comparison is called a baseline.
A baseline is a simple method that sets a minimum standard. If the model cannot improve on it in a meaningful way, the model is not worth much.
Four alternatives to compare
For a first framing review, compare:
- No model.
- Descriptive analysis.
- Rule-based action.
- Machine-learning prediction.
Let us apply this to learner support.
No model
The no-model option asks:
What happens if we do not build this?
This is a real option.
Maybe mentors already have enough capacity to respond to everyone who asks. Maybe the course is small. Maybe a model would add worry without improving support.
No model can be the right decision.
Descriptive analysis
Descriptive analysis asks:
Can a report or chart answer the real question?
For example:
- Which module has the highest drop-off?
- Which activity has the most repeated attempts?
- Which support message gets the best response?
If the main need is to improve course design, a descriptive report may be better than predicting individual learners.
Rule-based action
A rule-based action asks:
Can we use a clear rule instead of a model?
Example:
Send an optional support message to any learner who has not opened Module 1 for three days.
This rule is simple, explainable, and easy to change.
It may be enough.
Machine-learning prediction
ML prediction asks:
Can examples from the past help us predict which current cases need action?
It may be useful when:
- the pattern is too complex for a simple rule;
- past labels are available;
- the model can be evaluated honestly;
- the action is useful and safe;
- the improvement over simpler options is meaningful.
ML is not bad. It just needs to earn its place.
Baseline table
Use this table before modelling.
| Option | What it would do | Strength | Weakness |
|---|---|---|---|
| No model | Keep current support process | No new risk or maintenance | May miss learners who do not ask |
| Descriptive report | Show where learners drop off | Good for course improvement | Does not prioritise current learners |
| Rule | Message learners inactive for three days | Simple and explainable | May miss other support signals |
| ML model | Predict learners likely to need help | Can combine several signals | Needs labels, testing, monitoring, and safeguards |
Guided decision
Ask three questions.
- Would a simple rule solve most of the problem?
- Would a descriptive report lead to a better action?
- What must the model improve enough to justify its added complexity?
If you cannot answer the third question, the project is not ready.
Takeaway
A model should be compared with no model, a report, and a rule. This keeps the project honest. In the next unit, we will check whether the project is feasible enough to continue.