Users, Affected People, and Failure Modes

Unit ID: ML-M01-U03 Estimated active time: 25-35 minutes

A model is used by people

A model is not just code. It becomes part of a human process.

So we need to ask:

• Who uses the model result?
• Who is affected by the result?
• What can go wrong?
• Which mistakes are unacceptable?

These questions belong at the beginning, not only at the end.

User and affected person are not always the same

In the learner-support example, the direct user may be:

A course mentor.

The affected person is:

The learner who receives, or does not receive, support.

This difference matters.

The mentor may want an efficient support list. The learner may want privacy, fairness, respect, and a chance to continue without being labelled unfairly.

A useful model for the user can still create a bad experience for the affected person.

Failure modes

A failure mode is a way the system can go wrong.

For the learner-support example, possible failure modes include:

• A learner who needs support is missed.
• A learner who does not need support receives a worrying message.
• The model uses a proxy that unfairly reflects access, language, disability, work schedule, or internet quality.
• Mentors trust the score too much and stop using judgement.
• The support action makes learners feel watched instead of helped.
• The system keeps using old patterns after the course changes.

Write failure modes in plain language. Do not hide them behind technical words.

False positives and false negatives

For many classification problems, two basic mistakes are important.

A false positive means the model says yes, but the true answer is no.

A false negative means the model says no, but the true answer is yes.

In our example:

• False positive: the model flags a learner as needing support, but the learner would have completed without help.
• False negative: the model does not flag a learner who actually needed support.

Which one is worse?

The answer depends on the action.

If the action is a gentle optional message, false positives may be acceptable in small numbers. If the action is blocking access, assigning a low grade, or denying an opportunity, false positives become much more serious.

Unacceptable failure

Some mistakes are not just normal error. They should stop or redesign the project.

Examples:

• The model result is used to punish learners.
• Learners cannot ask for help unless the model flags them.
• The model works poorly for a group that is already less supported.
• The model requires sensitive personal data when a safer option exists.
• No person is responsible for reviewing serious cases.

An unacceptable failure mode should be written before the model is trained.

Guided practice

Read the problem frame:

At the end of day 3, predict whether a learner is likely to complete Module 1 by day 10, so a mentor can offer optional support on day 4.

Fill the table.

Now write two more unacceptable failures.

Takeaway

Problem framing includes people and failure modes. A model that is technically impressive but harmful in use is not a good ML project. In the next unit, we will compare the model idea with simpler alternatives.