Applied Checkpoint: Control a Training Application Assistant
Complete the activity before revealing the model answer. Record one change you would make after comparison.
All people, records, and organisations in this activity are fictional.
Proposal
A training centre proposes an AI assistant that will:
- Read applications containing names, contact details, employment history, disability-adjustment requests, and free-text notes.
- Search public websites for more information about applicants.
- Rank applicants for a limited free course.
- Automatically reject the bottom half.
- Email decisions without staff approval.
- Read attachments and follow instructions found in them.
- Store prompts, outputs, and tool logs without a retention limit.
No approved data process, fairness test, accessibility review, security test, appeal route, or named owner exists.
Text description of the visual
The figure is a pre-answer worksheet for the applied checkpoint. It lists proposal signals from the case: personal applications, public website search, ranking limited course places, automatic rejection and email, attachment instructions, and no named owner or retention limit. It then gives seven review areas for learners to complete: affected people, data and minimisation, untrusted content and actions, fairness and accessibility, rights or policy review, named owners, and escalation route with stop condition. The worksheet asks learners to produce one risk, one control, one owner, and one stop condition for each material issue before comparing with the model answer. It does not reveal the model response decision.
Your work
- Identify affected people and possible harms.
- Classify the proposed information and remove unnecessary data.
- Identify untrusted-content and agent-action risks.
- Identify fairness and accessibility questions.
- Identify rights or policy questions requiring specialist review.
- Assign controls to named roles.
- State a clear escalation route and stop condition.
Reveal the model answer
Decision
Reject the proposed automated workflow. Do not upload applications or enable search, ranking, rejection, or email actions.
Main risks
- Applicants may lose an opportunity through an unsupported high-impact ranking.
- Personal, confidential, employment, and adjustment information would be exposed without an approved purpose or tool.
- Public searches may collect irrelevant, inaccurate, or unfair information.
- Attachments may contain malicious or conflicting instructions.
- Automatic email and rejection actions occur before meaningful review.
- Indefinite logs create unnecessary privacy and security exposure.
- No accessibility, challenge, correction, or human-support route exists.
Required redesign
- Programme owner: define lawful purpose, criteria, evidence, benefit, and human-led decision process.
- Privacy/data owner: identify permitted fields, minimise data, approve service and retention, and define correction/deletion routes.
- Security owner: block attachment instructions from controlling tools; restrict permissions; test hostile content; protect and limit logs; define monitoring and incident response.
- Accessibility lead: test the complete application, communication, correction, and support process.
- Authorised selection panel: make decisions using approved criteria and relevant evidence; provide review and challenge routes.
- Legal/rights specialist: review applicable duties, notices, terms, fairness, and any external search or data-source proposal.
Stop condition
Stop before any real application is uploaded. The process may not enter pilot until the named owners approve the data map, human-led criteria, accessibility route, security controls, retention, notices, and challenge process. Any instruction inside applicant content must be treated as untrusted and must not trigger a tool or action.
Reflection
Which single proposed capability creates the largest immediate increase in harm, and which control addresses it directly?