Applied Checkpoint: Control a Training Application Assistant
Complete the activity before revealing the model answer. Record one change you would make after comparison.
All people, records, and organisations in this activity are fictional.
Proposal
A training centre proposes an AI assistant that will:
- Read applications containing names, contact details, employment history, disability-adjustment requests, and free-text notes.
- Search public websites for more information about applicants.
- Rank applicants for a limited free course.
- Automatically reject the bottom half.
- Email decisions without staff approval.
- Read attachments and follow instructions found in them.
- Store prompts, outputs, and tool logs without a retention limit.
No approved data process, fairness test, accessibility review, security test, appeal route, or named owner exists.
Your work
- Identify affected people and possible harms.
- Classify the proposed information and remove unnecessary data.
- Identify untrusted-content and agent-action risks.
- Identify fairness and accessibility questions.
- Identify rights or policy questions requiring specialist review.
- Assign controls to named roles.
- State a clear escalation route and stop condition.
Reveal the model answer
Decision
Reject the proposed automated workflow. Do not upload applications or enable search, ranking, rejection, or email actions.
Main risks
- Applicants may lose an opportunity through an unsupported high-impact ranking.
- Personal, confidential, employment, and adjustment information would be exposed without an approved purpose or tool.
- Public searches may collect irrelevant, inaccurate, or unfair information.
- Attachments may contain malicious or conflicting instructions.
- Automatic email and rejection actions occur before meaningful review.
- Indefinite logs create unnecessary privacy and security exposure.
- No accessibility, challenge, correction, or human-support route exists.
Required redesign
- Programme owner: define lawful purpose, criteria, evidence, benefit, and human-led decision process.
- Privacy/data owner: identify permitted fields, minimise data, approve service and retention, and define correction/deletion routes.
- Security owner: block attachment instructions from controlling tools; restrict permissions; test hostile content; protect and limit logs; define monitoring and incident response.
- Accessibility lead: test the complete application, communication, correction, and support process.
- Authorised selection panel: make decisions using approved criteria and relevant evidence; provide review and challenge routes.
- Legal/rights specialist: review applicable duties, notices, terms, fairness, and any external search or data-source proposal.
Stop condition
Stop before any real application is uploaded. The process may not enter pilot until the named owners approve the data map, human-led criteria, accessibility route, security controls, retention, notices, and challenge process. Any instruction inside applicant content must be treated as untrusted and must not trigger a tool or action.
Reflection
Which single proposed capability creates the largest immediate increase in harm, and which control addresses it directly?