The shortlist that missed the evidence
Audit an AI ranking against the job’s actual requirements.
The question
Cairn Operations advises a fictional UK client hiring a customer-operations analyst. You review a proposed AI shortlist before the hiring manager decides.
Analyze service data, explain findings to non-specialists and document repeatable processes. A degree is not required. Only job-related evidence from the application may be used.
| Relevant evidence | AI rank | |
|---|---|---|
| A | Three years of service analysis; quantified queue improvement; no degree. | 3 |
| B | Business degree; strong written sample; no analysis example. | 1 |
| C | Career returner; detailed process audit and dashboard sample. | 2 |
Invented applications. The assistant supplied no reasons for the ranks.
Interview the top two, reject the third automatically, and keep the rank as our audit record.
“Would you approve this process? Explain what AI may do and what the hiring manager must own.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Reject the unsupported ranking
What would you do here? Open the move you’d make.
Use the top two because the rank is consistent.
Tempting, but it misses something. A consistent system can still apply the wrong criteria.
Require a source-linked evidence map against the published criteria.
The strong move. This makes missing evidence and unsupported criteria visible.
See how a strong candidate says it
“I would not approve automatic rejection. Rank alone does not show how the published criteria were applied. A has direct analysis evidence that deserves review.”
Why it works: The task is to map evidence to criteria, not replace one unexplained rank with another.
Step 2: Limit the extraction task
“Ask AI to extract source passages for each criterion and flag missing evidence. Do not infer ability from degree status or a career break. Check extracts against the applications.”
Why it works: Checked evidence extraction has a clearer boundary than a final rank.
Step 3: Make human review real
“The hiring manager applies the same job-related rubric to all applicants and records reasons. Review must inspect the evidence, not rubber-stamp the AI order.”
Why it works: A human click alone is not an accountable decision.
Step 4: Test omissions before scaling
“Test varied application formats and backgrounds for missed relevant evidence. Follow the client’s approved recruitment and data rules. Track omissions, reviewer corrections and time.”
Why it works: Useful quality checks show what the extraction misses, not just how fast it runs.
The answer, in short
- Decision
- Use checked evidence extraction; keep shortlist decisions with the hiring manager.
- Because
- The rank lacks reasons and may omit directly relevant evidence.
- What would change it
- Reassess after criterion-level evidence and consistent review work across varied applications.
Common traps
- Treating a human click as meaningful oversight.
- Inventing protected traits or secret employer criteria.
Check your answer
- Did I use published requirements?
- Did I check the original application evidence?
- Did I define the manager’s review?
- Did I test varied application formats?
Last reviewed Nov 10, 2026