The Craft, Not the Calling: Why Assessor Judgement Responds to Practice
Photo: assessors in standardisation meeting discussing evidence samples around table, via www.twobrutal.co.uk
There is a sentence that surfaces in almost every assessment team at some point, usually in recruitment or in the aftermath of a difficult standardisation meeting: you can tell within five minutes whether someone has the eye for it. The sentiment is warm and the implication is costly.
If judgement is a gift, then training can only polish what is already present. Selection replaces development, instinct replaces evidence, and the assessor who struggles with borderline cases is quietly regarded as lacking something rather than as under-practised. For a sector that trades on the credibility of its decisions, that is an expensive position to occupy.
What the natural account conceals
The natural account is attractive because it explains real variation without requiring anyone to do anything. Some assessors are steadier than others. Some notice discrepancies faster. Some articulate a rationale without hesitation. But the explanation is doing less work than it appears to.
Much of what reads as instinct is exposure: years of seeing the same category of evidence, of hearing a mentor reason aloud, of being corrected early and often. The assessor who spots an authenticity problem in a portfolio usually has a store of prior examples to draw on. That store was built, not endowed.
The account also conceals a design failure. Where no structured practice exists, the only route to competence is apprenticeship by osmosis, which is slow, uneven and heavily dependent on which colleague happens to be available.
Judgement is not one skill
Treating judgement as a single quality makes it hard to train. It is better understood as a cluster of separable capabilities, each of which can be rehearsed:
- noticing what is present and, equally, what is missing from a body of evidence;
- weighing sufficiency against the standard rather than against expectation;
- articulating a rationale that another professional could follow and challenge;
- holding a decision under pressure from an employer, a candidate or a deadline;
- recognising the point at which deferral is the correct decision.
An assessor who is strong on the first two and weak on the fourth does not need a different temperament. They need practice in a specific and fairly narrow situation.
Four conditions for deliberate practice
Practice alone does not build judgement; repetition of the same comfortable task builds fluency and little else. Four conditions tend to distinguish practice that develops capability from practice that merely fills a diary.
A defined standard. The assessor must know what good reasoning looks like before they attempt it, not after. Showing two recorded rationales, one strong and one weak, and discussing why, sets that standard cheaply.
Immediate feedback. Feedback delivered a term later is a comment on history. Feedback delivered within the session, or within days, changes the next attempt.
Repetition with variation. The same core skill applied to different sectors, methods and levels. Judgement trained only on familiar evidence tends to collapse the first time the evidence looks unfamiliar.
A coach who can articulate the standard. Not simply a more experienced colleague, but one able to explain why a decision is defensible, in language the trainee can reuse.
Calibration: the loop that makes practice count
Without calibration, practice entrenches idiosyncrasy. An assessor can become highly skilled at applying a personal threshold that differs from the team's.
Standardisation sessions address this, provided they compare reasoning rather than only outcomes. A room that agrees a decision and moves on has tested nothing. A room that asks two assessors with the same outcome to explain their routes to it will usually find that one reached the right place for the wrong reason, and that discovery is the value of the exercise.
Blind review helps here. Circulate a sample of evidence in advance without commentary, collect independent decisions and rationales, then compare. The disagreements are the agenda.
Rehearsing the decisions that rarely arrive
Most assessors meet a genuinely difficult judgement a handful of times a year: evidence that is almost but not quite sufficient, a witness statement that reads as coached, a candidate whose adjustments have not been applied consistently, a small employer applying gentle pressure for a pass.
Because these cases are rare, they are rarely practised, and because they are rarely practised, they are handled unevenly. A scenario bank addresses the imbalance directly. Gather a dozen anonymised cases from the team's own history, strip the identifiers, and work through them in small groups with a written rationale required at the end. Ninety minutes a month, sustained across a year, will cover more difficult judgements than most assessors will meet in post.
Consistency without flattening
There is a legitimate worry beneath the resistance to all of this. Standardisation, pushed too far, produces assessors who think alike, notice alike and defer to the group. Sector specialists who bring deep occupational knowledge would be sanded down to a common denominator.
The distinction worth holding is between consistency of threshold and uniformity of method. A calibration exercise should establish where the line sits, not the route by which each assessor arrives there. Two assessors can agree that a body of evidence is insufficient while noticing entirely different things about it, and the team is stronger for that divergence, provided the threshold holds.
What this means for training design
A single induction day followed by an annual refresher will not build judgement, because it cannot supply the repetition or the feedback that the task requires. What works is less dramatic: short, spaced sessions, a written record of what was rehearsed rather than merely attended, and a standing expectation that assessors bring their difficult cases to the group rather than resolving them privately.
The myth of the natural assessor is convenient, and it costs a great deal. It excuses providers from designing practice, and it leaves capable practitioners believing that their uncertainty is a personal deficiency rather than a signal that they have not yet been given the right conditions in which to learn.