Calibrating the Assessor's Eye: A Repeatable Standardisation Session Format for UK Providers

Why completed portfolios flatter consistency

Most standardisation sessions begin with finished portfolios. Assessors bring decisions already made, evidence already mapped and rationales already written. The group then compares those artefacts and looks for agreement. The trouble is that the evidence has been shaped by the decision. Comparing completed portfolios reveals whether colleagues write similar rationales; it does not reveal whether they would reach similar judgements from the same raw material.

That distinction matters for internal quality assurance. An IQA function needs to know whether assessors are calibrated at the point of judgement, not simply whether they can produce tidy paperwork afterwards. A session that only reviews finished work can give a false sense of consistency and leave divergent judgement untouched.

The case for near-live evidence

Near-live evidence means material that has not yet been assessed and signed off. It might be a video observation, witness testimony, a candidate product with partial mapping, a simulation recording or a set of workplace photographs with brief notes. The common factor is that the assessor must decide in the room, under the same conditions of uncertainty that apply in practice.

This approach does not require elaborate production. A short phone video, an anonymised candidate file or a written witness statement can supply enough ambiguity to test judgement. The purpose is not to trick assessors but to surface the reasoning that sits beneath a decision.

A 90-minute session format

The following structure works for groups of four to ten assessors and can be repeated each half-term. It assumes one facilitator, a small set of near-live evidence and a shared standard or assessment plan.

Phase 1: Frame the decision (5 minutes)

The facilitator states the criterion under review and the decision to be made: for example, whether the evidence is sufficient, sufficient with conditions, or not yet sufficient. Participants are reminded that the aim is calibration, not consensus for its own sake.

Phase 2: Independent blind judgement (15 minutes)

Each assessor reviews the evidence alone and records a decision with a brief rationale. No discussion is permitted at this stage. The written record should state the evidence source, the criterion tested and the main reason for the decision.

Phase 3: Pair comparison (15 minutes)

Assessors compare decisions in pairs. They identify where they agree, where they diverge and which piece of evidence drove the difference. Pairs then prepare one question for the wider group.

Phase 4: Facilitated challenge (25 minutes)

The facilitator draws out divergent judgements without resolving them too quickly. The group tests the reasoning against the standard. The facilitator's role is to keep the focus on evidence and criteria rather than seniority or confidence of delivery.

Phase 5: Decision record and calibration log (20 minutes)

Each assessor completes a short decision record. The facilitator collates the range of decisions and the reasons for divergence into a calibration log. This log becomes the audit trail for IQA and CPD purposes.

Phase 6: Close and commit (10 minutes)

Participants identify one change they will make to their own practice, such as asking a more precise question at the evidence-gathering stage or recording a confidence rating. The facilitator notes follow-up actions.

Three sample scenarios

Scenario A: Early years observation. A video shows a Level 3 early years candidate supporting a child through a transition. The interaction is warm and effective, but the specific criterion concerns partnership with parents. The evidence contains no clear parental communication. Assessors must decide whether the observation is sufficient for that criterion or whether further evidence is required.

Scenario B: Engineering practical. A witness testimony states that the candidate followed safe isolation procedure. The video angle obscures the lock-off step. Assessors must weigh the witness statement against the visual gap and decide whether the evidence is sufficient, conditional or insufficient.

Scenario C: Digital portfolio. A marketing candidate submits a polished campaign report with analytics screenshots that lack date stamps. The report is persuasive but the evidence trail is incomplete. Assessors must decide whether to accept the portfolio, request clarification or treat the criterion as not yet met.

Facilitation questions that test judgement

• What did you see or hear that led to your decision?
• What did you not see that you would expect if the criterion were fully met?
• Which part of the evidence carries the most weight, and why?
• If a colleague reached the opposite decision, what would they be relying on?
• What would you need to see to change your mind?
• Where does the standard sit between these two positions?

These questions are deliberately neutral. They ask assessors to externalise reasoning without inviting defensiveness.

Decision-record prompts

A useful standardisation record captures the criterion, the evidence source, the sufficiency judgement, a confidence rating, any dissent and the follow-up action. It need not be lengthy. A one-page pro forma is sufficient if completed consistently. Over time, the records show whether the group is converging on similar reasoning or merely similar wording.

Evidencing impact for IQA and CPD

Standardisation records can feed directly into IQA sampling and CPD planning. A calibration profile built over several sessions shows where divergence persists and which criteria attract inconsistent judgements. That profile can inform targeted training, peer observation or further standardisation activity.

For CPD purposes, attendance and participation can be recorded alongside the specific judgement skills rehearsed. The record should note the scenarios used, the decisions reached and the follow-up actions taken. This is more defensible than a register of attendance alone.

Pitfalls to avoid

Three pitfalls recur. The first is seniority dominance, where the most experienced assessor speaks first and closes down debate. The second is scenario drift, where the discussion moves away from the criterion and into general complaints about evidence quality. The third is treating consensus as proof of correctness. Agreement is useful only when it is grounded in shared reasoning.

Conclusion

Standardisation is a rehearsal of judgement, not a review of paperwork. A session built around near-live evidence, structured challenge and written decision records gives assessors a genuine opportunity to calibrate their eye. It also gives IQA and CPD functions something more valuable than a set of completed portfolios: a record of how judgements are made, where they diverge and what the provider is doing about it.