A sales rep finishes a practice call with an AI customer who wanted to cancel. The results screen reads: Empathy 82, Problem solving 76, Composure 88. It looks clean and exact.
The 82 in this post is a stand-in. It is not a real user's score and it is not a threshold from a study. We picked it because a number like 82 feels measured in a way that "good", or a round 80, does not. That feeling is where a buyer's real question starts: what is holding this number up?
We work on that question every day at EVRE. This post covers what measurement research says about it, and what we do in our own product to keep a score steady.
Write the criterion first
"We measure empathy" is too broad to measure. Does empathy mean repeating the customer's concern back accurately? Naming how they feel? Checking what they need before offering a fix?
Each of those is a behavior you can look for, one at a time. At EVRE a criterion is a written question you can answer from the conversation itself. For the cancellation call:
- Did the rep restate the customer's main concern in the customer's own terms?
- Did they name the frustration instead of moving past it?
- Did they confirm what the customer needed before offering a retention deal?
- Did they avoid promising anything outside their authority?
Empathy 82 is a second layer, built from the answers. The testing standards put things in the same order. In the joint standards of the American educational and psychological associations, validity is the degree to which evidence supports the interpretation of a score for its intended use [1]. Without a written criterion there is nothing for that evidence to attach to.
Work on LLM evaluation points the same way. LLM-Rubric, published at ACL 2024, splits an evaluation into several explicit rubric questions instead of asking for one overall judgment, and calibrates the model's answers against human raters [4].
A sentence next to every result
If a report says "the rep was empathetic throughout", the first question is: where?
Every EVRE finding arrives with the sentence that produced it. "Named the frustration" shows the line the rep said. "Went beyond their authority" shows the promise. Whoever reads the report can check the call for themselves, and when the model gets something wrong, you can see exactly where.
The NIST AI Risk Management Framework does not define trustworthy AI as accuracy alone. Alongside validity and reliability it lists explainability, transparency and accountability [2]. For a behavior score that comes down to one requirement: a trail from the number back to the conversation.
Scoring the same conversation twice
A measurement should give a similar answer when it looks at the same performance again. The testing standards treat this as reliability: changing raters, repeating the measurement and changing conditions all move scores [1].
AI judges have known sources of that movement. In the MT-Bench study (NeurIPS 2023), a strong LLM judge agreed with human preferences over 80% of the time, the same level humans reach with each other. The same study found three systematic biases: sensitivity to the order of answers, a preference for longer answers, and a tendency to favor text that resembles the judge's own [5].
The second one matters most for behavior. A rep who talks longer has not communicated better because of it. One short, well-placed line that calms an angry customer should not lose to a long paragraph that only sounds professional.
Three things keep EVRE's scores steady:
- Fixed measurement settings. The settings of the model that measures are pinned in the product code and guarded by a test. A scenario cannot tune the instrument that measures it.
- Every result keeps its provenance. A result is stored with the scenario version, engine generation, model role and evaluation run that produced it. When a criterion changes, older results are still read under their own version.
- More than one reader. After a session, three specialist readers review the conversation from different angles and a fourth reconciles what they found.
Before a scenario goes live, we also run it many times against AI participants and watch, with deterministic code checks rather than the model's own judgment, whether the measurement responds to the same behavior the same way.
Human raters disagree too
The most sensible way to test an AI evaluator is to have domain experts score the same conversations. They use the same criteria and score independently. Then you look at more than the averages: where they agree, and where they split.
Treating the human score as a perfect answer key would be a mistake. LLM-Rubric found that humans do not fully agree with one another either [4]. JUDGE-BENCH, published at ACL 2025, compared 11 models on 20 human-labeled evaluation datasets, and quality varied sharply by model and by task [6]. A judge that works well on one job may not earn the same trust on another.
That is why we do not read model-versus-human differences as a leaderboard. We build our calibration sets from hard cases on purpose: moments where the evidence pulls different ways across the conversation, or where a behavior is only partly there. We break disagreement down by criterion, by where in the conversation it happens, and by how long the evidence is. That breakdown shows where a judge goes wrong, and why.
Automated scoring needs this checking for as long as it runs. ETS's work on its e-rater essay scorer describes monitoring human and machine agreement over time as an ongoing part of operating the system [7].
Is 82 really different from 79?
A two-digit score does not prove a two-digit level of precision. The testing standards ask that precision be taken into account before drawing conclusions from the difference between two scores [1]. If a system cannot reliably tell 82 from 79, it should not present those three points as a meaningful gap.
At EVRE a score never arrives on its own. Next to the 82 the rep sees which behaviors showed up, which were missing, and the sentences behind each. For someone trying to improve, that is the useful part: what to do differently on the next call.
The number itself is an active line of R&D for us, in three steps: calibrate against expert raters first, establish how human and model scores relate, then run the measurement on its own with periodic audits. The aim is for the score given to the same behavior to settle into a narrower range as calibration data builds up.
Coaching scores and hiring decisions
"You restated her concern accurately, but you offered the discount before you knew what she needed" is a coaching sentence. "Do not hire this candidate" is a decision of a different weight, and it needs stronger evidence.
The Society for Industrial and Organizational Psychology's 2023 guidance holds AI-based selection assessments to the same scientific standards as traditional employment tests: job relatedness, validity, reliability and fairness [3]. It also notes that a model's ability to extract many signals does not make those signals job-related. The EU AI Act treats certain uses in recruitment and in managing workers as high-risk, and what decides it is the system's intended purpose and its role in the decision [8].
The same criterion can feed a coaching signal, a post-training progress measure and an input to a hiring decision. Each of those uses needs its own weight of evidence.
Five questions to ask before you buy
If you are evaluating an AI roleplay tool, ask for these in the demo:
- The written criteria. What questions sit behind the score?
- The evidence sentence. Can you click a finding and land on the line in the conversation?
- A second scoring. How much does the result move when the same conversation is scored again?
- The human comparison. Where does it agree with expert raters, and where does it not?
- Version history. What happens to old results when a criterion or model changes?
In EVRE you can see the first two in every report. We walk through how we handle the other three in a demo, using your own scenario. See how we work with organizations on the business page, or request a demo.
For the team side of this, read Beyond the score screen. For the buying questions, read Evaluating AI roleplay for HR training.
