September 11, 2026

How to Measure an AI Agent Against Your Analysts

Axle Team

Key takeaways

There is no regulator-issued scorecard for AI agents. The April 2026 revised interagency model risk guidance (SR 26-2) explicitly carves generative and agentic AI out of scope — your own governance framework has to fill the gap.
The most defensible way to evaluate AI agent compliance performance is a blind parallel run against an adjudicated ground-truth set, not against raw analyst decisions.
Agreement rate alone is a weak metric. Measure missed-risk capture, evidence quality, and narrative quality separately — they fail independently.
Override analysis is your best ongoing signal after go-live. A rising override rate is an early warning, exactly as model risk guidance has long treated it.
Keep the evaluation artifacts. The exam question will not be "is the agent good?" but "show me how you know."

To evaluate AI agent compliance performance, run the agent in parallel with your analysts on the same live alert queue, score both against an adjudicated ground-truth set, and compare decision agreement, missed-risk capture, evidence quality, and narrative quality. Then track overrides in production and revalidate on a defined cycle, documenting everything for examiners.

How to evaluate AI agent compliance performance

Every BSA officer piloting an AI agent hits the same wall: the vendor demo looks strong, the sample outputs read well, and nobody can answer the CRO's question — is it actually as good as our analysts? Answering that requires treating the agent the way you would treat any high-materiality system: define what "performing as expected" means, measure it against a credible benchmark, and keep measuring after deployment.

The vocabulary already exists. The Federal Reserve, FDIC, and OCC issued revised supervisory guidance on model risk management in April 2026 (SR 26-2), which describes validation as evaluating "whether models perform as expected" including "an assessment of a model's reliability and its limitations." Notably, the agencies were candid about where agentic AI fits:

"Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance. Nonetheless, a banking organization's risk management and governance practices should guide the determination of appropriate governance and controls for any tools, processes, or systems not covered in this document."
— Federal Reserve, FDIC, and OCC, Supervisory Guidance on Model Risk Management, SR 26-2, April 17, 2026

Read that carefully: out of scope does not mean off the hook. You are expected to build the controls yourself. The practical move is to borrow the discipline of model validation — conceptual soundness, ongoing monitoring, outcomes analysis — and adapt it to an agent whose output is a decision plus a written rationale, not a score.

Build the benchmark before the pilot

The single most common mistake when evaluating AI agents for compliance work is benchmarking against whatever the analyst happened to decide on that alert. Analysts disagree with each other; if your ground truth is one analyst's call, you have capped the measurement at the noise level of your own team.

Build a golden dataset instead:

Sample 200–500 historical cases across alert types, risk tiers, and dispositions — include the hard ones (escalations, reopened cases, QA failures), not just clean closures.
Have senior reviewers adjudicate each case to a documented ground-truth disposition, resolving disagreements explicitly.
Freeze the set, version it, and record the adjudication rationale. This becomes your regression suite for every agent update and every model change.
Run the agent on the full set blind — no access to the original analyst's disposition.
Score analysts (or a sample of their historical decisions) against the same adjudicated truth, so the comparison is agent-vs-truth and analyst-vs-truth, not agent-vs-analyst.

What to measure — and what "good" looks like

Decision agreement: % of dispositions matching adjudicated ground truth, split by alert type and risk tier
Missed-risk capture: Of cases adjudicated as true escalations, how many did the agent escalate? Weight this heaviest
Evidence quality: Did the agent cite the right transactions, KYC facts, and screening hits? Scored on your existing QA rubric
Narrative quality: Would the write-up survive QA and, for SAR-adjacent work, support a filing decision?
Timeliness: Cycle time per case, measured on the live queue, not the demo environment

False negatives deserve asymmetric weight. An agent that closes 95% of cases correctly but misses the two genuine layering patterns in your sample is not "95% good" — it has failed the one property the program exists to protect.

After go-live: overrides are the metric that keeps working

Once the agent is in production, the parallel-run scores age quickly. The durable signal is the pattern of human overrides — every time a reviewer reverses or materially edits the agent's disposition, you have a labeled data point. The 2011 model risk guidance that shaped a decade of validation practice put it plainly:

"If the rate of overrides is high, or if the override process consistently improves model performance, it is often a sign that the underlying model needs revision or redevelopment."
— Federal Reserve, SR 11-7, Supervisory Guidance on Model Risk Management, April 2011

SR 26-2 superseded SR 11-7, but the logic survives: route every override into a monthly review, categorize the cause (missing data, judgment gap, policy drift), and set a threshold that triggers revalidation. This is also where "effective challenge" lives — SR 26-2 describes it as "the critical analysis conducted by objective experts who evaluate model risk and effect appropriate changes throughout the model lifecycle." Someone independent of the team that bought the agent should own this review.

Here is the practitioner observation most evaluations miss: the disagreements are worth more than the agreements. When we compare agent and analyst dispositions, the cases where they disagree are the highest-value QA sample in the program — score both sides blind with a senior reviewer, and you will find the split is rarely 100/0 in either direction. Some disagreements expose agent limitations; a surprising share expose analyst shortcuts that your QA sampling never caught, because random 5% sampling almost never lands on the interesting cases. An agent evaluation, done honestly, is also an audit of your human baseline. Budget for what you might find.

FATF made the complementary point about the end state:

"Combining the efficiency and accuracy of digital solutions with the knowledge and analytical skills of human experts produces more robust systems that can effectively respond to AML/CFT requirements whilst being fully auditable and accountable."
— FATF, Opportunities and Challenges of New Technologies for AML/CFT, July 2021

That is the design target: agents doing the volume, humans doing the challenge, and an evidence trail connecting the two. Axle's platform is built around this loop — every agent decision ships with cited evidence and a full audit trail so your QA team can score it the same way they score an analyst. If you are earlier in the process, the AI readiness playbook covers the groundwork, and the ROI calculator helps size the parallel-run business case.

FAQ

How long should a parallel run last before trusting an AI agent?

Long enough to cover a full cycle of your alert mix — for most transaction monitoring and screening queues that means at least one month-end and one quarter-end, plus enough volume for the false-negative measurement to be meaningful. Calendar time matters less than case diversity.

Do AI agents fall under model risk management requirements?

SR 26-2 states that generative and agentic AI models are not within its scope, but it also says your risk management and governance practices should determine appropriate controls for them. In practice, examiners will expect validation-like evidence: documented testing, ongoing monitoring, override analysis, and independent review.

What agreement rate with analysts is "good enough"?

There is no defensible universal number, because analysts disagree with each other and agreement with a flawed baseline is not accuracy. Evaluate AI agent compliance output against adjudicated ground truth, weight missed-risk capture heaviest, and set thresholds per alert type with your model risk or QA function.

Ready to run the comparison on your own queue? See how Axle's agents work — with evidence trails built for exactly this kind of evaluation — at axleruns.com/platform.

How to Measure an AI Agent Against Your Analysts

More from the blog