September 23, 2026

FairPlay Independently Evaluated Baley. Here Is What It Found.

Axle Team

FairPlay has completed an independent evaluation of Baley, Axle's screening agent, and published the results. Across 1,048 sanctions, PEP and adverse media cases built to be hard, Baley reached 97.5% accuracy, 99.2% precision and 97.5% recall. Nobody required this evaluation. We asked for it.

Why we asked to be tested

Marketing in compliance AI is full of claims, and vendors are usually cagey about the numbers behind them. We understand why. But our customers have to defend every screening decision to an examiner, and a vendor's own accuracy figure has never survived that conversation. "Trust us" is not a control.

So we decided to go the other way: benchmark our agents, hand the results to an independent party with no stake in the outcome, and publish whatever comes back. This evaluation is the first. Independent validation of agents like Baley is going to become a requirement, and we would rather build to that standard now than be told to later.

What FairPlay evaluated

FairPlay tested Baley on 1,048 screening cases across sanctions, politically exposed persons and adverse media. The set was built around the situations that actually break screening: transliterated names, aliases, legal name changes, conflicting dates of birth, and secondary identifiers that partially match. For each case Baley returned CLEAR or ESCALATE with a written rationale, and FairPlay graded the verdict.

The evaluation went beyond accuracy. FairPlay reviewed Baley's design and architecture, assessed the risks associated with its LLM components, and reviewed Axle's governance and change-management practices under model risk management principles, including SR 11-7 and SR 21-8.

The results

Overall accuracy: 97.5% across 1,048 scored cases
Precision: 99.2%
Recall: 97.5%
Specificity: 97.7%
By category: sanctions 97.9%, PEP 98.2%, adverse media 94.9%

In plain terms: when Baley reaches a verdict it is almost always right, and it catches the large majority of the cases that should be caught. The number we care about most is not the headline. It is which way the agent errs when it is wrong. Most of Baley's remaining misses sent a clean customer to a human for a second look. A false positive costs analyst time. A false negative costs a consent order. The two are not symmetric and should never be treated as if they were.

Adverse media is the lowest of the three categories, and we want to be direct about why. It is the most subjective part of screening. Whether a news item is reliable and relevant to a specific person is a judgment that experienced analysts disagree on, and it is also the area where an institution's own policy has the most influence on the right answer.

These are baseline numbers

Every figure above is Baley on a standard configuration, before it is tuned to an institution's own risk appetite and escalation policy. That tuning is the first thing we do before any customer goes live, and it is where each of these metrics improves. In this evaluation, a share of the misses traced to a default escalation setting rather than to the agent's reasoning. Those are exactly the settings that get configured to each institution's procedures during onboarding.

What our teams said

"AI agents are beginning to perform compliance work that used to sit squarely with human analysts. Banks shouldn't have to take a vendor's word for it that those agents work." — Kareem Saleh, Founder and CEO, FairPlay

"Any vendor can tell you their agent is accurate. Very few will hand it to a third party and publish whatever comes back. We asked FairPlay to test Baley on the cases that actually break screening: transliterated names, aliases, legal name changes, conflicting dates of birth. What our customers care about is not only how often the agent is right, but which way it errs when it is wrong. Those numbers are now on the record, and they can take them to their examiners." — Yanni Giannaros, Co-founder and CEO, Axle

What to ask any vendor

If you are evaluating AI for a compliance workflow, the accuracy number is the least useful thing a vendor will tell you. Ask these instead:

Who measured the accuracy, and did they have a stake in the result?
Was it measured on hard cases or on convenient ones?
Does the report show the direction of the errors, or only the headline?
What happens when the vendor ships a model update? Is there documented change management?
Can a decision from eight months ago be reproduced and explained?

Listen to the conversation

Yanni joined Kareem Saleh on FairPlay's Model Citizens podcast to walk through how the evaluation was run, how Baley reasons through a screening alert differently from a rules engine, the false positive versus false negative tradeoff, and where analysts stay in the loop as more of this work moves to agents.

Want to see how Baley works on your own screening alerts? Book a walkthrough or reach out at sales@axleruns.com.

See how Axle's agents work

Axle's AI agents investigate compliance alerts and document every decision for your analysts to review.

See Axle in action

The Alert Queue

Get the next issue in your inbox

Every other Tuesday, one short read for compliance and risk teams.

Every other Tuesday. No pitches. Unsubscribe anytime.

FairPlay Independently Evaluated Baley. Here Is What It Found.

See how Axle's agents work

Get the next issue in your inbox

More from the blog