August 10, 2026

How to Pilot an AI Agent in Your Compliance Program

Axle Team

Key takeaways

Regulators have encouraged AML pilots since 2018: the interagency Joint Statement says pilot programs "in and of themselves should not subject banks to supervisory criticism" — even if they fail.
The lowest-risk pilot design is shadow mode: run the AI agent on alerts your team already dispositioned and measure disagreement, not just accuracy.
Scope one queue and one decision type. Pilots that span onboarding, monitoring, and SAR drafting at once produce results nobody can interpret.
Define go/no-go metrics before the pilot starts — decision agreement, evidence completeness, QA acceptance — and hold the line in both directions.
Treat pilot documentation as the start of your model validation file. Write it as if an examiner will read it, because eventually one will.

To pilot an AI agent in AML compliance, pick a single alert or case queue, run the agent in shadow mode on work your analysts have already completed, and compare its decisions and written rationale against the human outcomes. Set go/no-go metrics before the pilot starts, involve model risk management early, and document everything as validation evidence.

That is the whole playbook in one paragraph. The rest of this post covers why an AI pilot in AML compliance carries less supervisory risk than most BSA officers assume, how to structure the work, and which metrics actually tell you whether to expand, extend, or kill it.

Why an AI pilot in AML compliance is safer than you think

The most common objection to piloting AI in a BSA/AML program is not technical. It is "what will our examiners say?" That question was answered in December 2018, when the Federal Reserve, FDIC, FinCEN, NCUA, and OCC jointly encouraged banks to test innovative approaches — naming artificial intelligence specifically.

"While the Agencies may provide feedback, pilot programs in and of themselves should not subject banks to supervisory criticism even if the pilot programs ultimately prove unsuccessful."
— Joint Statement on Innovative Efforts to Combat Money Laundering and Terrorist Financing, December 3, 2018

The statement goes further than most people remember. It directly addresses the scenario compliance officers fear most — a pilot that surfaces suspicious activity the incumbent system missed:

"For example, when banks test or implement artificial intelligence-based transaction monitoring systems and identify suspicious activity that would not otherwise have been identified under existing processes, the Agencies will not automatically assume that the banks' existing processes are deficient."
— Same joint statement

FATF reached a similar conclusion in its 2021 report on new technologies, which found they can make AML/CFT measures faster, cheaper, and more effective. None of this is a free pass: the same joint statement is explicit that banks must keep meeting BSA obligations while piloting, and that management should evaluate when an innovation is sufficiently developed to replace or augment existing processes. That evaluation is precisely what a well-designed pilot produces.

A six-step AI pilot design

Scope one queue, one decision. Pick a single alert type or case stage — sanctions alert adjudication, transaction monitoring triage, or EDD case assembly. Narrow scope is what makes the results interpretable.
Pull a completed-work sample. 200–500 alerts or cases your team already dispositioned, including their QA outcomes. This becomes your ground truth, and it costs the production team nothing.
Run in shadow mode. The agent works the same alerts with the same data access, producing a decision and a written rationale for each. Nothing it does touches production.
Measure disagreement, not just accuracy. Every case where agent and analyst diverge goes to a senior reviewer blind to which side produced which disposition. Some disagreements will be agent errors; a meaningful share are usually analyst errors. Count both.
Involve model risk management from week one. Treat the pilot documentation as the start of the validation file: data lineage, configuration versions, sampling method, error taxonomy.
Hold the go/no-go line. Decide in advance what expansion requires — for example, decision agreement in line with your human QA pass rate and zero unexplainable decisions — and don't move the goalposts in either direction.

Metrics that decide go/no-go

Decision agreement rate — whether the agent reaches the same disposition as your analysts. Measure the percentage of sampled cases with matching disposition, with disagreements adjudicated blind.
Evidence completeness — whether every conclusion is supported and citable. Score a QA checklist on a case sample, using the same rubric you use for humans.
QA acceptance rate — whether reviewers would sign off without edits. Track the percentage of agent write-ups accepted by QA with no substantive changes.
Escalation calibration — whether the agent knows what it cannot decide. Track the percentage of genuinely ambiguous cases the agent flags for human review rather than guessing.

Notice what is not on the list: raw alert-clearance speed. Speed matters — you can put numbers on your own queue with our ROI calculator — but a pilot that proves speed without proving decision quality proves nothing an examiner will care about.

Where pilots fail

The pattern we see most often is a workflow problem, not a model problem: teams grade the agent against the analyst's disposition as if the analyst is always right. On the first blind-adjudicated sample, the human baseline usually turns out noisier than anyone expected — the same alert dispositioned differently by two analysts, or cleared with a one-line note today's QA would reject. If you don't blind-adjudicate disagreements, you will score legitimate catches as errors and kill a pilot that was outperforming the baseline.

FATF's framing of the end state is the right one — technology plus human judgment, not either alone:

"When used responsibly and proportionally, innovative AML/CFT technologies can help identify risks and focus compliance efforts on existing and emerging challenges, but manual review and human input remains very important."
— FATF, Opportunities and Challenges of New Technologies for AML/CFT, July 2021

The other recurring failure is scoping the pilot around what is easiest to connect rather than what the team actually spends time on. If 60% of analyst hours go to evidence gathering and write-ups, a pilot that only scores risk adds little — pilot the work, not the model. That is how we scope pilots for Baley, our screening agent: the same alerts, the same evidence standards, the same QA rubric your analysts are held to.

FAQ

Do we need regulator approval to run an AI pilot in AML compliance?

No. The 2018 joint statement encourages early engagement with your regulator and says the agencies are open to discussing pilot programs, but there is no pre-approval requirement. The operative constraint is that pilots run alongside — not instead of — your existing BSA/AML processes.

How long should the pilot run?

Four to eight weeks of casework is typical when you start from a completed-alert sample: long enough to cover volume variation and at least one full QA cycle, short enough that the configuration being evaluated doesn't drift mid-pilot.

Does piloting create new regulatory expectations for our program?

The agencies stated in the joint statement that implementing innovative approaches will not result in additional regulatory expectations, and that a pilot exposing gaps in your program will not necessarily trigger supervisory action. Your existing obligations — an effective, risk-based program — are unchanged.

If you are scoping a pilot this quarter, start with the readiness questions: data access, QA baselines, model risk involvement, and queue selection. Our AI Readiness Playbook walks through each one, with the checklists we use when standing up pilots with compliance teams.

How to Pilot an AI Agent in Your Compliance Program

More from the blog