August 14, 2026

Questions to Ask Any AI Compliance Vendor (Before the Demo)

Axle Team

Key takeaways on AI compliance vendor evaluation

Regulators hold you accountable for your vendor's decisions. The 2023 interagency third-party guidance is explicit: outsourcing an activity never outsources the responsibility.
Most bad vendor picks are decided at the demo, on the vendor's data. The artifacts that matter — validation reports, audit logs, security documentation — exist before any demo.
The 2018 joint statement from FinCEN and the banking agencies protects well-structured pilots, even ones that surface gaps in your current program.
Ask for the decision-level audit trail of a single alert before you watch anything else. A vendor who can't produce one isn't ready for your examiners.

AI compliance vendor evaluation is the due diligence a financial institution runs before adopting AI for AML work: reviewing model governance and validation evidence, data security controls, decision-level audit trails, and pilot design, then documenting all of it to the standard set by the 2023 interagency guidance on third-party risk management. The questions below are the ones to settle before a demo is ever scheduled.

Why the evaluation starts before the demo

Demos are theater. The vendor controls the data, the alert mix, and the pacing, so the one thing a demo reliably proves is that the vendor can run a demo. The failure modes that end up in exam findings — drift on your alert mix, missing audit trails, quiet subcontractor dependencies — never appear in a scripted walkthrough.

The regulatory frame is the Interagency Guidance on Third-Party Relationships: Risk Management, issued by the Federal Reserve, FDIC, and OCC in June 2023. It replaced each agency's separate guidance and covers every stage of a vendor relationship, from planning through termination. Its central point:

"Importantly, the use of third parties does not diminish or remove banking organizations' responsibilities to ensure that activities are performed in a safe and sound manner and in compliance with applicable laws and regulations, including but not limited to those designed to protect consumers (such as fair lending laws and prohibitions against unfair, deceptive or abusive acts or practices) and those addressing financial crimes."
— Interagency Guidance on Third-Party Relationships: Risk Management (June 2023)

If an AI system clears a sanctions hit it should have escalated, the finding lands on your program, not the vendor's. That is why AI compliance vendor evaluation is a documentation exercise before it is a product comparison.

What the guidance tells you to check

The due diligence section of the guidance maps cleanly onto AI vendors: information security (it specifically mentions multifactor authentication, end-to-end encryption, and secure source code management), reliance on subcontractors — which for AI vendors includes foundation-model providers — audit rights, and performance measures. On that last one, the language reads like it was written for per-alert pricing:

"It is important to negotiate performance measures that do not incentivize imprudent performance or behavior, such as encouraging processing volume or speed without regard for accuracy, compliance requirements, or adverse effects on the banking organization or customers."
— Interagency Guidance on Third-Party Relationships: Risk Management (June 2023)

If the pricing model rewards throughput, the contract needs quality gates: QA pass rates, escalation accuracy, documentation completeness. Volume without accuracy is the exact behavior the agencies warned about.

The 13 questions

Model governance and validation

What documented model validation exists, who performed it, and will you share it under NDA?
How do you detect drift once our data flows through the system, and what triggers retraining or rollback?
Which decisions does the system make autonomously versus route to a human — and can we move that boundary ourselves?
What happens to performance when our alert mix changes: a new product, a new corridor, an updated sanctions program?

Data security and subcontractors

Where does our customer data live, who can access it, and is it ever used to train models shared with other customers?
Which subcontractors — including foundation-model providers — process our data, and under what contractual terms?
What do your latest SOC 2 report and penetration test show, and what is your incident notification commitment?

Auditability and explainability

For a single alert decision, can you produce the complete trail — inputs, sources consulted, reasoning, disposition — in a format an examiner can read?
Can our QA team sample and re-work the system's decisions the same way they sample human analysts today?
When the model is updated, can we tell which decisions were made under which version?

Pilot structure and exit

Will you run against our historical closed cases in parallel before touching production work?
Who scores the pilot — and are the metrics, sample, and pass thresholds agreed in writing before it starts?
If we terminate, how do we export our data and decision records, and in what format?

The pilot questions have regulatory cover. The 2018 joint statement from FinCEN, the Federal Reserve, FDIC, NCUA, and OCC addressed exactly this scenario:

"While the Agencies may provide feedback, pilot programs in and of themselves should not subject banks to supervisory criticism even if the pilot programs ultimately prove unsuccessful. Likewise, pilot programs that expose gaps in a BSA/AML compliance program will not necessarily result in supervisory action with respect to that program."
— Joint Statement on Innovative Efforts to Combat Money Laundering and Terrorist Financing (December 2018)

A parallel run on closed cases is the cleanest way to structure that pilot: no production risk, real data, and a direct comparison against decisions your team already made.

Demo red flags

Audit logs "aren't available in the demo environment." What it usually means: logging was bolted on after the fact. What to do: ask for a production-format log export of one demo decision.
Accuracy claims with no denominator ("2x more detections"). What it usually means: no agreed measurement methodology. What to do: get metric definitions in writing, computed on your data.
Demo runs only on curated synthetic alerts. What it usually means: edge cases untested. What to do: bring your own closed cases into the pilot.
Pricing is purely per alert processed. What it usually means: the volume incentive the 2023 guidance warns about. What to do: tie fees to quality-gated outcomes, not raw throughput.

The test that beats any demo

A workflow habit worth stealing: skip the demo-data argument entirely. Pull 25 alerts your team closed last quarter — 15 routine, five that went to SAR, five your QA marked as close calls — and have the vendor's system work them cold under the pilot agreement. Compare its dispositions and written rationales against what your team decided.

Two hours of reading that output tells you more than any demo, because close calls are where these systems earn or lose their keep. The trade-off is real: sharing the data requires security review first, so run this inside the pilot agreement rather than the sales cycle — which is itself a useful test of how quickly the vendor can clear your vendor-risk process.

FAQ

What should an AI compliance vendor evaluation checklist cover?

Five areas: model validation evidence, data security and subcontractor disclosure, decision-level auditability, pilot design with agreed metrics, and exit terms including data portability. Map each to the lifecycle stages in the 2023 interagency guidance — planning, due diligence, contract negotiation, ongoing monitoring, termination — and you have documentation an examiner will recognize.

Do regulators allow AI in AML compliance programs?

Yes. The 2018 joint statement encourages banks to consider, evaluate, and responsibly implement innovative approaches, including artificial intelligence, and commits the agencies not to penalize well-run pilots. No pre-approval is required, but your program obligations don't change — you stay responsible for outcomes and need the documentation to prove control.

How long should a pilot with an AI compliance vendor run?

Long enough to cover at least one full QA cycle and one model update from the vendor — in practice, most teams land on eight to twelve weeks. Shorter pilots test the happy path; the drift, versioning, and QA-sampling questions above only get answered with some calendar time.

Next step

Axle built its platform around the questions above — every agent decision is documented end to end, and your QA team can sample agent work the way it samples analysts. See how that works on the platform overview. If you're earlier in the process, the AI readiness playbook turns these questions into an internal scoping exercise, and the ROI calculator frames the business case your CFO will ask for.

Questions to Ask Any AI Compliance Vendor (Before the Demo)

More from the blog