October 5, 2026

How to Evaluate Watchlist Screening Software: A Scorecard

Axle Team

Key takeaways

Evaluate watchlist screening solutions on seven dimensions: list coverage, matching logic, secondary identifiers, alert volume, API and latency, audit trail, and adjudication workflow.
Weight the scorecard to your risk profile. A payments fintech and a community bank should not score latency the same way.
Compare alert volume only at matched detection. A vendor that generates fewer alerts by missing true matches is not better.
Run the bake-off on your own data, with a seeded test file of known hits, not on the vendor's demo set.
Adjudication workflow often drives more total cost than the license. Score it as heavily as matching.

How to evaluate watchlist screening solutions

The best way to evaluate watchlist screening solutions is a weighted scorecard tested on your own data. Score list coverage, matching logic, secondary identifier use, alerts per 1,000 records screened, API latency, audit trail, and adjudication workflow, then weight each dimension to your sanctions risk assessment and compare vendors at the same detection rate.

Most RFPs skip the last part. Feature checklists are easy to fill in, and every vendor ticks "fuzzy matching" and "configurable thresholds." The difference shows up when you run 50,000 of your own customer records through each engine and look at what comes back.

OFAC doesn't prescribe a tool, but it does set the bar for how you choose and maintain one:

To the extent information technology solutions factor into the organization's internal controls, the organization has selected and calibrated the solutions in a manner that is appropriate to address the organization's risk profile and compliance needs, and the organization routinely tests the solutions to ensure effectiveness.
— OFAC, A Framework for OFAC Compliance Commitments (May 2019)

"Selected and calibrated" is the phrase to hold onto. An examiner will ask why you picked the system and how you set it, and your scorecard is the first half of that answer.

The seven dimensions

1. List coverage and update cadence

Which lists are included out of the box: OFAC SDN and non-SDN lists, UN, EU, UK, and any local regimes you need. Then ask how fast a new designation reaches your screening engine, and whether delta files are supported. OFAC's framework lists failure to incorporate list updates among the root causes of sanctions violations.

2. Matching logic

Fuzzy matching is table stakes. What matters is which algorithms run, whether they're configurable by list and by data type, and how they handle transliteration, name reordering, and non-Latin scripts. Ask whether the vendor can explain why a given pair scored what it scored. For more on the mechanics, see our guide to fuzzy name matching in sanctions screening.

3. Secondary identifiers

Date of birth, nationality, place of birth, ID numbers, addresses, and for entities, registration numbers and SWIFT BICs. Good engines use these to score and suppress, not just to display. OFAC specifically cites missing identifiers as a failure mode:

At times, organizations have failed to update their sanctions screening software to incorporate updates to the SDN List or SSI List, failed to include pertinent identifiers such as SWIFT Business Identifier Codes for designated, blocked, or sanctioned financial institutions, or did not account for alternative spellings of prohibited countries or parties...
— OFAC, A Framework for OFAC Compliance Commitments, Root Causes, Section VI

4. Alert volume per 1,000 records

Measure alerts per 1,000 records screened at a fixed detection rate. Report it separately for customer screening and payment screening, since the two behave very differently.

5. API and latency

For real-time onboarding and payments, measure response time under your peak volume, not the vendor's median. Ask about batch throughput for full-book rescreens and how the system behaves when a list update and a rescreen collide. Our comparison of batch vs real-time screening covers when each mode matters.

6. Audit trail

Every alert should show the input record, the list entry and version, the match score, the rule or threshold that fired, the reviewer, the decision, and the rationale. If you change a threshold, you should be able to show what the system would have produced before and after.

7. Adjudication workflow

This is where your analysts spend their day. Wolfsberg sets a clear floor for what the reviewer screen must do:

The screening application must clearly present an alert for review by trained sanctions personnel... the application must present all relevant data from the FI and the sanctions lists for decision making and allow reviewers to make a decision based on the validity of that data and, thereafter, record relevant rationale.
— The Wolfsberg Group, Wolfsberg Guidance on Sanctions Screening (2019), Section 3.2

Score maker-checker support, good-guy list governance, bulk dispositions with controls, and how easily an alert moves to escalation.

A weighted scorecard you can adapt

The weights below are a starting point, not a standard. Adjust them to your sanctions risk assessment and document why.

List coverage and cadence — Suggested weight: 15%. What a 5 looks like: All required regimes, delta updates, documented update SLA. What a 1 looks like: Manual list loads, no stated update timing.
Matching logic — Suggested weight: 20%. What a 5 looks like: Configurable per list and field, explainable scores, non-Latin support. What a 1 looks like: Single global threshold, opaque scoring.
Secondary identifiers — Suggested weight: 15%. What a 5 looks like: DOB, nationality, IDs, BICs used in scoring and suppression. What a 1 looks like: Identifiers displayed only.
Alert volume at matched detection — Suggested weight: 15%. What a 5 looks like: Lowest alerts per 1,000 with no missed seeded hits. What a 1 looks like: Low volume achieved by missing seeded hits.
API and latency — Suggested weight: 10%. What a 5 looks like: Meets peak-volume latency and batch throughput targets. What a 1 looks like: Degrades at peak, no batch mode.
Audit trail — Suggested weight: 10%. What a 5 looks like: Full lineage per alert, threshold change history. What a 1 looks like: Decision only, no inputs or versions.
Adjudication workflow — Suggested weight: 15%. What a 5 looks like: Maker-checker, full context on one screen, structured rationale. What a 1 looks like: Free-text notes, context in other systems.

Score each vendor 1 to 5 per dimension, multiply by the weight, and sum. Keep the scoring sheet: it becomes part of your vendor selection file.

Practitioner view: run the bake-off at matched recall

Vendors quote false-positive reductions, but rarely at the same detection rate. Tighten any engine's threshold and alert volume drops. So does detection. A comparison that ignores that is a comparison of threshold settings, not products.

The fix is a two-part test file. Part one: a seeded set of known true matches built from list entries, with the variations that trip real systems, such as transliterations, reordered names, missing middle names, partial dates of birth, and entity name suffixes. Part two: a sample of your own customer and counterparty records, including last quarter's cleared alerts.

Set each engine so it catches every seeded hit, then count alerts per 1,000 on the real sample. That is the number that predicts your staffing. Wolfsberg's guidance points the same way, listing "the sophistication and configurability of the matching software" and workflow configurability among the elements to weigh in vendor selection.

One more trade-off: an engine that alerts a little more but presents every alert with full context can cost less to operate than a quieter engine that sends analysts into three other systems per alert. That's why adjudication gets real weight in the scorecard, and why some teams now put an AI review layer, like Baley, Axle's sanctions screening agent, after the engine to pre-assemble evidence for each alert. See also our breakdown of sanctions screening false positives.

FAQ

What should I look for in watchlist screening software?

List coverage and update speed, configurable and explainable fuzzy matching, real use of secondary identifiers, alert volume at a fixed detection rate, latency under peak load, a complete audit trail, and an adjudication workflow that captures rationale.

How do I compare false positive rates between screening vendors?

Test every vendor on the same file, with a seeded set of known true matches. Set each engine to catch all seeded hits, then compare alerts per 1,000 records screened. Otherwise you are comparing thresholds.

Does OFAC require a specific screening solution?

No. OFAC's 2019 framework expects technology to be selected and calibrated to your risk profile and routinely tested for effectiveness, but it does not endorse or require any product.

Reduce the adjudication load

Choosing the engine is half the problem. The other half is clearing what it produces, consistently and with a record an examiner can follow. See how Axle handles screening alerts, or estimate the analyst hours at stake with the ROI calculator.

See how Axle's agents work

Axle's AI agents investigate compliance alerts and document every decision for your analysts to review.

See Axle in action

The Alert Queue

Get the next issue in your inbox

Every other Tuesday, one short read for compliance and risk teams.

Every other Tuesday. No pitches. Unsubscribe anytime.

How to Evaluate Watchlist Screening Software: A Scorecard

See how Axle's agents work

Get the next issue in your inbox

More from the blog