In 30 seconds
- Name the use case and its decision point first; every check is measured against it.
- Fifteen checks across intake, claims, policy, documents and the joins between systems.
- Your own team can test most of them in a few days with a sample of real claims.
- Sort failures into blocks, degrades and audit risk, then fix the blockers on the path first.
Claims is where most carriers start with AI, for good reason: high volumes, expensive decisions and a lot of manual work. It's also where pilots most often fail, because claims data is created under time pressure at first notice and corrected for months afterward.
This checklist is organized by the systems a claims use case reads from. For each check there's a quick way to test it and what a failing answer looks like. A failing answer doesn't mean you can't build. It means you know what to fix first.
Start with the decision, not the data
Write down the use case leadership has funded and the exact moment the model decides. For triage it's first notice of loss. For a document model it might be when a file is assigned. For fraud it's often the first payment. Every check below is measured against that moment, because a model can only use what exists then.
If you can't name the decision point, you can't test whether the data is ready for it.
FNOL intake
| Check | A failing answer looks like | How to test it |
|---|---|---|
| Is the loss captured in structured fields? | Most of the detail is in a free-text description | Pull 100 recent first notices; count how many have cause, location and damage type as fields |
| Is every channel captured the same way? | Each channel has its own form and its own codes | Compare fields from phone, web, app and agent notices |
| Does anyone know what each intake field means? | Different answers, and no data dictionary to settle it | Ask two teams to define five intake fields |
Is the loss captured in structured fields?
A failing answer looks like: Most of the detail is in a free-text description
How to test it: Pull 100 recent first notices; count how many have cause, location and damage type as fields
Is every channel captured the same way?
A failing answer looks like: Each channel has its own form and its own codes
How to test it: Compare fields from phone, web, app and agent notices
Does anyone know what each intake field means?
A failing answer looks like: Different answers, and no data dictionary to settle it
How to test it: Ask two teams to define five intake fields
Claims system
| Check | A failing answer looks like | How to test it |
|---|---|---|
| Are key terms defined once? | Three definitions, so training labels disagree | Ask claims, finance and actuarial what an "open claim" is |
| Is history kept? | Changes overwrite the old values | Try to show a claim's status and reserve as of a past date |
| Where do the outcome labels come from? | Set by hand, late, or only on some claims | Trace how severity, fraud or litigation flags are set |
Are key terms defined once?
A failing answer looks like: Three definitions, so training labels disagree
How to test it: Ask claims, finance and actuarial what an "open claim" is
Is history kept?
A failing answer looks like: Changes overwrite the old values
How to test it: Try to show a claim's status and reserve as of a past date
Where do the outcome labels come from?
A failing answer looks like: Set by hand, late, or only on some claims
How to test it: Trace how severity, fraud or litigation flags are set
Policy and coverage
| Check | A failing answer looks like | How to test it |
|---|---|---|
| Can you see coverage as of the date of loss? | Only today's policy, or endorsements overwritten | Pick 20 claims; retrieve coverage as it stood that day |
| How fresh is coverage data for the claims team? | A weekly batch, when triage needs it at first notice | Check how policy data reaches claims |
| Is each insured one record? | The same insured under several IDs | Profile duplicate policyholders |
Can you see coverage as of the date of loss?
A failing answer looks like: Only today's policy, or endorsements overwritten
How to test it: Pick 20 claims; retrieve coverage as it stood that day
How fresh is coverage data for the claims team?
A failing answer looks like: A weekly batch, when triage needs it at first notice
How to test it: Check how policy data reaches claims
Is each insured one record?
A failing answer looks like: The same insured under several IDs
How to test it: Profile duplicate policyholders
Documents and notes
| Check | A failing answer looks like | How to test it |
|---|---|---|
| How much of the claim lives only in documents? | Estimates, statements and repair details exist only as images | Sample claim files; list facts found only in PDFs, photos or adjuster notes |
| Can documents be read by a machine today? | Scanned images with no extracted text | Check whether text is extracted and stored with the claim |
How much of the claim lives only in documents?
A failing answer looks like: Estimates, statements and repair details exist only as images
How to test it: Sample claim files; list facts found only in PDFs, photos or adjuster notes
Can documents be read by a machine today?
A failing answer looks like: Scanned images with no extracted text
How to test it: Check whether text is extracted and stored with the claim
Across systems
| Check | A failing answer looks like | How to test it |
|---|---|---|
| Is there one claim key across channels and systems? | Matches by name and date, or not at all | Join intake, claims and payments for 100 claims |
| Can a reported figure be traced to its source? | The trail ends at the warehouse load | Trace paid loss on one report back to the claims system |
| How long does an AI team wait for access? | Months, or one administrator approves everything | Time the last approved data request |
| Are personal fields classified? | Nobody's sure which fields hold PII | Check whether claimant PII is tagged and masked where it isn't needed |
Is there one claim key across channels and systems?
A failing answer looks like: Matches by name and date, or not at all
How to test it: Join intake, claims and payments for 100 claims
Can a reported figure be traced to its source?
A failing answer looks like: The trail ends at the warehouse load
How to test it: Trace paid loss on one report back to the claims system
How long does an AI team wait for access?
A failing answer looks like: Months, or one administrator approves everything
How to test it: Time the last approved data request
Are personal fields classified?
A failing answer looks like: Nobody's sure which fields hold PII
How to test it: Check whether claimant PII is tagged and masked where it isn't needed
What to do with the answers
Sort every failing answer into three groups: it blocks the use case, it degrades the model's accuracy, or it creates audit risk. Then fix only the blockers on the use case's path before you build.
- 1
Fix the blockers first, even small ones.
A missing claim key or unstructured loss capture at intake will stop a triage model regardless of everything else.
- 2
Retest on first-notice data.
Rebuild the test set from data as it existed at the decision point, and agree a written accuracy target on that basis.
- 3
Schedule the rest.
Degraders and audit risks go on a roadmap, sequenced by the next use case that needs them.
In our sample read-out for a fictional carrier, the FNOL triage pilot fails on two checks from this list: unstructured loss capture and an undocumented intake system. See the report card →
If you'd rather get a first read in five minutes, the free scorecard asks ten questions across the same five dimensions a Data Readiness Verdict grades on real extracts.

