Skip to content
Insights

Testing AI · Both industries

Why your AI demo is more accurate than your AI will ever be

Demos are scored on data that was cleaned after the fact. Production runs on data as it arrives. Here's how to tell, before you fund the build, which number you're looking at.

Emerald Isle Consulting · 5 min readPublished

What each test set can see

ProductionDemo test setDecisionpointNotes codedCoverage verifiedRecords linkedOutcome knownLEAKAGE · AFTER THE DECISION

In 30 seconds

  • Demos are scored on closed records that were corrected after the fact.
  • Production runs on data as it exists at the decision point, which is usually messier.
  • Leakage, information from after the decision, inflates demo accuracy.
  • Rebuild the test set as of the decision point, and agree a written target before you build.

Every AI steering committee has seen the slide: a model scoring in the low 90s on historical data, a confident recommendation to fund the build. Months later, in production, the same model is somewhere in the 50s or 60s, and nobody can quite explain what changed.

90s

accuracy in the steering committee demo, on historical data (illustrative)

50s–60s

the same model months later, on production data (illustrative)

Usually nothing changed about the model. What changed is the data. The demo was scored on one kind of data, and production runs on another.

The two datasets behind every AI decision

Insurance claims and healthcare claims share a pattern: information keeps arriving after the moment a decision has to be made. By the time a record is closed, someone has corrected it, coded it, verified it and filled in its gaps.

Models are almost always built from closed records, because closed records come with known outcomes to learn from. But they're deployed at the start of the process, where records are open, partial and messy.

Free-text descriptions

At decision time: Typed under time pressure

In the closed record: Coded by an expert

Key identifiers

At decision time: Missing, duplicated or split

In the closed record: Reconciled and linked

Coverage or eligibility

At decision time: A stale batch extract

In the closed record: Verified for the date in question

Documents

At decision time: Scanned PDFs and photos

In the closed record: Abstracted into fields

The outcome

At decision time: Unknown, which is the point

In the closed record: Known, and leaking into the inputs

Why the demo number is real, and irrelevant

The demo isn't faked. It is genuinely accurate at the question it was asked. The trouble is that the question was easier than the one production asks. It isn't predicting how a case will develop from what's known on day one. It's recognizing how a case developed, using details written down weeks later.

Data scientists call one form of this leakage: information from after the decision point creeping into the training data. A field that only gets filled in once a claim turns complex will look like a brilliant predictor of complexity. In production, it's empty at exactly the moment the model needs it.

A model tested on closed records is being graded on information it will never see when it matters.

The same pattern, two industries

In P&C insurance

A first-notice triage model is trained on closed claims, where an adjuster has coded the cause of loss, confirmed the parties and entered the estimate. At first notice, the model gets a call-center note and a scanned PDF.

In health plans

A fraud model is trained on closed investigations, where investigators have stitched one clinician's several provider IDs into one identity. In live claims, that clinician looks like several unrelated providers.

Different industries, same mistake: the training data had already been fixed by hand, and production data hadn't.

How to get a number you can trust

  1. 1

    Rebuild the test set as of the decision point.

    For every record in the test set, use only the data that existed at the moment the decision would have been made. This single change is usually where the demo number falls.

  2. 2

    Audit every input for when it gets filled in.

    Any field that's populated after the decision point is a leakage risk. Drop it, or replace it with what was actually known then.

  3. 3

    Grade the systems that feed the model.

    Point-in-time testing needs history the systems may not keep. If prior states are overwritten, that's a lineage gap, and it has to be fixed before you can even measure the model honestly.

  4. 4

    Write the target down before you build.

    Agree in writing what accuracy or precision is good enough, measured on decision-time data. It turns a demo into a commitment.

If the honest number is still good, you have a real model and a stronger case for funding it. If it isn't, you've learned it in weeks rather than after go-live, and you know exactly which data to fix first.