We Put 376 Claims Through an AI Fact-Check and It Rejected 207:
what that proved

We did this to ourselves, which is why we can describe it

This chapter is about a mistake we made, at scale, and what it revealed.

We had assembled 376 factual claims about the insurance industry — payout rates, commission structures, enforcement actions, regulatory rules. Before publishing any of it we ran the whole set through an adversarial check: a second pass instructed to try to disprove every claim.

It rejected 207 of them. More than half.

That looked like rigour. It was not.

What the rejections actually were

When we went back and read the reasons, only about twenty of the two hundred and seven involved anyone retrieving a document that said something different. The rest were:

• The web address returned an error.
• The report was behind a paywall.
• "I could not find a primary source."
• And, in one case stated plainly: *the source report exists but I verified not one figure in

it.*

None of those are findings. A dead link is not evidence about whether a number is true.

So we put them all back and went to find the documents ourselves — going to the issuing institution directly, and separately working backwards from academic citations to the original, so that a reorganized website could not block both routes.

388 documents opened. The result:

About six in ten rejections were wrong. The claims were true. The documents existed. In two separate cases a report declared unreachable was downloaded in full, from a stable address, on the first attempt.

Why the error ran one way

Here is the part worth carrying out of this chapter, because it is not about software at all.

When a primary source is hard to reach — a dead government link, a members-only trade report, a paywalled journal — what stays freely reachable is whatever is most published and most optimized for being found. In consumer finance, that is overwhelmingly material produced by the industry being examined, or the official framing of the body being questioned. Both publish plenty, free, and well.

So scoring "no reachable source" as evidence against a claim is not neutral. It quietly hands the argument to whoever publishes most.

No one has to intend this. There is no decision, no policy, nobody to appeal to. It falls out of the arithmetic of what is easy to find.

The same shape, without a computer anywhere

Once you can see it, it turns up everywhere in financial life.

• The fee that is "standard" — because the only figures easy to find are published by the people

charging it.

• The product that "everyone recommends" — where the recommendations are the marketing budget.
• The comparison table whose ranking rules you cannot see, on a site paid per referral.
• The claim you cannot check because the underlying data is sold rather than published.

In each case the absence of accessible contrary evidence is being read as agreement. It isn't. It is an absence.

Where the rule came from

Out of this we adopted a rule and now apply it to everything we publish:

A claim is only rejected when a document was retrieved, read, and found to say otherwise.

Anything less is unconfirmed, never false. A 404, a paywall, "I couldn't find it," "I didn't check" — all of those mean the work has not been done yet.

And a second rule, because the first is not enough on its own: if you discount a source for having an interest in the answer, apply the same test to the industry's own material. Our check had discounted a legislative committee report as advocacy while accepting an insurance industry framing without comment. That asymmetry is the bias, not the skepticism.

The honest ledger

Thirty-three claims really were false, and the re-check found them. It also produced forty corrections to details in claims that were substantially right — a ratio quoted as a median when it was a mean, a combined ratio that belonged to a different six months, an estate threshold that had moved.

That matters more than the vindication. A check that only ever finds you were right is not a check. This one found errors in both directions, which is the only reason to believe either result.

Where Plenee fits

Every figure in this Academy names its document. Where we could not confirm something, the chapter says so — you will find "unconfirmed" in these pages, and that is deliberate. The alternative is filling a gap with something plausible, which is the failure this whole chapter describes.

The short version

We ran 376 of our own claims through an adversarial check and it rejected 207. On re-examination only about twenty rejections involved anyone reading a contradicting document; the rest were dead links, paywalls and unchecked assertions. We opened 388 documents and found 129 of the rejected claims were simply true. The error ran one way, and it always will: when the real source is hard to reach, what remains easy to find is whatever the industry publishes. So "I couldn't find evidence" is not a finding — it is a description of the search.