Case study · AI search
AI Consistency Audit
When someone asks an AI about a firm, it doesn't call them. It reads the site, the directories, the bar listing, and a dozen aggregators — and those sources disagree with each other. The model picks one, and nobody knows which, or that the conflict existed at all.
Clients anonymised — aggregate figures from the live tracker
A firm's About page says founded 1976; LinkedIn says 1978. The homepage claims a $5.2 million verdict; the results page says $5.28 million. The contradictions are real, they're spread across hundreds of pages nobody opens side by side, and they're what AI is grounding its answers on.
This is a Claude Code project that finds them. It reports and sorts — it does not decide what's authoritative. Where the correct value can't be settled from public sources, it says so and hands the question to the specialist who owns that area.
01 — Why it can't be done by hand
The largest single site ran to 1,176 pages and 94,724 extracted claims. Reading that looking for two numbers that disagree is not realistic work for a person, and the contradictions that matter tend to live on the pages nobody would think to open — location pages, alternate-language pages, old practice-area pages.
02 — Scripts count, the model judges
What the scripts do
Roughly 1,300 lines of Python across a crawler, an extractor and a conflict differ. They fetch every page from the sitemap and internal links, respect robots.txt, and cache the HTML.
extract.py pulls structured claims — phone numbers and the
tel: href they link to, addresses, currency figures,
founding years, headcounts, schema blocks, and each page's publish date.
Every claim carries its source URL.
They are deliberately not allowed to decide what's a problem. They flag anything that differs and mark it unrated. That keeps them reproducible and incapable of inventing anything.
What the model does
Claude Code reads the candidate pile and does the part that needs judgment: the directory and AI-answer research the scripts can't do, the severity call, and the fix wording.
Is this a contradiction, or a blog post from 2019 correctly describing 2019? Is it a real conflict, or one of the known-good differences that must never be flagged? Which of these actually matters?
The rules live in a 268-line instruction file rather than in someone's head, so every firm gets identical treatment and a lesson learned once improves every future audit.
The same division shows up in the caching. Pages are read from the saved copy, not re-fetched — so a finding stays provable after the page changes, and a re-run can diff against last month. One re-run showed a questionable claim had spread from one page to fifteen. You can only see that if you still have last month's copy.
03 — Everything is reported, then sorted
Two live sources state incompatible values for the same present-tense fact. Both sides must have been fetched and read — if either is a search snippet or a blocked page, it is demoted, not reported.
Not false, but lagging. A hedge that reality overtook — "more than 60" when the figure is now 68. A directory holding an older number than the site. An evergreen page that was supposed to stay current.
A possible issue that couldn't be confirmed because the source was blocked or the only evidence was a snippet. Not a finding yet — it says exactly what to open and what would confirm or kill it.
Looks like a conflict, isn't. Rounding, formatting, name variants, per-platform review counts, and dated posts stating what was true when published. A 2025 post saying 68 employees is not an error today.
The line between drift and cleared is the one that took longest to get right: dated and clearly historical is cleared, an undated evergreen page that should have stayed current is drift.
04 — What one finding looks like
Founding year
About page — "serving the community since 1976"
Bar directory listing — "in practice since 1978"
- Why it matters
- Both are indexed and both get cited; the answer a model gives depends on which it reached for.
- Suggested fix
- Confirm the incorporation date, then set both to a fixed anchor — "serving clients since 1976" — rather than a year count that goes stale annually.
- Evidence
- Both sides fetched and read.
- Owner
- Website edit · directory update
Fixes are grouped by who owns them — website edits, directory updates, needs client access, needs manual verification first — and ordered by whether AI actually cites the source. A wrong value on a page the models quote matters more than a wrong value nothing reads.
05 — What the build taught
State coverage or the count is a lie
The first pass on two firms read only about a quarter of each site, and real contradictions were hiding in the rest. Both were redone. Every audit now opens with the percentage of the site actually read, because a finding count without it reads as a total when it is only a floor.
Write down what not to flag
A site carries the main number while directories carry per-location numbers — by design. Hedged phrasing is house style, not vagueness. Review counts differ per platform. Without those rules written down, the report fills with false positives and stops being read.
Silence is not a clean result
Every directory has to come back marked checked, blocked, or not found. A missing row looks identical to a clean one. Directory blocking is structural — the big aggregators refuse automated access — so the honest move is to name it rather than quietly leave a gap.