Verifiability and trust

1. Verifiability and trust

We’ve now seen how AI can augment every stage of analytics. The last thing to talk about is how to trust the work — and how to make sure other stakeholders can trust your work when you're using AI.

2. A near miss...

Here’s a recent exercise I did with Claude. Creating a simple dashboard showing the change in active workers and retired people in the USA. Great, right? It’s aesthetically pleasing and has a reassuring mix of KPIs and charts. But all is not as it seems.

3. A near miss...

The KPI in the top left is wrong: it chose the wrong year, and it miscalculated the value.

4. AI projects confidence

The problem isn't just that AI gets things wrong sometimes. The problem is that when it does, the output looks just as polished as when it gets things right. There's no visual signal saying "I might be wrong here." If you don't check, you ship the error. And it's your name on the deck or dashboard.

5. Verification isn't doing it twice

Our initial instinct might be to verify by redoing the work in a BI tool. Run the analysis again in Tableau. Recreate it in Excel. But that throws away the time AI just saved us. Verification doesn't have to mean duplication. Treat AI's output like work from a fast junior analyst — powerful, but not always trustworthy. Our job isn't to redo this work; it's to stress-test it.

6. The verification workflow

Here's a four-step framework called S.P.O.T. that catches most AI failures without redoing all the manual work.

7. The verification workflow

One, Sample and trace: Pick just one specific data point, filter the raw data manually to that data point, and verify that the AI’s calculation holds true for that single node.

8. The verification workflow

Two, Peer-review: Copy the AI’s generated code or output, paste it into a different AI Assistant, and ask it to aggressively find logic flaws.

9. The verification workflow

Three, Order of magnitude: Step back and do a business sanity check. Does this massive insight actually align with historical volumes and physical realities?

10. The verification workflow

Four, Test boundaries: Ask the assistant to run its own code and analytic ideas against edge cases—like null values, duplicates, or zeros—to see if there's any logic breaks hidden within.

11. Match the rigor to the stakes

For exploratory or ad-hoc work apply the framework more lightly. For a production dashboard or a board presentation, work closely through all steps— and triangulate against known benchmarks, an independent analysis, and your own domain knowledge. The point is to make the verification proportional to what's on the line.

12. Human-in-the-loop is the job

Don’t treat verification as the last step, you should be collaborating, challenging and verifying the AI agent throughout the analytical process. AI handles the speed, but you handle the judgement.

13. AI sycophancy

One last pattern worth flagging on the way out: AI sycophancy. AI assistants have a tendency to agree with you. Hint at an optimistic read and it'll find one. Push back on a correct answer and it may well abandon its answer. There are two places to fight it. At the prompt stage, keep your language neutral — "how has revenue trended this quarter?" beats "is revenue up this quarter?". At the verification stage, be hardest on findings that confirm what you wanted to believe. A finding that matches expectations doesn't trigger the same scrutiny as a surprising one. That's a trap.

14. Let's practice!

Ok, it's time to be the human in the loop!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.