A big firm just got caught by the thing I worry about most

You will have seen the KPMG story this week. A report, parts of it AI generated, claims attributed to people who say they never said them, and the whole thing pulled after the fact. The profession reacted the way it tends to when it is one of the big names and not them, with a certain quiet satisfaction.

I watched it with more sympathy than that. Because there but for the grace of God go a great many firms, including some who are feeling rather smug this week.

But let me say the important part plainly. This was not an AI failure. The AI did exactly what AI does. It produced something that read beautifully, sounded authoritative, and was wrong. The failure was human, and it was a failure of process, not of technology.

And that is the thing I cannot get enough colleagues to take seriously. AI sounds exactly as confident when it is wrong as when it is right. There is no wobble in the voice, no "I am not sure about this one". So the mistake that gets through is never the obvious one. You catch the obvious one. It is the plausible one, the quote that sounds like something that person really would have said, the figure that sits neatly in the table, that sails past everybody precisely because nothing about it looks wrong.

So how does a firm full of clever, careful people end up publishing fiction under its own name?

Because of how nearly everyone has adopted this technology. They have bolted it on as a clever assistant. Someone asks the machine for a draft, the draft comes back polished, a tired human at the end of a long week gives it the lightest glance, and out it goes. The tool has quietly been promoted to author, and nobody actually decided to do that. It happened by drift.

And that is the whole difference between using AI as a copilot and building it into a workflow. A copilot sits beside you and you trust it as much or as little as you choose, which on a busy day means too much. A workflow is a different animal. In a proper one, nothing the machine produces reaches a client, a regulator or the public until a named person has checked the part that matters against something real. The checking is not left to whoever is least busy. It is built into the process so that it cannot be skipped.

At our firm we took the workflow road rather than the copilot road, and I will be honest, it was a great deal more work. You have to decide, for every output, what "checked" actually means and who owns it. But the part that surprised me was what it did for the people. Once the machine reliably does the dull, heavy lifting, your people are freed for the work that needs judgement, which is the more interesting work, not less. We did not lose anyone to AI. We moved them up to it.

Here is my prediction. In a year or two, the firms that come out ahead will not be the ones that adopted AI fastest. They will be the ones that were most disciplined about where a human still has to sign. The fast adopters who skipped that step will each have their own version of this week's story, and some of them will not get the chance to pull the report before a client has already acted on it.

Think of a polished AI draft as a candidate with an immaculate CV and a wonderful handshake. It might well be exactly who it claims to be. You still take up the references.

None of this is an argument against the technology. I am as committed to it as anyone in this profession. It is an argument for treating it as what it is, a brilliant, tireless assistant that is occasionally, and always confidently, wrong, and should never be the last set of eyes on anything that carries your name. The firms that learn that the easy way, by reading about someone else this week, are the lucky ones …

I would be curious whether others are seeing the same quiet drift, the machine promoted to author without anyone quite deciding to allow it. Because that, far more than the technology itself, is the thing that should keep us up at night.

Where this argument went next: The Partner Still Signs. The standard behind it is evidence-led AI accountancy, and the mechanics are the Trace-to-Source Standard.