The Audit Trail That Audits Itself

In my last piece, I discussed that when regulators catch something in your onboarding stack, the fine lands on you, not on your vendors, and that your only real defense is evidence: applicant by applicant, what was checked, what came back, and why the decision was made.

The sharpest response I heard back was a question: fine, I have the audit trail. Who reads it?

That question deserves an uncomfortable answer. Nobody reads it. Not until the examiner does. Most audit trails in banking are write-only memory: expensively collected, carefully stored, and unread until the one reader you fear most opens them first.

Sampling is the flaw

Here is the mechanic nobody says out loud. When an internal audit or compliance team reviews onboarding decisions, it samples. A few dozen files a quarter, pulled from tens of thousands of decisions, checked by hand against policy. Sampling is not a flaw in your team. It is a concession to human reading capacity, and for decades it was the only option.

The cost of that concession is time. An irregularity that starts today, a misconfigured threshold, a vendor that quietly changed an API response, a fraud pattern that adapted to your rules, lives in the unread majority of your data until a sample happens to catch it or the damage surfaces on its own.

Remember Metropolitan Commercial Bank. The surge in fraudulent prepaid account openings was visible in the data while it was happening. More than $300 million in pandemic benefits moved through those accounts before the scheme was shut down, and the bank paid roughly $30 million in penalties. The evidence existed. The reading capacity did not.

Software does not need the concession. A machine can read every decision, every day, against every rule. Which means sampling is no longer a constraint you have to accept. It is a choice you are making.

The two moments you control

There are two moments where you can take control of this problem: before you sign the vendor contract, and after the system goes live. I work on both, and they are different disciplines.

Before you sign: the evidence readiness assessment

Most vendor diligence asks what a tool does. Almost none asks what the tool will let you prove later. Before an institution signs an onboarding vendor, I run five tests:

• Per-applicant reconstruction. Can we rebuild any single decision end to end: inputs, signals returned, thresholds applied, outcome? If the answer involves the word "ticket," that is a no.
• Signal downloadability. Are the raw signals available as structured data you can store and query, or as screenshots and PDFs?
• API versioning discipline. When the vendor changes what a field means, how do you find out: a versioned changelog, or a production incident?
• Threshold governance. Are risk thresholds configuration you own and can point to in your policy, or vendor defaults you inherited without ever deciding?
• Failure-mode logging. When a check times out or errors, what gets recorded, and what did the customer journey do in that moment?

The output is a scored evidence map: what you will be able to prove after go-live, what you will not, and the contract language to demand before signature. It is far cheaper to negotiate observability before signing than to retrofit it after.

After go-live: the auto-audit layer

Once the stack is live, I build what I call an auto-audit layer underneath the onboarding flow. Every decision event streams into it, and three layers read every one of them at the moment it happens.

Layer one is policy as code. The institution's onboarding policy, its thresholds, required checks, and allowed outcomes, lives as executable rules, and every production decision is replayed against them. A decision made with a threshold that does not match policy, or with a required signal missing, gets flagged at the first occurrence. This layer is deterministic. It catches misconfigurations and silent vendor-side changes the moment they first touch an applicant, not at the next quarterly sample.

Layer two is statistical. Models monitor drift and anomalies across the whole population: score distributions shifting, approval rates moving by channel or segment, override rates creeping, volume spiking at odd hours. Layer one catches broken rules; this layer catches broken patterns. It is the layer that would have seen an MCB-style surge on day one instead of month six.

Layer three is narrative, and it is where large language models earn their place. Every flag gets a plain-English explanation: what fired, which policy it relates to, what changed, and what a reviewer should look at first. The same layer assembles examiner-ready evidence packs on demand: pick any applicant, any date range, or any flag, and get the complete reconstructed story with the underlying data attached.

On top sits a dashboard: live population coverage, flags at first occurrence, and drill-down to any single applicant's signals. And one rule I do not compromise on: every flag lands in a human review queue, and the AI never closes its own findings. Detection is automated. Judgment is not.

The auditor cannot be a black box either

There is an obvious trap in this design: replacing an unexplainable vendor stack with an unexplainable AI auditor. So the auto-audit layer is held to the same standard it enforces, consistent with the spirit of the Federal Reserve's model risk management guidance, SR 11-7: versioned models and prompts, logged reasoning behind every flag, outputs a human can challenge, and periodic validation against known cases. If your AI auditor cannot explain itself to an examiner, it is not an audit system. It is a liability with a dashboard.

Where this leaves you

If you are evaluating an onboarding vendor right now, the evidence readiness assessment happens before you sign, and it changes what you sign. If your stack is already live, the auto-audit layer is built underneath it without replacing anything; it reads your decisions, it does not make them.

I do both. If either sounds like a problem you are carrying, my messages are open.

And for everyone else, one question I would like answered in the comments: how does your institution review its onboarding decisions today, sampling or population? And if it is sampling, what share of your decisions has any human ever looked at twice?