Inside My First Digital Employee: The Architecture of the AI Auditor

In my previous article I argued that the first AI employee a bank hires should be an internal auditor. The most common reply was: show the build. My first attempt at that answer stayed at the block-diagram level, and a fair critique came back: too shallow. So this is the version I would hand a new engineer on my team, written like a user guide. Executives can read the two scenario walkthroughs and the honesty section at the end. Builders, everything in between is for you.

The case that motivates everything

In January 2023, the Justice Department announced a settlement of over $31 million with City National Bank, the largest redlining settlement in the Department's history. The allegation: the bank avoided providing mortgage lending services to majority-Black and Hispanic neighborhoods in Los Angeles County. Two facts from the DOJ's release stay with me. Other banks received more than six times as many mortgage applications from those neighborhoods as City National did. And across twenty years of opening and acquiring branches, the bank opened exactly one in a majority-Black and Hispanic neighborhood.

Here is what makes this case the right test for an AI auditor: nothing about that pattern was hidden. The six-to-one application gap was a ratio computable from data the bank already held, plus public HMDA data, and it sat there for years. Nobody was assigned to compute it every week, and no system forced anyone to look. That is the failure mode this architecture is built to end. Keep the question in mind as we go: what would it take for this pattern to surface in the system's first quarter of operation, with a case file attached, instead of surfacing in a federal complaint?

The contract everything hangs on: the decision event

The whole system consumes one input: a decision event, emitted for every automated decision the institution makes at onboarding or origination. If a platform cannot emit this event, that is finding zero, and it is exactly what my pre-contract evidence readiness assessment exists to catch before you sign. The shape, trimmed for print:

{
  "decision_id": "d-20260815-000482910",
  "journey": "mortgage_prequal",
  "timestamp": "2026-08-15T14:02:11Z",
  "channel": "mobile",
  "applicant_ref": "tok_9f2c...",            // tokenized, PII stays in the vault
  "geo": { "census_tract": "06037-2411.02" },
  "checks": [
    { "name": "identity_verification", "vendor_ref": "idv-primary",
      "signals": { "trust_score": 642 }, "latency_ms": 240 },
    { "name": "credit_screen", "signals": { "band": "B" } }
  ],
  "policy_version": "onb-pol-14.2",
  "thresholds_applied": { "auto_continue": 630, "review_band": [500, 630] },
  "outcome": "approved",
  "decided_by": "rules_engine_v8"
}

Three design choices matter here. The applicant is a token, so the audit plane never holds raw PII. The policy version travels inside the event, so every decision permanently knows which rules were supposed to govern it. And vendor signals are captured as returned, which is what makes findings citable later.

Layer one: policy replay, and the fat-finger scenario

Policy replay is the deterministic heart of the audit plane. The institution's onboarding policy lives as versioned, executable rules, and every production decision is re-evaluated against the version it claims to have used:

def replay(event):
    policy   = policy_store.get(event.policy_version)   # immutable, versioned
    expected = policy.evaluate(event.checks)            # deterministic re-run

    if expected.outcome != event.outcome:
        return Flag("POLICY_MISMATCH", severity="high",
                    evidence=[event.decision_id, expected.trace])

    for check in policy.required_checks(event.journey):
        if check not in event.checks:
            return Flag("MISSING_REQUIRED_CHECK", severity="high",
                        evidence=[event.decision_id, check])

    if event.thresholds_applied != policy.thresholds(event.journey):
        return Flag("THRESHOLD_DRIFT", severity="critical",
                    evidence=[event.thresholds_applied, policy.thresholds])

Scenario A: the misconfigured threshold

A config push on a Tuesday evening is supposed to set the auto-continue threshold to 630. Someone types 530. Every applicant scoring between 530 and 629 now sails through a gate that policy says should route them to review.

In the sampling world, this lives until the next quarterly file review, and only surfaces then if one of the two dozen sampled files happens to fall in the gap. In this system, the very first decision after the push carries thresholds_applied of 530 while policy version onb-pol-14.2 says 630. Replay raises THRESHOLD_DRIFT on that first decision. The flag lands in the queue with both values quoted, operations rolls the config back within the hour, and the case file lists exactly which applicants were decided during the gap, because every one of them raised the same flag. Remediation scope is a query, not an archaeology project.

Layer two: population monitors, and the City National scenario

Replay catches broken rules. The second layer catches broken patterns, and it is where the redlining case lives. Monitors are declarative configurations, versioned like policy:

monitor geo_application_flow:
  population: journeys in [mortgage_prequal, heloc, deposit_opening]
  window:     rolling_quarter
  group_by:   census_tract majority_demographic      # public census mapping
  metric:     applications_per_1k_owner_households
  baseline:   peer_institutions_from_public_HMDA
  trigger:    ratio_vs_baseline < 0.5 for 2 consecutive periods
  route_to:   fair_lending_review_queue

monitor geo_outcome_disparity:
  metric:     denial_rate adjusted_for [credit_band, income_band, product]
  group_by:   census_tract majority_demographic
  trigger:    adjusted_ratio > 1.5 sustained 2 periods
  route_to:   fair_lending_review_queue

Scenario B: the pattern behind the $31 million settlement

Run the DOJ's own numbers through the first monitor. Peer banks were receiving more than six times the applications from majority-Black and Hispanic tracts, which means the institution's ratio against the public baseline was sitting around 0.16, far below the 0.5 trigger. This monitor does not need years. It fires in its second evaluation period, which is to say in the system's first two quarters of operation, and it fires every period after that until someone deals with it.

Be precise about what happens next, because this is where the design earns its keep. The system does not conclude discrimination. It cannot, and it must not; disparate treatment and disparate impact are legal judgments that belong to the fair lending officer and counsel. What the system does is three things the manual world reliably fails to do: it computes the ratio every period instead of annually or never, it opens a case with the evidence attached, and its queue rules refuse to let the case be silently ignored, because an untouched high-severity finding escalates on a clock. The City National pattern was never invisible. It was uncomputed and unowned. This layer makes it computed and owned.

Layer three: the investigation agent

When a monitor or replay raises a flag, the LangGraph triage router opens a case and hands it to the investigation agent, which runs on the Claude Agent SDK. The graph node, trimmed:

def investigate(state: CaseState) -> CaseState:
    case = state.flag
    finding = agent.run(
        system    = SYSTEM_PROMPT,                     # versioned, XML-structured
        skills    = load_skill_for(case.type),         # e.g. fair-lending-invest-v3
        tools     = [policy_corpus, decision_store, analytics],   # read-only MCP
        subagents = { "policy":  policy_retrieval,
                      "history": signal_history,
                      "cohort":  population_comparison },         # parallel fan-out
        output    = FINDING_SCHEMA_V3 )

    if not validate(finding, FINDING_SCHEMA_V3):       # schema gate at the boundary
        finding = retry_once() or route_to_human_raw(case)

    state.queue.push(finding)
    return state                                       # graph, not agent, moves on

The system prompt is structured, versioned, and boring on purpose. The skeleton:

<role>You are a junior audit investigator. You never decide.
You assemble evidence for a human who does.</role>

<constraints>
- Cite policy only from policy_corpus tool results, by section id.
- Every claim carries an evidence_ref. No ref, no claim.
- If the evidence is insufficient, say so. Never fill a gap.
</constraints>

<output_contract>FINDING_SCHEMA_V3</output_contract>
<escalation>severity high or above: tag the fair lending officer</escalation>

A real tool exchange from the Scenario B case type, so you can see why retrieval beats recall. The agent does not remember the fair lending policy. It asks for it:

call:    policy_corpus.search({ "query": "geographic monitoring of application
                                          flow and denial rates", "top_k": 3 })
returns: [{ "section": "FLP-4.2",
            "quote": "The bank monitors application flow and outcome rates
                      by geography each quarter and investigates sustained
                      deviations from peer baselines.",
            "source": "Fair Lending Policy v7, approved 2026-03-02" }]

An agent that recalls policy from training will eventually hallucinate policy. An agent that retrieves and quotes it can be checked in one click, and the check is part of the finding. Which brings us to the output. The agent's work product is not prose; it is a schema-validated object:

{
  "finding_id": "F-2026-0815-114",
  "flag_type": "geo_application_flow",
  "severity": "high",
  "summary": "Application flow from majority-Black and Hispanic tracts at
              0.19x peer baseline for two consecutive quarters, breaching
              the FLP-4.2 trigger of 0.5.",
  "policy_citations": [ { "section": "FLP-4.2", "quote": "..." } ],
  "evidence_refs": [ "monitor-run-2026Q2-geo-flow", "hmda-baseline-2026Q1" ],
  "recommended_disposition": "Route to Fair Lending Officer. Assess branch
              coverage, marketing spend, and loan officer assignment by tract.",
  "agent_version": { "prompt": "v14", "model": "pinned-2026-06",
                     "skills": ["fair-lending-invest-v3"] }
}

A finding that fails schema validation never reaches a human. Structured output is the cheapest hallucination control you will ever deploy: the model cannot cite a section that does not parse, and it cannot skip the evidence_refs field.

Hooks: the guardrails that do not negotiate

@pre_tool_use
def enforce_read_only(call):
    if call.tool not in READ_ONLY_ALLOWLIST:
        halt_run(reason=call)                    # hard stop, not a polite refusal

@post_tool_use
def capture_evidence(call, result):
    evidence_store.append(run_id, call, result)  # append-only, addressable

Two properties fall out of this. The read-only boundary is enforced twice, once here and once in the MCP servers' own credentials, so a cleverly worded case cannot talk the system past it. And the agent's investigation trail is captured as a side effect of running, not as an act of discipline. The agent's memory is a log, and the log is examiner-ready by construction.

The human queue: where autonomy stops

The reviewer opens a case and sees the finding, the quoted policy sections, and drill-down to every referenced decision event. They have three buttons: confirm, reject, escalate. The agent has none of them; it never closes its own findings. One more rule with teeth: a high-severity finding untouched for its SLA window escalates automatically up the review chain. That rule is aimed directly at the City National failure mode, where an uncomfortable number can sit unexamined because no process forces the examination.

Evals: the auditor is audited

golden = load_cases("adjudicated/")     # findings humans confirmed or rejected,
                                        # plus one synthetic redlining quarter

for change in [prompt_edit, skill_edit, model_upgrade]:
    results = shadow_run(golden, candidate=change)
    require(results.precision >= 0.95)
    require(results.recall    >= baseline.recall)
    require(results.catches("synthetic_redlining_2026Q_X"))   # non-negotiable

The golden set includes a synthetic quarter that reproduces the City National ratio. No prompt change, skill change, or model upgrade ships unless the candidate still catches it. Model versions are pinned and new versions shadow-run before promotion. This is model risk management in the working spirit of the Federal Reserve's SR 11-7: versioned, validated, monitored, and challengeable.

What this system does not do

• It does not decide that discrimination occurred. Disparate impact and disparate treatment are legal conclusions that belong to humans; the system computes disparities and forces a documented human decision.
• It does not replace a fair lending compliance program, HMDA reporting, or counsel. It compresses the time between a pattern existing and a qualified human looking at it, from years to weeks.
• The monitor definitions above are illustrative, not a certified fair lending statistical methodology. Thresholds, adjustment variables, and peer baselines get set with the compliance team, in policy, before the first run.
• It does not touch production. Read-only in the credentials, read-only in the hooks, read-only in the architecture.

Where this leaves you

The $31 million question is not whether your institution has a pattern like City National's. It is whether anything in your building computes the ratio that would reveal it, every period, with a queue that refuses to look away. That is a solvable engineering problem, and this article is the solution I build: an evidence contract, a deterministic replay layer, declarative monitors, an investigation agent that cites its sources, and guardrails that do not negotiate.

I design and build these systems for banks and credit unions. If you want the evidence readiness assessment before your next vendor contract, or this audit plane under your existing stack, my messages are open.