The Riskiest Node Was Auto-Approval - Catching Silent Over-Trust with a Subtraction

When the upstream node wavered and the downstream node got more certain, an alert fires. That one subtraction pushes a 3.6 node to 4.9. Legal contract review and Korean credit underwriting, run through the same table, put the auto-approval node and the automated decision node at the top. Why this node is exposed in three directions at once, what arithmetic catches a failure that raises no error, and why being colored red and escalating to a human are two different events.

Part 2 covered the 36-cell table and the structure that lets the handoff score rise during a run. This part is what came out when it was run against actual workflows.

The two top scores landed in the same seat

Legal contract review and credit underwriting differ in domain, regulation, and node composition. The top node in each had the same character.

The three values match to the decimal. Not a coincidence: both are high for the same reasons.

General failure is high because the outcome does not come back. A clause extraction node missing a clause still has a chance of being caught downstream; a wrong auto-approval means the contract is signed and the loan is disbursed. At the same error rate, different recoverability should score differently.

Security is high because of permission scope. This node holds the authority to execute an externally consequential action without human confirmation. That is the shape the excessive agency item in the OWASP LLM Top 10 points at, and across a workflow usually only one or two nodes meet the condition.

Handoff is high because of how the node is passed through. The upstream threshold decision becomes a silent gate, and its basis is not carried downstream. If the audit log holds only an “auto-approved” stamp, there is nothing to trace back.

Placed next to a low node

A top score only means something next to a low one. The legal intake and classify node scores 3.4, 3.0, 3.4, more than a point below on every axis.

That node malfunctions too. Misclassify a contract type and the next stage uses the wrong playbook. The score is low for three reasons: the error can still surface downstream, permissions stop at reading and tagging, and a wrong classification shows up in the output. Recoverable, narrow, not silent.

Those three columns are the scoring criteria. Measure risk by error rate alone and the auto-approval node looks safer: simple rules, structured inputs, possibly low malfunction frequency. Measure by recoverability, permission breadth, and visibility instead and the ordering inverts.

Catching silent over-trust with a subtraction

What makes these nodes awkward is not that the outcome is bad but that nothing signals while the failure happens. So “silent” had to become a computable value.

The method is a subtraction: the downstream node’s confidence minus the upstream node’s.

decay = downstream_conf - upstream_conf
over_trust = max(0.0, decay) if upstream_conf < 0.8 else 0.0
if over_trust > 0.2:
    band, alert = "over_trust_alert", True

Count that difference only when the upstream was below 0.8, and alert when it exceeds 0.2. The upstream wavered and the downstream got more certain anyway, which is the shape of a failure that proceeds without error. The other direction is watched too: if the upstream was above 0.9 and the downstream does not carry that through, an under-use warning fires.

Confidence itself is deliberately not one signal. It is a weighted sum: 0.3 from a token log-probability proxy, 0.3 from the model’s self-report, 0.2 from whether the tool call exited cleanly with structured output, 0.2 from how close it came to timing out. Lean on log-probability alone and the whole metric wobbles when that one does.

Two more metrics attach here. The intent preservation score embeds the upstream output and downstream diagnosis and takes cosine similarity, alerting below 0.5. The judge metric has a separate language model score the alignment of both outputs, signaling human review below 0.6; its default backend is a mock, so a real judgment requires naming one.

The real incidents have this shape

Two of the verified 2026 incidents show it exactly.

In the AWS coding agent case the agent did not throw an error; it executed what it had determined. There was no point at which a human could intervene. Amazon disputed the framing, calling it misconfigured access controls rather than AI, and the remediation it introduced was senior-engineer sign-off on AI-assisted changes. That the fix was adding a gate says the cause was permission scope.

The second is harder. At Amazon retail an agent surfacing internal guidance pulled stale references, the person who received the advice acted on it, and four high-severity incidents hit in one week in March, one lasting six hours. A human was in the loop here. The human was there and it did not stop.

---
config:
  look: handDrawn
  theme: neutral
---
flowchart LR
    A["Upstream node<br/>confidence 0.55"] -->|"no error"| B["Downstream node<br/>confidence 0.85"]
    B -->|"over-trust gap 0.30"| C["Alert · handoff +0.5"]
    C --> D["Auto-approval gate"]
    D -->|"stamp only"| E["Audit log"]

The same happens with a human at the end of the chain. The human treats the preceding judgments as the evidence and does not check the evidence behind them.

Two legal precedents attach here. One is Moffatt v. Air Canada, decided by the BC Civil Resolution Tribunal in February 2024. The fare policy stated by the chatbot differed from the actual one, and the airline’s argument that the chatbot was a separate entity was not accepted.

The more recent one is the Higher Regional Court of Hamm’s ruling of 12 May 2026: even where a company proves it supplied correct data to its chatbot, responsibility for the output still attaches to the company. A defense of “our inputs were right” does not hold, which goes a step further than the earlier case. Responsibility for an automatically issued judgment is not reduced by the fact that it was automated, nor by the fact that the inputs were correct. That leaves whether the basis for the judgment can be explained afterward, and a stamp-only audit record does not explain it.

The attacker-free cases are the majority

These cases share one more thing: no attacker.

Five of the six 2026 incidents collected had none, and 188 of the 344 verified enterprise cases in the Cyera dataset had none either. The harm came from an agent doing what it was told, faster than a human could intervene, with permissions broad enough to make the action irreversible.

That observation shook the table once. Excessive permissions sit on the security axis because OWASP and MITRE ATLAS put them there, and both presume an adversary. If attacker-free incidents are the majority, those cells’ mitigations can get mis-scoped toward adversarial controls. Rather than moving the axis, the affected cells now carry a no-attacker marker and their mitigations are aligned toward permission scope and gates.

The arithmetic that pushes a 3.6 node to 4.9

Running the credit underwriting workflow gives different results depending on execution conditions, and that difference is the most instructive part of this table.

Run it in an environment without the embedding model and the runtime metrics are not computed. Only the ontology lookup remains, and the final score is the weighted sum of axis means.

Maximum 4.76, three RED.

Turn the metrics on and it changes. If the credit scoring node raises an intent preservation alert and an over-trust alert and its judge score falls below 0.6, the boost is 0.5 plus 0.5 plus 0.3, or 1.3. The handoff axis moves from 3.6 to 4.9 and the final score becomes:

handoff = min(5.0, 3.6 + 1.3) = 4.9
final   = 0.4 × 4.9 + 0.3 × 3.4 + 0.3 × 3.6 = 4.06   → RED

A node whose three raw axis scores are all in the threes crosses into RED. By the ontology alone it is an ordinary node; on this run the joins around it actually diverged. The automated decision node also gains a boost, and once its handoff hits the 5.0 cap the final rises to 4.88.

So the same workflow comes out at 4.76 with three RED in degraded mode and 4.88 with four RED in full mode. The table knows node types; only the metrics know what happened on this particular run.

The arithmetic above substitutes a combination straight from the boost table. Which combination produced any given on-screen value requires reading the logs; the screen alone does not let you reconstruct it. Unless the diagnosis records which metric added how much, that score cannot be explained later, which is a criticism that lands on this tool itself.

Being red and reaching a human are different

Designing a gate usually starts with “who approves.” What these cases show is that the problem is not the approver but when it escalates.

So the gate condition became a function of computed results rather than an operator’s judgment. There are four conditions, and any one of them sends it to a human.

Note that the color band and the gate threshold differ. A node is painted red at 4.0; it reaches a human at 4.5. In the arithmetic above, the credit scoring node became RED at 4.06 but does not trip the gate. The heatmap says risky and no approval screen is created. The automated decision node trips it at 4.88.

That gap is needed in practice. If color equals escalation, every node between yellow and red goes to a human, approval screens become common, and nobody reads them. Separating what is shown from what is stopped is what keeps a gate a gate.

And score is not the only opener. Even with a maximum final below 4.5, an over-trust gap above 0.2 on any node pair sends it to a human on its own. That condition exists to catch the case where each node is fine and the joins are not.

Fewer than four organizations in ten have a gate

Whether this design is needed at all is answerable from field data. The Cloud Security Alliance and Token Security surveyed 418 organizations in April 2026 and found that 38% require human approval when an agent steps outside its permission scope. Only 11% automatically block unauthorized actions.

Lay that next to the incidents above and it fits. The remediation in the AWS case was adding an approval step, and more than half the verified incidents had no attacker. What was needed was not defense but a gate, and six organizations in ten do not have one. Those that do still have to decide where to put it, which is the question this table answers.

What the approver actually holds

When the gate fires, an approval screen is created carrying the workflow name, the maximum risk score, the count of nodes over threshold, the reason it escalated, and a one-page summary. The reason string names the condition verbatim, in the form “over-trust gap 0.30 exceeds threshold 0.2.” Leave the approver to re-read the entire heatmap and the gate becomes a formality.

There are three options. Approving as-is resumes the process and records the approval alongside the diagnosis at that moment. Conditional approval attaches the must-fix mitigations as a condition. Rejection sends the design back for revision.

Conditional approval will be the most used in practice. With a deployment date set, rejection is costly and approving as-is defeats the purpose. So what goes into the must-fix tier determines whether the gate has force. A long must-fix tier makes conditional approval effectively a rejection; an empty one makes the gate a formality. That is why the auto-approval node’s handoff cell keeps exactly one must-fix item: permanently retaining the reasoning trace behind each auto-approval.

Reviewer atrophy is still unmeasured

Reviewer rotation, in the optional tier, has the weakest evidence. The hypothesis is that as the auto-approval ratio rises, human reviewers handle real cases less often and their capability degrades. No public precedent was found.

Measurement is hard because it does not register as an incident. It surfaces only as the outcome that a human failed to catch something, and the cause is then recorded as individual reviewer error. Measuring the relation would require sampling auto-approved cases, routing them back to humans, and comparing accuracy. No organization with such a procedure was found in the material.

Weak evidence keeps it out of the must-fix tier. It was not deleted either. Deleting it means rediscovering it next cycle; leaving it in the optional tier means only promoting it when evidence arrives.

What a pre-deployment check cannot see

This table reads structure at design time: what permissions a node holds, where the gates are, whether outcomes are recoverable. What surfaces only during execution, it does not see.

The trajectory-level, span-level, and claim-level failure localization covered in the Locating Agent Failure series all operate on records accumulated after a run. Finding which call first went wrong requires that the call happened.

The two do not substitute for each other. That a design has no gate is readable without any execution record; that a gate exists and humans wave things through is readable only from execution records. The AWS case is caught by the former, the Amazon retail case by the latter. Putting runtime metrics inside the diagnosis pushes that boundary a little: a pre-deployment check borrowing some execution signal.

Closing - the highest-value seat carries the highest risk

The top scores landing on the same node in both workflows is not an artifact of the table’s design. The three axes were scored on different bases and met at the same node.

That node is also where automation is worth the most. Remove one stage of human review and throughput rises while cost falls, so automation projects target exactly it. For the same reason it produces irreversible outcomes, holds broad permissions, and passes through without leaving a basis.

The seat you want to automate and the seat that needs a gate are the same seat, and that is the structure of this problem. Leave “should we add a gate” as a discussion and it tilts toward no. Have arithmetic fire it at a fixed threshold and the discussion disappears.

What remains is where to put that gate. Inside the diagnosis code as a conditional and the gate is code; inside the process definition and the gate is a picture. The difference is larger than it looks.

Sources

• AI Incident Database, incident 1442 (AWS Kiro), incident 639 (Moffatt v. Air Canada)
• Fortune, 12 March 2026 reporting on four high-severity incidents involving an internal Amazon retail agent
• Cloud Security Alliance and Token Security, agent governance survey (April 2026, n=418)
• Cyera, Agent-Inflicted Damage: Inside the Real-World Failures of Enterprise AI Systems
• OWASP, Top 10 for LLM Applications 2025, excessive agency item
• Moffatt v. Air Canada, 2024 BCCRT 149; OLG Hamm, 12 May 2026
• Scoring in scripts/agents/aggregator.py, gate conditions in scripts/core/tools.py (evaluate_hitl), thresholds in scripts/core/contracts.py (public repository)