The Axis No Public Dataset Had Got the Largest Weight - A 36-Cell Risk Ontology

Two of the three axes were ported from public frameworks; one was built because nothing existed. The one that was built takes the largest share of the final score. How 36 cells scoring 12 nodes on 3 axes were assembled, why the axis that looks between nodes got a weight of 0.4, and why that axis alone can gain points during a run.

Part 1 listed “incident data has never been organized by node” as one reason pre-deployment workflow review lives only inside consulting. This is the record of that re-anchoring work.

Three axes, with different shares

When you look at one workflow node and call it risky, that risk is three different kinds mixed together.

The first is the node failing at its own job: a clause extraction node missing a clause, a risk-flagging node whose threshold drifts. The second is attack surface: prompt injection, retrieval poisoning, excessive permissions. The third is what breaks at the joins even when every node does its own job.

The first two have plenty of public material to anchor to.

The right-hand column is the project’s judgment. The largest share went to the axis with nothing to consult.

There is a reason. The OWASP LLM Top 10 is written for a single application, MITRE ATLAS catalogues attack techniques aimed at a model, and NIST AI RMF covers organizational management processes. All three presume one system as a unit. The point where node A’s output diverges from node B’s input assumptions appears nowhere. Yet in real incidents the damage tends to happen at exactly those joins. Every node behaved normally, and the “verified” signal handed forward by one node was read with a different meaning by the next.

Why no standard formed there is guessable. Transitions between nodes are a design choice, not a standard. The same task gets cut and joined differently by different designs, so a common list is hard to extract. Hard to standardize and small in risk are different claims.

Re-anchoring someone else’s table is its own work

The missing handoff axis was the visible problem, but the first two could not be copied over either. Every public framework is sorted on a different unit.

The OWASP LLM Top 10 is organized by vulnerability class, with no statement of which workflow node each applies to. MITRE ATLAS is organized by attack technique, grouped into tactical stages. The MIT AI Risk Repository classifies mitigations by causal domain.

Anchoring all three onto node types means deciding, for each item, which kind of node actually manifests it. Excessive agency formally applies to most nodes, but the nodes producing an irreversible outcome are two: auto-approval and automated decision. That is why only those two carry a security score of 4.8 while the rest stay in the threes. Porting a table is not copying items; it is deciding where each one attaches, and that decision is recorded as a sentence in every cell.

Some things were deliberately left out. Items a workflow designer cannot act on at design time, such as training data poisoning or model extraction, were not placed into cells. A pre-deployment check whose recommendation ends at “change your model vendor” is not usable by the recipient.

What the 36 cells look like

Two sample workflows: legal contract review and Korean credit underwriting. No customer material; both synthetic. Six nodes from each gives 12 nodes, scored on 3 axes for 36 cells.

The choice was deliberate. Legal contract review documents the basis for a judgment and still has a human reviewer at the end. Credit underwriting has regulation attaching differently at each node and fully automated decisions that actually exist. One keeps the human, the other aims to remove them, which shows both ends of the handoff axis.

Taking only six nodes from each was deliberate too. Scoring everything raises the cell count while most of them cluster in the threes and the table just gets heavier.

Counting cells at 4.0 or above gives 17 RED and 19 YELLOW. Counting RED nodes gives six, which would suggest 18 cells, but the actual count is 17: legal N3’s security axis falls at 3.5, below the band. A risk-flagging node carries high malfunction risk with a comparatively narrow attack surface, and its mitigations should differ accordingly. That mismatch was left in place rather than smoothed over.

An axis score is the mean risk score of the cells attached to that node. In code:

AXIS_WEIGHTS = {'handoff': 0.4, 'security': 0.3, 'general_failure': 0.3}
COLOR_THRESHOLDS = {'RED': 4.0, 'YELLOW': 2.5}   # below that, GREEN
SCORE_CAP = 5.0

final = (0.4 * handoff_with_boost
       + 0.3 * base_security
       + 0.3 * base_general)

Node color has three bands: RED at 4.0 and above, YELLOW at 2.5 and above, GREEN below. And there is a term named handoff_with_boost, because of the three axes only handoff can gain value during a run. More on that below.

Opening one cell shows more than a score. The handoff cell for the auto-approval node:

- cell_id: N5a_handoff
  node: N5a_auto_approve
  axis: handoff
  upstream_dependency: N4_compare_playbook
  downstream_dependents: [N7_signoff, N8_audit]
  primary_handoff_risk: HITL_bypass
  secondary_handoff_risks:
    - no_appeal_mechanism
    - reviewer_atrophy
  risk_level: CRITICAL
  risk_score: 4.7
  owasp_asi_refs: ["ASI09", "ASI08"]
  incident_evidence_refs: ["pocketos-production-db-deletion"]
  mitigation_options:
    must_fix:
      action: "Permanent retention of the reasoning trace behind each auto-approval"
    recommend:
      action: "Appeal channel with a 30-day review window"
    optional:
      action: "Reviewer rotation once the auto-approve ratio crosses a threshold"

Mitigations come in three tiers: must fix, recommend, optional. When a consulting deliverable lists recommendations flat, the recipient has to re-decide what to touch first. Tiering puts that decision into the table.

The bug where the input pre-labeled its own answers

The sample workflow documents carried an “expected diagnosis” column. They were built by hand, so which nodes should come out RED was written in advance. The early implementation read that column to pick which nodes to score.

The problem is that the input then labels its own answers. If the sample says “N5a is RED,” the tool returns N5a as RED and that looks like the tool’s verdict. Now candidate nodes are those the ontology has cells for, and color comes only from the computed score. The engine carries the note:

Which nodes are RED/AMBER is then decided by the ontology SCORE (below) - NOT read from the input.

This class of leakage is the hardest thing to catch when building a scoring tool, because the results look plausible.

The handoff axis was filled with six patterns and three metrics

There was one way to fill a missing axis: write down what breaks at the joins, by type, and attach external evidence and an observable signal to each.

Each pattern carries its external reference: AgentBench’s multi-agent error traces reporting an 18% misalignment rate, the PoisonedRAG work, WebArena’s recursion failure cases.

Three of the six were later corroborated externally. The OWASP GenAI Security Project published its Top 10 for Agentic Applications on 9 December 2025, peer-reviewed by more than 100 experts, and ASI07, ASI08, and ASI09 map near-verbatim onto handoff misalignment, cascading failure, and trust collapse.

Those three have one more property. They have no counterpart in the LLM Top 10. They cannot appear on a list that looks at one agent; they only appear when you look between nodes. An axis built from in-house heuristics reappeared in the same place in an external standard, and the fact that exactly those three have no LLM-list counterpart supports keeping it as a separate axis.

Patterns are prose for humans. Prose catches nothing at runtime. So three metrics were attached, each with a formula.

Intent Preservation Score (IPS) embeds the upstream output text and the downstream diagnosis text and takes the cosine similarity. At 0.7 and above it is healthy, 0.5 to 0.7 is a watch band, below 0.5 is a context decay alert. Embeddings use BGE-M3, falling back to a hash-based function where the model is unavailable.

Confidence decay is one subtraction. Take the downstream node’s confidence minus the upstream node’s. Count that difference only when the upstream was below 0.8, and alert when it exceeds 0.2. The upstream wavered and the downstream got more certain anyway. The other direction is watched too: if the upstream was above 0.9 and the downstream does not carry that through, an under-use warning fires.

Confidence itself is not a single signal. It is a weighted sum: 0.3 from a token log-probability proxy, 0.3 from asking the model to self-report, 0.2 from whether the tool call exited cleanly with structured output, 0.2 from how close it came to timing out. Lean on one and the whole metric wobbles when that one does.

The judge metric (LaaJ) hands a separate language model the outputs of both nodes and takes back an alignment score. Below 0.6 means human review, 0.8 and above means trusted. The default backend is a mock, so running a real judgment requires naming one. Calling a judge on every handoff costs money, and the design document recommends 10% spot sampling.

The score can rise mid-run

This is the most unusual part of the table. Of the three axes, only handoff gains points from what actually happened during the run.

The boost is added to the handoff base and clipped at 5.0, then carries the 0.4 weight into the final score. The ontology sets a floor and the run pushes above it.

---
config:
  look: handDrawn
  theme: neutral
---
flowchart LR
    C1["Cell mean<br/>general failure"] --> W["Weighted sum"]
    C2["Cell mean<br/>security"] --> W
    C3["Cell mean<br/>handoff"] --> B["Runtime boost<br/>capped at 5.0"]
    R["Runtime metrics<br/>IPS · decay · judge"] --> B
    B --> W
    W --> S["Final score"]
    S --> COL["Color band<br/>4.0 / 2.5"]
    S --> G["Gate check<br/>4.5 or a metric alert"]

This is needed because the ontology only knows node types. “An auto-approval node carries high handoff risk” is something a table can say; whether the upstream actually wavered on this particular run is not. The metrics measure that.

Metric computation is independent per node pair, so pairs run in parallel. A judge call blocks per pair, so running them sequentially makes total time the sum of pairs; a thread pool reduces it to the slowest pair. That change is what brought the demo diagnosis down from minutes to seconds.

Keeping the table alive

One principle set at the start was to keep the ontology as data, not code. The cells live in a YAML file and the engine only reads it. Editing the table and deploying the code become separate acts.

The first update cycle ran in August 2026.

Three cells carrying an explicit non-application marker say something about the table’s character. Legal nodes have no corresponding provision under Korea’s AI Basic Act, and rather than leaving those blank, the cells say “no corresponding provision.” A blank cannot be distinguished from “not looked at yet.”

The added MITRE techniques target agents: poisoning a tool’s definition, implementation, and runtime response separately; distributing poisoned AI artifacts; tampering with prompt construction logic. The local snapshot was the February 2026 edition and upstream added these in July.

The MIT material was compared and deferred. Checking v4, published 4 December 2025, against the structure in use showed the causal and domain classifications unchanged; only the classification statistics were stale. No restructuring, just a number refresh, so it moved to the next cycle.

This cycle’s updates did not change any diagnosis score. The regression check returned 3 RED for legal and 3 RED for credit underwriting, the same as before. Evidence and references got thicker without the verdicts moving. Without that check, updating the table quietly becomes changing the verdicts.

The order of checks is fixed: whether referenced frameworks have new editions, whether regulatory timing moved, whether newly verified incidents exist. Judgment load rises as you go. A standard update means redrawing mappings; attaching an incident means deciding which axis and which failure mode it belongs to. Enough of those decisions accumulate and you get an observation that forces a look at the axis layout, as happened when five of six incidents turned out to have no attacker.

Closing - the table is the asset

Asked which part of this project cannot be ported, the answer is the table. The runtime can be swapped and the language model can be swapped, but the judgment rules in 36 cells have to be rebuilt.

That is why the table is kept as data with its own update cycle. Regulatory dates moved once in six months, incident records keep accumulating, standards keep arriving. A table starts going stale the day it is built, and the speed of that staleness is set by the update speed of the material it references.

The most striking thing in the results was the distribution. In both workflows the highest-scoring node was neither the most sophisticated one nor the one using a language model. It was the node where the human lets go.

Sources

• OWASP GenAI Security Project, Top 10 for Agentic Applications (published 2025-12-09) and Top 10 for LLM Applications 2025
• MITRE ATLAS v5.4.0 and the upstream 2026.07 data repository
• MIT AI Risk Repository, v4 update December 2025
• NIST AI RMF 1.0 and the Generative AI Profile (NIST AI 600-1)
• Liu et al., AgentBench (2024)
• Zou et al., PoisonedRAG (2024)
• Zhou et al., WebArena (2024)
• Scoring logic in scripts/agents/aggregator.py, metrics in scripts/metrics/{ips,confdecay,laaj}.py, ontology in scripts/data/mapping-ontology-v0.1.yaml (public repository)