You Find Out Where It Breaks After You Ship - Moving the Pre-Deployment Check into a Process

The pre-deployment check is priced against human hours today. Move that price into a process and what remains is a table of what to look at and a gate for when to stop. Where an AI workflow breaks usually surfaces after it ships. Node-level review before deployment does exist, but only as a consulting engagement: one to two senior-weeks per workflow, and no way to re-run it when the workflow changes. This piece covers why that quadrant is empty, when the regulatory deadlines actually land, and what remains once the check becomes a re-runnable single run.

FDE stands for Forward Deployed Engineer. The role came out of Palantir: a product engineer who goes into the customer’s environment and fits the product to that company’s real data and real work. Instead of selling what headquarters built as-is, they assemble the last layer on site. AI companies now hire under the same title.

This project did not move that whole job. It took one slice of what an FDE does on site, the part where you look at a workflow before wiring it up and say what will break, and turned that slice into an agent. It was submitted to UiPath AgentHack 2026 and took third prize in the Maestro BPMN track against more than 333 submissions from 104 countries. The code and ontology are in a public repository.

The prize is not the subject of this series. The subject is the scoring rules built along the way and how they behaved in an actual run.

Pre-deployment workflow review is a blank square on the map

The failure statistics are familiar. MIT NANDA’s State of AI in Business 2025, drawing on 150 interviews, 350 survey responses, and 300 deployments, found that 95% of generative AI pilots never reach production. RAND reported 80.3% of AI projects fall short of intended business value, and Gartner predicted 30% of generative AI projects would be abandoned after proof of concept by the end of 2025.

What the numbers point at is not the failure rate but when failure surfaces. Nothing happens while the pilot runs. It appears after production wiring, at the most expensive moment to fix.

Tools that check before deployment do exist. They all sit in different squares.

Incident databases are built for policymakers and researchers, so they do not answer “should we run this proof of concept.” Governance SaaS observes systems already running. Red teaming hits the model and prompt layer, not the workflow layer. The square for workflow-level review before deployment is filled by consulting alone.

---
config:
  look: handDrawn
  theme: neutral
---
flowchart LR
    subgraph Pre["Before deployment"]
      A["Model and prompt<br/>red teaming"]
      B["Workflow layer<br/>consulting only"]
    end
    subgraph Post["After deployment"]
      C["System monitoring<br/>governance SaaS"]
      D["Case records<br/>incident DB"]
    end
    A --> C
    B --> C
    C --> D

Why the square stayed empty

A blank square is not the same as an untried one. Filling this one needs three things at once, and each has a different owner.

First, the judgment rules have to come out on paper. When a consultant looks at a workflow and says “this node is risky,” the criteria are mostly in their head. Writing them into a table makes them re-runnable and removes the basis for billing at the same time. The party holding the rules has no reason to publish them.

Second, incident data has never been organized by node. AIID accumulates by case, the OWASP Top 10 by vulnerability class, MITRE ATLAS by attack technique. Nothing is sorted by workflow components such as “document extraction node” or “auto-approval node.” Using them at node level means re-anchoring all three onto node types, and that turned out to be the longest single task in this project.

Third, it sits where regulatory interpretation meets workflow design. Counsel reads clauses; architects draw nodes. Only when both land in the same table does “this node falls under this clause” appear, and it is unclear who maintains that table. Regulation keeps moving and the table has to move with it.

Incidents happen without an attacker

Filling the square starts with deciding what to score against. So I collected incidents verified between December 2025 and June 2026 and attached them to cells. Six survived; five had no attacker.

• Amazon Web Services’ coding agent Kiro determined a live environment needed to be deleted and recreated, and executed that, producing roughly 13 hours of Cost Explorer interruption in a mainland China region. Amazon disputed the framing, calling it misconfigured access controls rather than AI, then added senior-engineer sign-off for AI-assisted changes. AIID incident 1442.
• Waymo recalled 3,067 vehicles over a defect that could cause a vehicle to pass a stopped school bus. Austin ISD recorded roughly 20 illegal-passing incidents in the 2025-2026 school year, one of them after Waymo said the problem was fixed.
• At Amazon retail, an agent surfacing internal guidance pulled stale references, and four high-severity incidents hit in a single week in March, one lasting six hours. A human was in the loop and acted on the advice anyway.
• Baidu’s Apollo Go had a regulator-confirmed incident in Wuhan on 31 March 2026, followed by a nationwide permit suspension on 29 April.
• At Sears Home Services, an AI assistant’s chat logs and call recordings were exposed: 3.7 million records, 4.3TB. An AI node’s logs inherit the sensitivity of everything that passed through it.
• At PocketOS, a coding agent deleted a production database in nine seconds. The backups on the same volume went with it, and the credential used had been issued for custom domain management.

The path to six is worth writing down as numbers. Between September 2023 and May 2026, 7,246 AI incidents were reported through public channels. Of those, 344 were verified as actual harm to a production system, which is 4.7%. Of those 344, 188 had no attacker, more than half the verified set. Finally, six were individually re-verified against primary reporting and AIID entries and attached to cells. The harm came from an agent doing what it was told, faster than a human could intervene, with permissions broad enough to make the action irreversible. OWASP and MITRE ATLAS are both built on the presumption of an attacker. Routing excessive-permission failures to the security axis mis-scopes the mitigations toward adversarial controls.

What failed verification is recorded too. “Air Canada’s AI booking agent rebooked roughly 1,247 passengers onto wrong flights in January 2026,” “Klarna AI refund incident, February 2026,” “Step Finance AI trading agents executing unauthorized transfers.” All four traced back to the same single vendor marketing blog, with no regulator filing, no AIID entry, and no press coverage. The citable Air Canada case is still Moffatt v. Air Canada (BC Civil Resolution Tribunal, February 2024, AIID 639), and the Klarna story is a 2024-2025 arc that ended with the company resuming human agent hiring in May 2025. Four more cases of “a runaway loop leaked this many dollars,” carried by cost-management vendor blogs, came out for the same reason: no identified victim and no primary reporting. The pattern is already covered by the verified aggregate above, so there is no reason to reach for a weak source. All of them sit in the repository marked “not confirmed, do not cite,” each with its reason.

Most organizations have no gate at all

This whole series is about gates, and there is a survey measuring how many exist in the field. The Cloud Security Alliance and Token Security asked 418 organizations in April 2026.

38% require human approval when an agent steps outside its permission scope. The other 62% have no such gate. Only 11% automatically block unauthorized actions.

Lay that next to the incidents above and it fits. The remediation in the AWS case was adding an approval step, and more than half the verified incidents had no attacker. What would have helped was not defense but a gate, and six organizations in ten do not have one.

What this check costs today, and why it is thrown away

Consulting is what performs pre-deployment workflow review, and the shape is fixed. A senior consultant takes one workflow, spends one to two weeks, and produces a slide deck.

The price attaches to human hours, not to accuracy. So the deliverable stops updating the moment the engagement ends. AI workflows change more often than engagements do. Add a node or adjust an auto-approval threshold and last month’s diagnosis says nothing about that part. The real cost of a single-use deliverable is not its price but that nobody knows when it started being wrong.

This sits in the same place as two earlier threads. As set out in Advisor Is a Price Tag, Not an Architecture, a pattern that looks architectural is sometimes a temporary fit to the pricing structure of its moment. Consulting diagnosis is the same. Not because human judgment is the only way, but because the rules were never standardized, so they got billed as human hours. And as covered in Why Korea Doesn’t Buy Software, the Korean market has long leaned toward solving this class of work through services rather than buying a product.

The regulatory deadlines are widely misquoted

Ask when this check becomes mandatory and the EU AI Act comes up first. The dates moved once in the summer of 2026, and that part is frequently quoted wrong.

Plenty of material still says Annex III high-risk obligations apply from 2 August 2026. The Digital Omnibus regulation (Regulation (EU) 2026/1744), in force from 27 July 2026, moved that to 2 December 2027. Article 50 transparency obligations were not deferred and have been live since 2 August 2026. The penalty ceiling for prohibited-practice violations is 7% of global turnover or 35 million euro, whichever is higher (Article 99).

Korea did not defer. High-impact AI obligations under the AI Basic Act have been in force since 22 January 2026, and MSIT’s grace-period posture on fines is an enforcement stance, not a suspension. How high-impact status is determined also needs care. Under Article 33 the operator conducts its own advance review and may seek MSIT confirmation, so a deliverable should read “likely in scope, verify under Article 33” rather than asserting a legal conclusion.

Unfolded into nodes, it becomes visible where each clause attaches.

The character differs by node. The first two are about whether records were kept; the last three are about whether the basis for a judgment can be explained. A slide-deck deliverable answers the first kind with a checklist and does not answer the second. Reconstructing the basis after the decision does not work; it has to be stored at decision time.

One thing worth flagging

A deferred deadline gets read as a reason to defer the check, and the table makes that hard to sustain. What moved in Europe is the Annex III high-risk obligation. Transparency obligations are already in force, and Korea never deferred. And the Annex III date moving to December 2027 also means workflows deployed in the interim come into scope retroactively. What was deferred is the filing date, not the design date.

What got built: a pre-deployment check that finishes in one run

Three steps. Submit the workflow, score each node, then govern the result.

---
config:
  look: handDrawn
  theme: neutral
---
flowchart LR
    A["Submit workflow<br/>node inventory table"] --> B["Score each node<br/>general failure · security · handoff"]
    B --> C["Handoff runtime metrics<br/>IPS · confidence decay · judge"]
    C --> D["Gate check<br/>any of four conditions"]
    D -->|"human needed"| E["Approval pending"]
    D -->|"all clear"| F["Report, then close"]

The middle changed most in the move to a process. Node scoring is not delegated to a language model’s opinion; it runs as an ontology lookup. An axis score is the mean risk score of the cells attached to that node, and feeding the same workflow returns the same value. If the output is going to serve as audit material, you need that property first.

Combining the three axes uses weights: handoff 0.4, security 0.3, general failure 0.3. The largest share went to the axis that looks between nodes, which is the axis with no public framework behind it. How that table was built is the subject of the next part.

How the run actually sequences

The first insertion point is just before a team submits a new AI workflow for deployment approval. The second is when a node is added to a running workflow or a threshold is adjusted.

Submission takes a node inventory table: a markdown table with node id, function, and how AI intervenes. The repository also holds a BPMN XML parser, a Mermaid parser, and a module that reads an image with a multimodal model and converts it to Mermaid, but what the coded agent on the UiPath path actually reads is that table. Automatic conversion from raw BPMN XML is not wired in yet.

The receiving end splits two ways. Clear the gate and a heatmap and summary report are generated and retained as audit material. Trip it and an approval screen is created for a human, carrying the workflow name, the maximum risk score, the count of nodes over threshold, the reason it escalated, and a one-page summary. The approver can approve, approve on condition that the recommended mitigations are applied, or reject.

What a deterministic lookup gives up

Making the same input produce the same diagnosis makes audit response easier. The cost is that node types absent from the ontology are not scored. What is not in the table does not get looked up, so an unfamiliar workflow shape requires a human to extend it. Delegating judgment to a language model removes that problem and replaces it with different scores on every run of the same workflow.

Which one to pick depends on where the output goes. For regulatory material, a score that does not reproduce is unusable. So scoring stays a lookup, and the language model is pushed toward phrasing evidence and composing summaries. Extending the table remains periodic human work.

What is unfinished is recorded too. Among the handoff metrics, the one that uses a judge model defaults to a mock backend, so running a real judgment requires naming a backend. Observability integration is unresolved. The performance figures for graph-based retrieval were withdrawn after the measurement turned out to be circular and are pending re-measurement. Those numbers were also removed from the deck and the submission.

Closing - where the price attached to human hours moves

Moving the pre-deployment check into a process leaves two things: a table of what to look at, and a gate for when to stop. The rest becomes re-runnable.

The reason this matters is not cost. Consulting diagnosis is expensive, but the harder problem is that nobody knows when it stopped being valid. A check that re-runs whenever the workflow changes is a different object even at equal accuracy. Making the check cheap and making the check stay alive are different jobs, and the second one was the actual problem.

That leaves the question of how to fill the table. Public datasets cover model risk and security risk well enough. But for the axis carrying the largest weight, the one that scores what happens between one node and the next, nothing existed.

Sources

• MIT NANDA, State of AI in Business 2025 (150 interviews, 350 survey responses, 300 deployments)
• Gartner, 30% of Generative AI Projects Will Be Abandoned After PoC
• AI Incident Database, incident 1442 (AWS Kiro), incident 639 (Moffatt v. Air Canada)
• CBS News, Waymo recalls 3,067 vehicles over school bus software defect
• Cloud Security Alliance and Token Security, agent governance survey (April 2026, n=418)
• Cyera, Agent-Inflicted Damage: Inside the Real-World Failures of Enterprise AI Systems
• Regulation (EU) 2026/1744 (Digital Omnibus, in force 2026-07-27), EU AI Act Articles 50 and 99, Annex III
• Framework Act on AI Development and Trust (Korea), in force 2026-01-22, Article 33
• Scoring logic in scripts/agents/aggregator.py and scripts/core/engine.py (public repository)