When AI Models Get More Capable, Governance Gets More Important — Not Less

The moment that should change how you think about AI governance

In March 2026, Nicholas Carlini — a Research Scientist at Anthropic — pulled up a terminal at the [un]prompted conference in San Francisco and gave Claude a simple instruction: find vulnerabilities.

Ninety minutes later, Claude had discovered a blind SQL injection in Ghost CMS — a publishing platform with 50,000 GitHub stars that had never had a critical security vulnerability in its history. The vulnerability allowed an unauthenticated user to compromise the admin database and take full control. Claude then found heap buffer overflow vulnerabilities in the Linux kernel’s NFS daemon — bugs that had existed since 2003, predating the Git version control system itself.

This wasn’t a one-off demo. Anthropic’s Frontier Red Team has found and validated more than 500 high-severity vulnerabilities in open-source software using Claude — with no custom scaffolding, no specialized prompting, and no task-specific harnesses. In a subsequent interview with David Adrian, Deirdre Connolly, and Thomas Ptacek on the Security, Cryptography, Whatever podcast, Carlini described the setup: he runs Claude Code with --dangerously-skip-permissions and a prompt he didn’t even write — Claude wrote the agent that finds the bugs. No authorization constraints. No boundary enforcement. No audit trail of agent actions. Just a model pointed at code and told to find something. And it works — because the model reasons about code the way a skilled security researcher would, reading commit histories, identifying patterns, constructing proof-of-concept exploits.

Every security finding in this post is based on Opus 4.6 — the model available today. And now Claude Mythos — a reported step change beyond Opus 4.6 — is in internal testing. Security stocks dropped 5-9% on the leak alone. A step change in capability doesn’t narrow the governance gap. It makes the gap more stark and more serious to address — because the agents operating without governance infrastructure are about to become dramatically more powerful.

The security community is already calling this a “bitter lesson” — the realization that scaling works, that model capability improvements overwhelm hand-built tooling, and that elaborate scaffolding is replaced by the next model checkpoint. Carlini: “Scaling works, which is an unfortunate thing that I didn’t want to believe, but empirically has turned out to be true.” Ptacek: “All of the fun problems are gone.”

Nate B. Jones extends this insight to the broader AI community in his video “Claude Mythos Changes Everything. Your AI Stack Isn’t Ready”: as models get dramatically more capable, the systems around them need to get dramatically simpler. Stop over-specifying process, specify what and why, and let the model handle how.

The security researchers and Jones are both right. And they’re both incomplete. Here’s the distinction that matters.

The bitter lesson applies to mechanism — not governance

Jones’s argument is correct for mechanism scaffolding: procedural prompt instructions that compensate for model limitations. A 3,000-token system prompt that says “first classify the intent into one of 14 categories, then route to the appropriate handler, then retrieve the top five knowledge-base articles using hybrid search with alpha equals .7” — yes, simplify that. A smarter model can figure out the procedure. Specify the outcome and the constraints, and let the model work.

But there is a category of infrastructure that does not simplify as models improve. It gets more important:

• Who is accountable for each step of a multi-agent workflow
• Which decisions must be deterministic — compliance classifications, escalation routing, boundary enforcement — versus which genuinely benefit from LLM judgment
• What happens when things go wrong — structured exception handling with typed responses, not generic retry loops
• What constraints propagate from one delegation boundary to the next — a read-only constraint at step 1 must be structurally enforced at step 5, not hoped for
• What audit trail proves the agent did what it claims, accessed only what it was authorized to access, and classified findings by governance logic rather than probabilistic inference

This is governance infrastructure. It’s not scaffolding compensating for model weakness. It’s architecture that becomes more critical as models become more capable — because more capable models operating with less human oversight at higher speed need stronger structural governance, not weaker.

The security findings make this concrete. A model that can find zero-day vulnerabilities in Ghost CMS and the Linux kernel in 90 minutes is phenomenally capable. Now ask: who governs the security agent?

The question nobody is asking

When Carlini demonstrated Claude finding zero-days, the security community’s response was appropriate alarm about offensive capabilities. But almost nobody asked the governance question: what execution governance infrastructure should exist around a security agent?

Carlini runs --dangerously-skip-permissions because no governance infrastructure exists for the alternative. The flag name itself is the tell — the security community’s most capable practitioner runs ungoverned agents because governed agent execution hasn’t been formalized. That’s the gap.

Consider what’s needed:

Identity and authorization. The agent performing the audit must be identifiable as a governed entity — not anonymous automation running under shared credentials. Its authorized scope must be explicit: read configuration, yes. Modify configuration, never.

Deterministic decision separation. When the agent classifies a finding — “this MFA configuration is non-compliant” — that classification must be deterministic. A decision table with defined rules, not LLM inference. The agent can observe the configuration (that’s what LLMs are good at). The classification must be governance logic — reproducible, auditable, ungameable.

Structured exception handling. When the agent encounters a configuration it cannot interpret, it must not guess. It must escalate — to a specific handler, with specific evidence, through a defined escalation path. This is a typed exception with a governed response, not a retry loop.

Boundary constraints that propagate. A read-only constraint at the start of the audit must be structurally enforced at every subsequent step. Not restated in each prompt. Not hoped for. Architecturally excluded — the agent cannot click Edit, Modify, or Delete because those actions are outside its authorized scope at every governance boundary.

Audit trail as governance evidence. Not logging. Governance evidence: what was observed, by what logic it was classified, what constraints were in force, what the agent was authorized to do and what it was structurally excluded from doing.

None of this simplifies as models improve. All of it gets more important.

The discipline that already solved this

Every governance requirement listed above has a named, standardized solution. The discipline is Business Process Management — codified in the ABPMP BPM CBOK, standardized through BPMN 2.0 and DMN 1.0 (both OMG standards), and operational in banking, healthcare, manufacturing, government, and insurance for over two decades.

Responsibility models (RACI). Deterministic decision tables (DMN with UNIQUE hit policy). Structured exception handling (BPMN event taxonomy — typed exceptions with governed responses). Governed decomposition (subprocesses with their own governance interfaces). Activity-level governance attributes (21 typed attributes derived from RACI, SIPOC, Value Stream Mapping, and ISO 31000). Audit trails as governance evidence.

The AI agent community is reinventing these patterns from scratch, informally, and incompletely. They add “guardrails” without structured policy linkage. They implement “tasks” without governance attributes. They describe “handoffs” without governed delegation interfaces. They are solving a solved problem, badly.

The asymmetry that proves the point

Carlini’s interview reveals two asymmetries that make the governance argument concrete.

Finding bugs is easy. Fixing bugs is governance. Carlini’s team found 122 crashing inputs in Firefox — all confirmed real bugs by Mozilla, 22 assigned CVEs. But patching is harder than finding. As Carlini puts it: developers spend almost as much time reviewing AI-generated patches as writing fixes themselves. Why? Because patching requires understanding context, maintaining functionality, ensuring accountability — exactly the process governance concerns the BPM discipline addresses. Offense is mechanism work that scales with model capability. Defense requires governance infrastructure.

Bug bounties are drowning. Google Chrome’s David Adrian reported 5x the submissions in 2025 relative to 2024. March 2026 — just 19 days in — already had double February’s total. Mozilla’s biggest month for bug reports coincided with Carlini’s batch, but even excluding Anthropic’s findings, it was the biggest month in two years. Bug bounty programs are being overwhelmed because they lack the triage infrastructure to handle AI-generated volume. Severity classification by decision table, not human judgment for every report. Escalation routing by governance logic, not queue position. This is a governance infrastructure crisis happening right now — and DMN decision tables and BPMN escalation flows are the structural answer.

What governed security actually looks like

We designed a governed security compliance audit to demonstrate the difference. A Claude agent navigates a real identity management console, reads MFA configuration, classifies findings, and produces a compliance report — all governed by BPM execution governance:

• 14-step BPMN process model. Navigate → Authenticate → Read configuration → Classify finding → Record → Report. Each step carries typed governance attributes.
• Read-only boundary constraint. The agent is structurally excluded from clicking Edit, Modify, Delete, or any write-capable control. This is an architectural boundary, not an advisory guardrail.
• DMN decision table (UNIQUE hit policy). MFA compliance classification: 9 rules, zero ambiguity, zero LLM judgment on compliance determinations. The agent observes (LLM capability). The decision table classifies (deterministic governance logic). Observation and judgment are separated.
• Escalation routing (FIRST hit policy). Critical findings route to CISO immediately. Indeterminate findings escalate to a human auditor. The routing is a governance decision encoded in a decision table, not an LLM inference.
• Rule 9: the escalation trigger. When the agent cannot read or interpret configuration, it does not guess. It escalates with screenshots as evidence. Governance by design, not guardrails by hope.

Every element comes from established BPM patterns. The novel contribution is the application to agent execution. This is what structural governance looks like — and it becomes more valuable, not less, as agent capability increases.

The three-layer architecture

Safe, accountable AI agent deployment requires three governance layers. Each is independently necessary:

Constitutional AI (the substrate). Training-time governance shaping the model’s character. The behavioral floor beneath which no agent operates. This is where Anthropic’s work lives — and it’s exceptional.

Intent Stack (governance context). A four-layer runtime governance architecture specifying what is delegated, by whom, under what authority, with what constraints. Four governance concerns: Intent Discovery, Intent Formalization, Specification, Runtime Alignment. Published at intentstack.org, CC BY 4.0.

BPM/Agent Stack (execution governance). Execution governance specifying how authorized work gets done: process models, decision tables, activity attributes, exception handling, responsibility assignment, controlled vocabularies. Formally governs Orchestration, Integration, and Execution. Published at bpmstack.org, CC BY 4.0.

Together, seven governance concerns across two companion specifications. The Intent Stack governs what and why — governance context. The BPM/Agent Stack governs how — with what process, roles, logic, and controls. Neither duplicates the other.

Jones’s “bitter lesson” applies within the BPM/Agent Stack — simplify the procedural mechanism, let the model handle execution. But the governance architecture itself — the three layers, the structural separation of deterministic and probabilistic decisions, the responsibility models, the boundary propagation, the audit trails — that’s permanent infrastructure. It doesn’t simplify. It scales.

The timing

The NIST NCCoE published a concept paper in February 2026 — “Accelerating the Adoption of Software and AI Agent Identity and Authorization” — seeking public input on exactly this problem: how to identify, authorize, and audit AI agents operating within enterprise systems. Comments are due April 2, 2026.

The BPM 2026 conference in Toronto (September 28 – October 2) is where the academic community is converging on the same gap: “Agentic BPM: A Research Manifesto” (Kampik et al., March 2026) explicitly seeks the governance infrastructure that the BPM/Agent Stack provides.

The skills ecosystem — Anthropic’s SKILL.md format, adopted by OpenAI and Microsoft — has created a universal deployment primitive for agent methodology. Skills govern individual activities well. But the moment skills compose into workflows, every BPM governance concern re-emerges: routing logic, constraint propagation, responsibility assignment, exception handling, deterministic decision separation. Skills are the new activities. The composition gap between skills and governed workflows is the pre-BPMN workflow problem recreated at AI-agent scale.

Mythos-class models make individual skills simpler — less procedural scaffolding needed. They make the composition gap wider — more capable models composing more skills at higher speed with less oversight need more governance at composition boundaries, not less.

A true fact about the world

Carlini, in that same interview, puts it plainly:

“It was nice when we didn’t know about Spectre. That was just a nice world to live in. But too bad — it turns out a true fact about the world is you can do side channels. Same thing with ROP. Wouldn’t it be nice if write-or-execute was just the perfect solution? But it’s not. Here is a true fact about the world: you can do this thing. Now we have this other thing that is true in the world. We have these language models that can find these bugs and potentially soon exploit them. And maybe we wish that we didn’t have these things, but they exist, and we should measure the capabilities that they have so that we aren’t blind to what happens.”

Every uncomfortable discovery in that lineage — Spectre, ROP, stack smashing — produced structural countermeasures. Microcode patches. ASLR. DEP. Stack canaries. Secure development lifecycles. The security community didn’t wish those problems away. They built infrastructure.

The governance gap is the next true fact about the world. AI agents are operating in enterprise environments without runtime execution governance. The capability is here. The governance infrastructure is not. And a step change in capability — Mythos — is about to make the gap more stark.

Simplify the mechanism. Strengthen the architecture. Build the infrastructure before it’s needed, not after.

Rob Kline is the creator of the Intent Stack Reference Model and the BPM/Agent Stack. He brings decades of deep expertise in Business Process Management, Business Rule Management, taxonomy, and ontology — applied to the execution governance gap in AI agent deployment.

Contact: rob.kline@practicalstrategy.ai | intentstack.org | bpmstack.org