
AI governance is a data infrastructure problem.
Most organizations treat AI governance as a policy question. What actually blocks them sits one layer down, in the data: a model may be capable while the organization around it is not legible.
In a salt works, low walls and gates decide where the water goes.
Tanush Jagdish · June 2026
Conversations about AI governance almost always start at the top: policies, committees, approvals, model risk, acceptable use, and human review. Those concerns are real.
But I keep watching the same thing happen. An organization buys into the idea, writes the memo, and stands up the committee. Then it tries to put an agent into meaningful work, and the first thing that stops it is nowhere in the policy. That obstacle sits lower in the stack.
An agent needs to know which source is authoritative, and that is where organizations hit the wall.
It needs documents with sections, owners, dates, and status. It needs to tell draft guidance from approved policy, a stale spreadsheet from a live system, and a customer note from a contractual obligation. When that structure doesn’t exist, “human in the loop” stops being a safety control. It becomes a person cleaning up the same context failure over and over, indefinitely.
So here is the claim this piece makes: AI governance is a data infrastructure problem. Policy is the part you can see. Data is the part that decides whether any of it works.
#1A capable model inside an illegible organization
AI readiness is not just a software-procurement question. A firm can buy the best model available and still be unable to use it safely, because its knowledge lives across shared drives, inboxes, slide decks, PDFs, and people’s private memory.
A model may be capable while the organization around it is not legible. Everyone benchmarks the model; almost no one benchmarks whether their own organization can be read, and that second question is usually what blocks progress.
Existing frameworks mostly point elsewhere. Anthropic’s Responsible Scaling Policy governs how a frontier model is built and released. Under the EU AI Act, most obligations fall on the providers of AI systems, while NIST’s AI Risk Management Framework and ISO/IEC 42001 set management-level controls across the life cycle. None of them says much about the daily practice of a firm whose staff use coding agents and chat assistants all day: which sources those agents may read, how their output gets checked, and what gets logged. Governing that daily use is the open problem.
#2Belief is not measurement
Evidence for this starts with what happens when you measure instead of asking people how they feel.
Two 2025 studies asked whether AI makes developers faster. Only one measured the answer, and its developers believed the opposite of what it found.
Believed faster, measured slower.
A self-reported gain, with no controlled measurement beside it.
Sources: METR, randomized controlled trial with experienced open-source developers, July 2025; Anthropic, “How AI Is Transforming Work at Anthropic,” survey of 132 engineers and researchers, August 2025.
METR ran a randomized controlled trial with experienced open-source developers in 2025. Before the study, the developers expected AI to make them 24% faster; afterward, they believed it had made them about 20% faster. Measured, they were 19% slower. Belief and measurement ended up roughly 40 points apart.
Anthropic surveyed 132 of its own engineers and researchers in August 2025 and also studied internal usage data. Respondents reported a productivity gain of about 50%. That figure may be accurate, but it is a self-report, the same kind of number METR’s developers got wrong.
Neither study settles how much AI helps. Together they show why self-assessment cannot stand in for measurement. Without instruments that record what work actually took, organizations fall back on belief, and belief feels like governance without being governance.
Our own research found the same shape. In The Confidence-Capability Gap, confidence coded from 54 interview transcripts averaged 66 out of 100, while demonstrated capability averaged 37, and confidence ran higher in 53 of the 54: people sound more ready than they can show, with no instrument in between to tell the two apart. Its companion, The Widening AI Gap, covers the capability side.
#3Everyone skips the middle layer
So where does legibility come from? Most organizations get this wrong, and the mistake is one of investment allocation.
The AI Agent
Built · fundedModels, prompts, and interfaces: the visible part, and where the money goes.
The AI Sandbox
MissingA structured-access layer that reconciles the fragments into one current, consistent picture of what is true. Before an agent can act, it needs structured access to that picture. Almost no one has built this.
Enterprise Data
ScatteredFive answers to every question, scattered across five systems, with no agreement on which one is true.
That middle layer is the governance layer, even though it looks like infrastructure rather than a binder.
Look at where the money goes: to the top of the stack, the agent. Models, prompts, and interfaces are the visible part.
Real constraints sit in the middle layer, which almost no one has built. Underneath the agent sits the enterprise’s actual knowledge: a wiki, a SharePoint site, a pile of PDFs, an inbox, a ServiceNow instance. Every question has five answers scattered across five systems, with no agreement on which is true.
Between the agent and that mess there should be a structured-access layer, which we call the AI Sandbox, that reconciles the fragments into one current, consistent picture of what is true. Before an agent can act, it needs structured access to that picture. Most organizations skip this layer. They jump from messy data straight to giving the agent access, then act surprised when the agent confidently grounds a board memo in last quarter’s stale draft.
That middle layer is the governance layer, even though it looks like infrastructure rather than a binder.
#4Stop asking “is AI allowed?” Start using a matrix.
A binary question, whether AI is allowed here or not, collapses two different axes into one and forces you either to over-restrict or to over-trust. You need both axes.
Supervised Delivery
Agents produce complete drafts, and a human reviews them before anything reaches the wider organization or an outside party.
Path step 2Full Agency
AI operates as the product, and external stakeholders interact with its output directly.
Path step 4Copilot
Questions and analysis. A human reviews and acts on every output. Almost everyone is here.
Path step 1Internal Agents
Agents run autonomously, but only on internal workflows: a safe place to test real automation before exposing it.
Path step 3Mature your context management first, the legibility layer, then test autonomy internally, then extend exposure. Build the governance at each stage before you expand to the next.
One axis is autonomy: how much the agent does on its own, from answering a question you act on to taking the action itself. Exposure is the other axis: who sees the output, from purely internal to external.
Together they define four quadrants and a path through them:
- 1.Copilot. Questions and analysis; a human reviews and acts on every output. Almost everyone is here.
- 2.Supervised Delivery. Agents produce complete drafts, and a human reviews them before anything reaches the wider organization or an outside party.
- 3.Internal Agents. Agents run autonomously, but only on internal workflows: a safe place to test real automation before exposing it.
- 4.Full Agency. AI operates as the product, and external stakeholders interact with its output directly.
Most important, the recommended path is not the diagonal. You don’t earn external full agency by becoming a little more autonomous and a little more exposed at the same time. Mature your context management first, which is the legibility layer; then test autonomy internally; then extend exposure. Build the governance at each stage before expanding to the next. Part 1 · Chat makes the same point about verification: match enforcement to the stakes.
#5Orchestration is governance you can see
An agent’s own runtime behavior is already a governance surface, if your architecture exposes it.
“Prepare quarterly risk assessment for the audit committee.”
Skills
Loaded instructionsRisk frameworks · board-reporting conventions
Subagents
Spawned processesResearch · due-diligence sub-tasks
Tools & APIs
External connectionsGRC platform · policy database
Which instructions did it load? Which subprocesses did it spawn? Which systems did it touch? When those choices are logged and scoped, orchestration is governance you can see, turn by turn.
Take a real task: “Prepare the quarterly risk assessment for the audit committee.” A capable agent doesn’t do that with one capability; it orchestrates. It loads skills, such as your risk frameworks and board-reporting conventions. It spawns subagents for the research and due-diligence subtasks. It connects to tools and APIs, such as your GRC platform and policy database.
Each of those steps can be governed. Which instructions did the agent load? Which subprocesses did it spawn? Which systems did it touch? In an architecture that logs and scopes those choices, orchestration becomes governance you can see, turn by turn. Part 5 · Managed architecture shows how that orchestration runs, with a dispatcher, isolated runtimes, and per-user secrets, and Part 6 · Institutional brain shows how it extends to a team.
Useful governance, in other words, is operational: source-backed memory, document lineage, permission-aware retrieval, cited outputs, approval gates, versioned artifacts, and logs that show what the agent saw and did. Each item on that list is data infrastructure. Together they make governance far less performative, because the evidence exists inside the work instead of in a policy binder written after the fact.
#6Leading companies build both at once
You can already see the market split between organizations that treat governance as infrastructure and those that treat it as paperwork.
Anthropic
Internal engineeringStudies its own AI use through a staff survey, interviews, and usage transcripts.
Staff report using AI in about 60% of their work (2025 survey).
Every
Product engineeringCompound engineering: plan, work, review, compound. Each lesson becomes a rule its agents read.
Planning and review take about 80% of engineers’ time.
Shopify
CEO memo, 2025Reflexive AI use is a baseline expectation; teams show why AI can’t do a job before asking for headcount.
AI use is part of performance and peer reviews.
Duolingo
Content creationAI drafts course content; people review it.
148 new courses in under a year; the first 100 took about 12 years.
Stripe
Data and developer toolingA text-to-SQL assistant and a central MCP server give staff and agents structured access to company data and tools.
Access runs through one governed layer instead of ad hoc connections.
In these accounts, none began with a complete governance policy. Each found an entry point that fit its context and grew governance alongside capability.
Public accounts from five companies show the pattern. Anthropic studies its own AI use through surveys, interviews, and usage transcripts, which builds measurement into the work. Every codified a loop it calls compound engineering (plan, work, review, compound), with most engineering time spent on planning and review and each lesson written into rules its agents read: governance as a habit, not a memo. Shopify made reflexive AI use a baseline expectation, asks teams to show why AI cannot do a job before adding headcount, and includes AI use in performance reviews. Duolingo has AI draft course content that people then review. Stripe built internal tools that give employees and agents structured access to company data.
In these accounts, none of the companies began with a complete governance policy. Each found an entry point that fit its context and scaled governance alongside capability. You don’t write the binder first and deploy second. You build a legible substrate, and the controls grow out of it.
#7Meta-governance: let the agent draft its own rules
One more move surprises people: use the agent to draft its own governance documents.
At Olito, we use our coding agent to help write the policies that govern its use. That sounds circular, but it is rigorous. An agent can describe its own capabilities, so it surfaces what it can and cannot do. Writing rules with it forces you to state assumptions you would otherwise leave implicit. And the result is a living document that changes as the agent’s capabilities change, instead of a static PDF that is stale the day it is signed.
Part 4 · Personal brain describes the same self-improving loop, here pointed at the governance layer: notice a mistake, correct it in the moment, write the rule down, and it does not recur. A CLAUDE.md file and a governance policy compound through the same mechanism.
Who qualifies as a principal who gets their own brain?
Can one person’s drafting history see another’s?
When the system learns a rule, does it propagate to everyone or stay in one layer?
What’s the default when an action exceeds a principal’s authority?
What can each principal see of the system’s own activity?
What never crosses between people: equity, candidate evals, private notes?
Who is allowed to add a rule that binds the others?
Is the audit trail on by default, or only when something looks off?
Each is an L1, L2, or L3 institutional-brain question, and each is a governance question. An institutional brain is the governance layer.
Scaling past one person to a team makes the governance questions sharper, not vaguer. Who qualifies as a principal with their own brain? Can one person’s drafting history see another’s? When the system learns a new rule, does it propagate to everyone or stay in one person’s layer? What never crosses between people, such as equity conversations, candidate evaluations, or private notes? Is the audit trail on by default, or only when something looks off? Part 6 · Institutional brain treats these as the L1, L2, and L3 questions, and each is a governance question. An institutional brain is the governance layer.
#8A test that matters
Set aside the committees, and one test matters for a regulated team. Whether the model can answer is beside the point. Can the organization show why the answer was grounded, which source controlled, who approved the action, and how the system learned without silently rewriting the rules?
If you can answer those four questions with evidence that lives inside the work, you have governance. If you can only answer them with a binder written after the fact, you have paperwork. Data infrastructure is the difference between the two, and almost no one is building it.
Read next. Part 4 · Personal brain covers source-backed memory, document lineage, and versioned artifacts (CLAUDE.md and AGENTS.md files, READMEs, and the self-improving loop). Part 6 · Institutional brain covers permission-aware retrieval, approvals, and cross-brain audit. Part 5 · Managed architecture shows how orchestration and approval gates run in a managed agent. And the companion research piece, The Autonomous Organization, takes the same log of what the agent saw and did and treats it as the input that lets an organization improve itself. Governance as data and the autonomous organization are the same claim from two directions.
More from our research.
Working on the same question?
We look for people to interview, data to compare against, and teams willing to test what we find. Leave a note and we will write back.