AI Agent Governance for Software Teams: A Practical Framework

Your team added a coding agent in the spring. Now there are four: one writing code, one triaging bugs, one drafting tickets from meetings, and one somebody set up last week that nobody quite remembers approving. Each one is useful. Together, they change things faster than anyone is checking.

AI agent governance is the set of rules that decide what your AI agents may do on their own, what needs a person to approve, and how every change is recorded so it can be traced and undone. For a software team, that comes down to six things: know which agents can change what, tier actions by how hard they are to undo, require a source for every change, make every change reversible, give agents the least access that works, and watch whether review is getting lighter or heavier.

Most guides to AI agent governance are written for security teams at large companies: identity, data loss, model risk. That matters. This guide is about the day-to-day version a software team needs: who’s allowed to change a ticket’s priority, close an issue, or tell a customer something, when the one doing it isn’t a person.

Governance vs. guardrails

People use the two words interchangeably, but they do different jobs.

• Guardrails stop a single action from going wrong: a blocked command, a read-only token, a confirmation prompt before a destructive tool runs.
• Governance decides the rules those guardrails enforce, and checks they’re still right as you add agents: which actions need a person, who that person is, and how you’ll know if something slipped through.

You need both. Guardrails without governance means every agent has a different, accidental set of rules. Governance without guardrails means rules in a doc that nothing enforces.

The framework

1. Know which agents can change what

Start with an inventory. For each agent, write down which tools it connects to and whether it can read, write, or both. A coding agent with the GitHub MCP server and a write token can open pull requests, comment, and close issues. The same agent with a read-only connection can only look.

If you can’t fill this in, that’s the first finding. Most teams discover at least one agent with more write access than anyone intended.

2. Tier actions by how hard they are to undo

Not every change needs a person. Requiring approval for everything just trains people to click “approve” without reading. Sort actions by two questions: how easy is this to undo, and does it create a commitment someone else will rely on?

Write the tiers down, per action, not per agent. “The triage agent can’t change priority” is easier to enforce and easier to explain than “the triage agent is trusted at level 2.”

3. Require a source for every change

Every change an agent makes should say where it came from: which meeting, message, document, or issue it’s based on, and who said it. “Moved to In Review” is a fact. “Moved to In Review because Priya said in #platform on Tuesday that the API change is done” is something a person can check in ten seconds.

Sourcing is also what makes review fast. A reviewer who can see the source doesn’t have to reconstruct the agent’s reasoning. A reviewer who can’t has to choose between trusting it blindly and redoing the work.

4. Make every change reversible

Agents will get things wrong. The question is whether fixing it takes one click or an afternoon. For every change, keep:

• what it looked like before
• what it looks like now
• which agent did it, and who approved it
• a way to put it back

That record is your audit trail. When someone asks “how do you audit AI agent activity?”, this is the answer: not a separate logging project, but a before-and-after on every change, kept where the work lives. We wrote more about what undo should look like for an AI-made change.

5. Give agents the least access that works

Start every agent with read-only access and add write access one tool at a time, when there’s a clear reason. Most MCP servers make this easy:

• Linear’s MCP server has a read-only address.
• GitHub’s MCP server has a read-only mode, and its tools are split into toolsets you can switch on one by one.
• Fine-grained tokens can be limited to specific repositories.

Our list of the best MCP servers for engineering teams notes which ones have read-only options, and the GitHub MCP server guide shows how to turn them on.

6. Watch whether review is getting lighter or heavier

This is the check most frameworks skip. If people are reviewing more agent changes each month, not fewer, something is off. Either the tiers are too strict, or the agents aren’t earning trust.

Once a month, look at three numbers:

• how many changes went through automatically
• how many were approved by a person
• how many were rejected or undone

If rejections are near zero for an action, it’s a candidate to move down a tier. If undos are climbing, move it up.

A one-page AI agent governance policy

If you want something to paste into your team’s wiki, this is enough to start:

AI agent governance policy: [team name]

1. Inventory: every agent, the tools it connects to, read or write. Owner: [name]. Reviewed monthly.
2. Tiers:
   - Automatic: labels, links, typo fixes.
   - Person approves: status, scope, cycle moves.
   - Person only: priority, ownership, closing work, customer-facing, deletes, production.
3. Every change records its source (meeting, message, doc, or issue) and who said it.
4. Every change keeps a before and after, and can be undone.
5. Agents start read-only. Write access is added per tool, with a reason.
6. Monthly check: auto / approved / rejected / undone counts. Adjust tiers.

Where Thor fits

Thor is built around this framework for one specific job: keeping a software team’s record of the work current. Thor is the context infrastructure for AI-native software teams. It keeps one current record of what your team decided across Slack, GitHub, Linear, and meetings, and it holds itself to the same rules this guide describes:

• Sourced: every update Thor proposes links back to the meeting, message, or doc it came from.
• Reversible: changes aren’t silent overwrites. You can see what changed and undo it.
• Governed: Thor suggests, and people decide what becomes official. Small, easy-to-undo updates can go through under clear rules. Anything that sets a real commitment, like an owner, a priority change, or a promise to a customer, waits for a person.

We’ve written more about what “human in the loop” should actually mean when agents move this fast.

Thor doesn’t replace your security team’s agent controls: identity, data access, and model risk still need their own tools. The NIST AI Risk Management Framework is a good starting point for that broader layer. This framework covers the part closest to the work.

FAQ

What is AI agent governance?

The rules that decide what AI agents can do on their own, what needs a person to approve, and how every change is recorded so it can be traced and undone. For software teams, it covers things like who can change a ticket’s priority, close an issue, or message a customer when the one doing it is an agent.

What’s the difference between AI agent governance and guardrails?

Guardrails stop a single action from going wrong, like a read-only token or a confirmation prompt. Governance decides what those guardrails should be, who approves what, and checks the rules still fit as you add agents.

How do you audit AI agent activity?

Keep a before-and-after record for every change an agent makes: what changed, which agent did it, who approved it, and the source it was based on. Store it where the work lives, so anyone can trace or undo a change without a separate investigation.

Which agent actions should always need a person?

Anything that creates a commitment or is hard to reverse: changing priority or ownership, closing work as done, anything customer-facing, deleting data, and deploying to production. Small, easy-to-undo changes like labels and links can usually run automatically, as long as they’re logged.

How many agents can a team run before it needs a governance framework?

There’s no fixed number. The signal is review load: once people can’t keep up with what agents are changing, or nobody can say which agent has write access to what, it’s time. For most teams that’s earlier than they expect, often the second or third agent with write access.

See how Thor proposes updates with a source attached, and waits for a person on anything that matters. Book a demo.