Businesses are getting excited about AI agents.
The issue is that many are not ready for what happens when software starts making decisions, triggering workflows, moving data or taking action across systems.
The Warning Signal from the Bank of England
The Bank of England has flagged that agentic AI may call for new rules across financial services. Payments, trading and operational resilience all sit in scope. Sarah Breeden, the Bank’s Deputy Governor for Financial Stability, made a sharp point: traditional human oversight may not stretch far enough for systems that act on their own.
That matters right now.
Companies are rushing to adopt AI agents, and the technology is moving faster than the controls around it. Software that decides and acts is a different kind of risk to software that only advises.
Wall Street has already lived through a version of this warning.
In 2012, Knight Capital deployed new trading software with a dormant piece of old code still buried inside it. When markets opened, that code activated and began firing off erroneous orders across 154 stocks.
There was no kill switch. It took the firm 45 minutes to identify the problem and shut the system down – by which point Knight had lost $440 million, nearly three times its annual earnings, and was forced into an emergency rescue financing deal just to survive.
What Changes When Software Starts Acting on Its Own
Agentic AI moves past answering questions. It triggers workflows, moves data and takes action across connected systems. A model that once drafted a payment instruction can now send it.
That shift changes the security question. When AI agents act, identity, access and control become the sharp edge.
As Databricks argues, scale does not come from getting approvals – it comes from operating AI safely on an ongoing basis, and that takes security, risk and business teams working together.
There is a quieter risk too. Autonomy erodes human decision-making, one nudge at a time. Human autonomy has to be treated as a design property that governance has to protect – not an afterthought.
Bigger prompts do not fix any of this.
You cannot write your way to safety inside a text box. Control has to be built into the system, not typed into the instructions.
Governance Is the Strategy, Not the Paperwork
Real governance is not policies, dashboards and risk councils. It is knowing who has the authority to shut a misbehaving agent down. MIT Sloan Management Review frames it plainly: leaders at every large company will tell you they govern their AI, yet most cannot name the person who can switch a harmful model off.
Adobe is a useful real-world example of what closing that gap actually looks like.
The company built a federated governance model with named owners for every AI system and a centralised steering committee, with escalation authority, reporting into the trust and security organisation, not the product team.
Building AI Autonomy You Can Trust
The path forward is practical. A few steps make autonomy accountable.
Define clear ownership and a real off switch before agents touch live payments, customer data or trading workflows. Someone must own the decision to stop, and that off switch has to work. Knight Capital and Adobe sit at opposite ends of this lesson: one lost $440 million because no one could act fast enough when a system went wrong; the other built the organizational structure in advance so that someone always can.
Build security and oversight into the architecture. Identity and access controls belong in the system itself, so autonomy stays accountable at every step rather than at a single review meeting.
Treat governance as an operational requirement that enables AI value, not a compliance box to tick. Strong controls are what let a business run agents in production and keep trusting the output.
The takeaway is simple.
Businesses ready for agentic AI govern it well. They do not simply prompt it harder.
