Give 'Em an Inch... They Hack the Whole Universe

WHEN BAD BOTS BEHAVE BADLY... With self-delegating bots executing actions at machine speed, enterprises are debating the final shield they must deploy to stop rogue AI in its tracks, writes Shilpa Ranipeta.

Google's Gemini AI hacked three companies in May by simply accessing the internet, finding public information online and guessing credentials to access websites of those three companies. The incident, disclosed last week, is the first known case of a Google AI system autonomously hacking external systems. With similar incidents involving OpenAI, Anthropic and Meta models also emerging in recent months, the concern is shifting from what AI can find to what it can do once it gets access.

With AI agents now capable of accessing systems, moving across digital environments and communicating with other agents, a bigger question is emerging for enterprises: Are they prepared for software that can make decisions, delegate tasks, act on its own and, in some cases, end up attacking their systems?

For companies deploying or interacting with such agents, that means rethinking how decisions move from AI to action. An agent may be allowed to reason and decide what needs to be done, but it should not necessarily have the authority to execute that decision on its own.

The distinction becomes even more important as agents begin interacting with other agents across enterprise systems. Rajkumar Paulraj, VP, engineering and country head India at Redwood Software, says the execution layer should sit below the agentic layer, handling identity, permissions, policy and governance. "The final governance of whether the disbursement happens or not, still resides in the execution layer," he says.

In practice, that could mean an agent recommending a loan disbursement, accessing a database or changing a system can make the decision, but the execution layer decides whether that decision gets carried out, with human intervention for actions that warrant it. Permissions would also need to be limited by transaction, time and context rather than granted broadly. While a kill switch is an essential backstop when an agent goes rogue and performs actions outside the sandbox, enterprises will need to check an agent's decisions even before they become actions.

Shomiron Das Gupta, CEO and founder of AI enterprise telemetry platform Bloo, proposes a "dual-factor approval system" in which two independent systems validate a decision before execution. Even the kill switch, he argues, may be too slow to contain an autonomous system acting at machine speed. Then comes the question of whether enterprises can actually see what their agents are doing, which enterprises now do through logging. But while traditional logging will remain essential, every agent needs an identity, authorisation and an audit trail, but that may only show the surface of the problem.

"You need to go one step further to try and figure out what are the logic circuits that you're actually triggering off and why it took a decision," Das Gupta says.

The telemetry becomes even more important when several agents are involved. Without a detailed, immutable record of those interactions, reconstructing what happened and who authorised what becomes difficult. But how prepared are enterprises to put all of this in place? Experts say larger companies are already applying traditional information-security and risk principles to AI governance, but this has not necessarily reached smaller businesses.

Preparedness is also increasingly becoming a compliance question with companies unable to figure out what they have deployed within their systems.