When AI Agents Go Off Script: A New Kind of Cyber Security Risk

Artificial intelligence is quickly moving beyond simply answering questions. For the last few years, most businesses have experienced AI through tools such as ChatGPT and Microsoft Copilot: ask a question, provide some information and receive an answer.

The next generation of AI is different. AI agents are designed not only to provide information, but to take actions on our behalf. Rather than simply drafting an email, an agent could potentially send it. Instead of explaining how to update a CRM record, it could make the change itself, retrieve information from other systems, create documents, manage workflows or complete several steps towards a larger objective.

For businesses looking to improve productivity, the opportunity is significant. But giving AI the ability to act also changes the cyber security conversation.

That became particularly clear following an unusual incident disclosed by OpenAI this summer. During internal cyber security evaluations, AI models circumvented controls designed to isolate them from the internet, exploited vulnerabilities within shared infrastructure and ultimately accessed systems belonging to AI platform Hugging Face.

The models were operating in a deliberately challenging test environment with reduced safeguards, rather than within normal customer-facing products. Even so, the incident provided an interesting glimpse into what increasingly capable autonomous AI systems can potentially do when they are given access to tools and infrastructure.

The UK’s National Cyber Security Centre has also highlighted recent examples of agentic AI systems carrying out unintended or unsanctioned activity. Its message isn’t that businesses should avoid AI, but that organisations need to think carefully about how autonomous systems are deployed, what they can access and what controls exist if something doesn’t behave as expected.

Until now, much of the conversation around AI has focused on whether we can trust the information it produces. As agents become capable of interacting directly with business systems, organisations increasingly need to consider another question: what happens when AI is allowed to act?

From AI assistants to AI agents

Most organisations currently use AI as an assistant. Someone might ask Microsoft Copilot to summarise a Teams meeting, analyse an Excel spreadsheet, create a presentation or draft an email. The AI produces something useful, but a person usually reviews the result and decides what happens next.

AI agents introduce another level of autonomy. Imagine asking a system to find every customer whose software agreement expires during the next 90 days, identify their account manager, prepare a renewal email, create a follow-up task and update the CRM. Rather than helping with each stage individually, an agent could potentially work through the entire process itself.

That distinction matters from a security perspective. To complete a workflow like this, an AI agent may require access to customer records, Microsoft 365, your CRM and other business applications. It may also need permission not simply to view information, but to change records or communicate externally.

Every additional connection therefore increases what an agent is capable of achieving. At the same time, it increases the potential consequences if the system makes the wrong decision, interprets an instruction incorrectly or encounters something malicious.

Why does agentic AI create a different cyber risk?

Many of the underlying risks associated with AI agents aren’t actually new. Businesses have always needed to think carefully about access controls, data security, application permissions, monitoring and incident response.

What changes with AI agents is the combination of access, autonomy and speed.

If an employee accidentally sends an incorrect document to a customer, the mistake might affect a single email. An automated process making the same incorrect decision could potentially affect hundreds of customers or records before anybody notices. Automation doesn’t necessarily make mistakes more likely, but it can dramatically increase their scale and impact.

There is also an element of unpredictability. Conventional software normally follows a clearly defined sequence of instructions, whereas an AI agent may be given an objective and determine how best to achieve it. If that objective is vague, or the agent has access to more systems than it actually needs, it could take a route its user never intended.

That is why the amount of autonomy given to an AI system matters. An assistant suggesting an email response for a person to review presents a very different risk from an autonomous system capable of changing live business data or communicating externally without human involvement.

The greater the potential impact of something going wrong, the stronger the controls around the system need to be.

Permissions could become one of the biggest AI security questions

One of the most established principles in cyber security is least privilege. Put simply, people and systems should only receive the access they genuinely need to perform their role.

If somebody only needs to read a document, they don’t necessarily need permission to delete it. If an application needs access to calendar information, it shouldn’t automatically receive access to every file and mailbox across the business.

The same principle should apply to AI.

Suppose an agent is being used to analyse opportunities within your CRM. It may need permission to read customer records and perhaps update a small number of fields, but it probably doesn’t need permission to delete those records. It certainly shouldn’t automatically receive access to unrelated HR, payroll or financial information.

External communication deserves similar consideration. There may be situations where allowing an AI agent to send messages automatically makes sense, but that should be a conscious decision based on the process and level of risk involved rather than something enabled simply because the technology allows it.

As businesses experiment with more advanced AI, asking “What is the minimum access this agent actually needs?” should become a standard part of the implementation process.

Human oversight still has an important role

This links closely to something discussed during our recent Microsoft: Automating Your Business with Copilot webinar. Matt Elson was joined by Sebastian Lourenço, AI Business Solution Specialist at Giacom, to explore how businesses can move beyond simply using AI as a chatbot and start applying Copilot to genuine day-to-day processes.

One of the most useful themes from the session was understanding what not to automate.

There are plenty of relatively low-risk activities where AI can save considerable time, including summarising meetings, extracting information, preparing documents, analysing spreadsheets, categorising content and producing first drafts.

The calculation becomes different when a process involves sensitive information, customers, money or decisions with significant consequences. Using Copilot to create the first draft of an internal meeting summary presents relatively little risk. Allowing an autonomous system to amend customer contracts without human review clearly presents considerably more.

Both involve AI, but they shouldn’t require the same level of oversight.

The goal therefore isn’t necessarily to remove people from every process. In many cases, the most effective automation will combine the speed and efficiency of AI with a sensible human approval point at the stage where judgement really matters.

Five things businesses should consider before giving AI more autonomy

Businesses don’t necessarily need a huge AI governance programme before they begin experimenting. A few sensible principles can make a significant difference.

1. Understand how AI is already being used.
Employees may already be using Microsoft Copilot, ChatGPT, meeting assistants, AI functionality inside CRM platforms or features built into other software. Before deciding what your AI policy should look like, understand what’s already happening across the organisation.

2. Control what AI can access.
Different information carries very different levels of risk. Asking an AI tool to work with public marketing material is not the same as giving it access to customer information, financial records or confidential company data. Permissions should reflect both the sensitivity of the information and the task the system actually needs to perform.

3. Keep humans involved where the consequences are higher.
An AI system can prepare an email while a person still approves it before it is sent. Similarly, an agent might identify a recommended change within a business system without automatically making that change. A small approval step can significantly reduce the potential impact of an incorrect decision.

4. Monitor what automated systems are doing.
As agents begin interacting with more applications, organisations need visibility over their activity. Ideally, you should be able to understand what information an agent accessed, what actions it performed and whether anything unusual occurred.

5. Know how to stop the process.
If an automated workflow starts behaving unexpectedly, somebody needs to know how to disable it, who is responsible for investigating the issue and which systems may have been affected. That is much easier to establish before an incident than during one.

None of these controls are intended to stop businesses benefiting from AI. In many respects, they’re simply an extension of the approach organisations already take when introducing any technology with access to important company systems and data.

Start with the process, not the technology

With impressive new AI demonstrations appearing almost every week, it’s easy to start with the technology and then look for something to automate.

A more effective approach is to begin with the business problem.

Before introducing an agent, look carefully at how the process works today. Where is the team spending unnecessary time? Which stages are repetitive? What information is required? Where does somebody genuinely need to apply judgement, and what happens when something unusual occurs?

Once that process is understood, it becomes much easier to identify the parts where AI could make a meaningful difference.

This was another important takeaway from our Copilot webinar. The goal shouldn’t be to automate something simply because you can. Start with a repetitive process that genuinely costs the business time, understand how it currently works and then identify where AI could make it faster, easier or more accurate.

That approach usually leads to better automation and also makes the security considerations much easier to understand.

The productivity opportunity is still enormous

The conversation around security shouldn’t distract from the potential benefits of AI.

During our webinar, Sebastian demonstrated practical Copilot examples across familiar Microsoft applications including Outlook, Teams, Word and Excel, alongside more advanced agents capable of working across different business systems.

He also highlighted an average saving of around 26 minutes per working day from standard Copilot usage, equivalent to approximately nine hours per month. Across an organisation, even relatively modest savings like that can quickly become significant.

More advanced agents could take those efficiencies much further. Tasks that currently involve opening an email, checking a spreadsheet, copying information into a CRM and creating a follow-up task could increasingly be connected into a single workflow.

The biggest benefit, however, may not come from simply automating as much as possible. The businesses that gain the most from AI are likely to be those that understand where it genuinely adds value and put the right foundations around it from the beginning.

Strong IT foundations matter even more in the AI era

AI increasingly sits within the technology businesses already use every day. Microsoft Copilot operates across Microsoft 365, while newer agents can potentially interact with Teams, Outlook, SharePoint, cloud services and other business applications.

As a result, conversations about AI adoption are increasingly becoming conversations about identity management, permissions, data governance and cyber security too.

If your Microsoft 365 environment is well managed, access is controlled and you understand where important company information is stored, introducing AI becomes considerably easier. If permissions are already inconsistent, former employees still have access to systems or nobody is entirely sure who can see particular data, giving an AI agent access to those environments can compound the problem.

In that sense, AI strategy and IT strategy are becoming increasingly difficult to separate. Businesses with strong technology foundations will be in a much better position to take advantage of new AI capabilities as they develop.

Using AI safely doesn’t mean slowing down

The lesson from recent AI security incidents shouldn’t be that businesses need to avoid autonomous AI. Agentic systems are likely to create significant opportunities, particularly when they are applied to repetitive and well-understood processes.

What changes is the level of responsibility required when AI moves from simply providing information to actively taking actions. The more access and autonomy a system receives, the more carefully its permissions, monitoring and oversight need to be considered.

The sensible approach is therefore to start with a process you understand, give the AI only the access it genuinely needs and keep people involved where decisions carry greater consequences. Combine that with strong cyber security and well-managed IT foundations, and businesses can experiment with AI without giving up control of the systems and information they rely on.

Because increasingly, the question isn’t whether organisations will use AI agents. It’s how they can use them safely, securely and effectively.

Want to explore what AI could do for your business?

Microsoft Copilot and the latest generation of AI tools can deliver genuine productivity improvements, but successful adoption isn’t simply about purchasing licences or switching new features on.

Your Microsoft 365 environment, data, security, permissions and wider IT infrastructure all influence what you can automate and how safely you can do it.

INDIGO IT can help you identify practical opportunities to use AI within your business while making sure the right technology and security foundations are in place.