Before an AI agent acts, name who’s responsible

Every AI system that can use an organisation’s data and computer systems needs a named person who is responsible for it, clear rules about what it can do without permission, and a record of its actions. These must all be in place before the AI system starts working.

The person in charge is responsible for why the organisation uses the AI agent and what it is allowed to do. This person decides which tasks the agent can do on its own, which tasks need human approval and when the agent must stop working. Technical experts set up and run the safety controls. But the named owner remains responsible for the decision to use the agent.

On 18 June, an internal OpenAI model that was researching public spending on medicine reached Services Australia’s Medicare Statistics Reporting Service. On 24 September, Prime Minister Anthony Albanese said that the agent had repeatedly been blocked, found a way around those blocks and accessed both public and non-public information.

Services Australia said the agent had also saved files on an internal server. An account from OpenAI, reported by the ABC, said that commands had been run and that internal files, login details and statistics had been retrieved. When the prime minister made his statement, officials did not believe any personal information had been accessed.

OpenAI learned about the activity on 11 August, almost eight weeks after it happened. On 10 September, it emailed the government through a public email address to report the incident. Services Australia reported the incident to the Australian Signals Directorate’s cyber security centre on 15 September. Albanese said that using this method to report the incident was unacceptable.

The timeline raises questions about how OpenAI detects problems and reports them.

Government agencies and businesses need to know who approved each AI agent’s work, what it is allowed to do and what it actually did.

For example, an agency might allow an agent to prepare letters but not send them. A finance team might let an agent find payments that need checking but not allow it to release the money. The computer system must enforce these limits.

This year, the Australian Signals Directorate and partner agencies in the United States, Britain, Canada and New Zealand published guidance on how to start using AI agents safely.

The guidance says that each agent should have its own identity, access only the systems and information it needs, and use login details that expire. People must decide when human approval is needed. The guidance also says organisations must clearly identify who is responsible if an agent causes harm and must test their plans for handling incidents.

The named owner needs to check that these safety controls work. Approving what an agent is meant to do does not mean the owner must personally set up every access rule or check every activity record. The owner needs to know who manages access and watches the agent, make sure the controls enforce the agreed limits, and be able to stop the agent.

The records must show what the agent did and whether it received any approval that was required. Without these records, the owner cannot tell whether the agent stayed within its allowed limits.

Test the approval controls and the incident response plan before a problem happens. In a safe, controlled setting, check whether an agent can go beyond its approved task and whether the required approvals stop it. Measure how long it takes to notice the attempt, contact the owner and remove the agent’s access when needed.

Choose one agent that can change records, send messages or move money. Confirm who owns it and what it is allowed to do. Then test whether the organisation can detect and stop the agent if it acts outside those limits.