Jekyll and Hyde: getting the AI balance right

Anthropic is investing $1 billion in having independent experts work closely with its AI teams to check its technology. Here is what that could mean for healthcare leaders in Australia.

Anthropic's founder thinks a lot about the two very different futures AI could bring. Could AI cause a disaster like Judgement Day in the Terminator movies? Or could it create the future Elon Musk predicts, where people no longer need to work or use money?

So it is not surprising that Anthropic, the company that makes Claude, is one of the first companies to bring independent AI safety experts inside its operations. What stands out is how much money Anthropic is putting into this work. It shows how seriously the company takes AI safety.

What can $1 billion spent on checking AI buy?

The full details are still being decided. But the main plan is for Anthropic to give a team of outside experts access similar to what its own employees have. These experts will test Anthropic's AI models. They will check whether the company follows its safety rules and promises. They will report problems and look for risks the company may have missed. The aim is to make sure Anthropic does not develop AI faster than it can understand and control it.

How should Australian healthcare organisations check and manage AI?

AI is being used more and more quickly in Australian healthcare, and there are good reasons for this. Take AI scribes, for example. These tools can listen during medical appointments and help write clinical notes. They promise to help with two big problems at the same time: healthcare workers feeling exhausted by their jobs and the need to get more work done. The idea is that healthcare workers can spend less time doing paperwork and have more time and energy to care for patients.

It is easy to understand why so many healthcare organisations are testing AI. Examples include Calvary Health Care, Monash Health and Northern Health. It also helps explain why the number of GPs using AI scribes has doubled in just over a year.

There has been progress in creating general rules and advice about using AI safely. But many healthcare organisations still do not have the practical systems they need to manage AI risks. Work is being done to fix this. A recent research paper written partly by Farah Magrabi explains a practical way for healthcare organisations to set rules, responsibilities and checks for AI.

But looking only at the risks of AI can also be a problem. In its March 2025 report, the Peterson Health Technology Institute's (PHTI) AI Taskforce pointed out an important problem with trials of AI scribes. Healthcare leaders in the United States do not have enough information to tell whether AI is actually providing the benefits it promises.

Are healthcare workers feeling less exhausted? Are they more likely to stay in their jobs? Are patients having better experiences? Does the time saved by AI help healthcare workers get more done or provide better care? And are these improvements worth the money spent?

Without answers to these questions, healthcare leaders will not have the evidence they need to decide whether to use AI more widely.

The PHTI Taskforce warns that if healthcare organisations do not have reliable ways to measure results, bringing in AI quickly could actually make healthcare more expensive overall.

That is why experts who check AI from inside an organisation need to focus on two things: whether AI is safe and whether it is useful. Their work needs to give healthcare leaders the evidence they need to decide which AI projects to expand, which ones to change and when to stop using them.

How would checking both AI safety and benefits work in real life?

We are organising a series of roundtable discussions to help create a system for checking and managing AI from within Australian healthcare organisations.

Who should take part?

Leaders of Australian healthcare organisations that have already started testing AI and now need to decide whether to use it more widely.

What we will discuss

• What does your AI trial need to prove before you decide to use it more widely?

• Do you have the right ways to measure results and enough information to judge the benefits, costs and risks?

• What regular checks and supervision need to be in place before you start using AI more widely?

We will share some of the latest research about AI's risks and benefits. We will listen to people's experiences and discuss how having experts regularly check AI could help healthcare leaders decide whether to expand its use.

If you are trying to decide whether to expand an AI trial, sign up for a roundtable discussion or send me an email.

Brendan Roche is a partner at HealthConsult, a management consulting company that specialises in helping organisations in healthcare, aged care and disability services.