Developers, not doctors, should be held accountable for generative AI medical devices, providers say

Hospitals, health systems, doctors and telehealth companies are asking the government not to hold healthcare providers responsible for the safety and performance of medical devices that use generative AI. They argue that the companies that develop these tools should be responsible instead.

In comments sent to the Food and Drug Administration (FDA), the American Hospital Association, American Medical Association and American Telemedicine Association said that asking providers to monitor devices that use generative AI, including chatbots and tools that combine medical information, would put too much pressure on them. Small and rural medical practices could face an especially heavy burden.

Instead, developers should be responsible for monitoring AI devices used in healthcare, and the FDA should create ways for providers to report problems with these devices.

"Manufacturers should remain responsible for the safety and performance of the devices they sell," said ATA Action, the lobbying group for the American Telemedicine Association, in its comments.

As AI tools improve very quickly, the healthcare industry is trying to work out who should be responsible when these tools make mistakes. If providers take on that responsibility, they could face more malpractice lawsuits. Technology companies could also face lawsuits for failing to take proper care. Neither side wants to be held responsible for the tools' mistakes.

Doctors say one problem is that they cannot clearly see how AI models work. Hospitals told the FDA that health systems usually have established processes for reviewing devices that use AI. But without a clear understanding of how the models work, providers may struggle to find problems in them.

"While hospitals and health systems support evaluation and monitoring activities, vendors and developers must be responsible for the ongoing performance, maintenance and integrity of the tools they market," the AHA said.

The comments responded to the FDA's request for feedback from the industry on its proposed plan to regulate medical devices that use generative AI. These devices have features that make them different from other AI tools. For example, they can accept open-ended questions and instructions and may give different answers to similar inputs, the FDA said. Several industry groups, including the AHA, AMA, ATA and Workgroup for Electronic Data Interchange, sent comments to the agency.

Healthcare groups question whether human supervision is enough and ask for AI labels

On other parts of the FDA's plan, healthcare provider groups generally agreed with the agency, including its proposal to judge the risks of medical devices that use generative AI.

The FDA proposes judging risk based on what a device does. For example, a device that provides information that does not require immediate action, such as predicting a future heart or blood vessel problem, would be considered low-risk because its possible effects are limited. But a device that makes decisions entirely on its own could have serious consequences.

Still, some groups suggested changes to the plan.

The AHA said that looking only at what a device does may not be enough to judge its risk. Some tasks can create several kinds of danger. For example, an AI tool that recommends emergency care might cause people with non-emergency conditions to seek treatment they do not need. It might also fail to tell someone who needs emergency care to get help.

The plan should consider more factors, such as how likely a particular device is to make mistakes or cause harm. It should not judge risk only by what the tool does, such as whether it simply provides information, leads to an action or works under the supervision of a human doctor.

The AMA also made recommendations about human supervision. It said that keeping a person involved in decisions, a common safety measure for AI tools, does not always reduce risk. Doctors may make mistakes when checking AI suggestions. They may also rely too heavily on the tools and become less able to supervise them properly. For this reason, the AMA said the risk assessment should also consider the people supervising the AI.

The FDA's discussion paper also explains ways to test these devices before they reach the market and monitor them after they become available.

The Workgroup for Electronic Data Interchange (WEDI) also suggested that the FDA create a system for reporting harmful events and develop "nutrition" labels. These labels would help patients and doctors understand what an AI tool does and what risks it may pose.

This is not the first time someone has proposed nutrition-style labels for AI tools used in healthcare. The Biden administration had issued a rule requiring developers to create these kinds of labels, but the Trump administration withdrew the rule last year.

Trust in AI used in healthcare is important for getting people to use it. Doctors say they want AI tools that have been checked by a trusted organization and monitored regularly. They also want clear rules about who is legally responsible when something goes wrong and stronger oversight of AI medical devices by government agencies such as the FDA.

Doctors may get what they want soon. Rick Abramson, director of the FDA's Digital Health Center of Excellence, said at a conference earlier this week that the agency plans to release draft guidance on how it will regulate medical devices that use generative AI in the coming months.