As AI Agents Take on More Tasks in Healthcare, Experts Warn Oversight Must Keep Pace

Imagine handing a new employee the keys to your clinic on their first day, no training, no supervisor checking their work, just a hope that things go fine. That's roughly the risk healthcare organizations face as they deploy increasingly autonomous AI agents into workflows that touch real patients.

Ankit Jain, cofounder and CEO of Infinitus, raised this concern in a recent discussion with HIMSS TV, arguing that as AI agents gain more independence in healthcare settings, organizations need concrete safeguards to prevent unintended or unauthorized actions from affecting patient care. It's a warning that lands at a pivotal moment. AI agents are no longer just answering questions or summarizing notes. They're scheduling procedures, verifying insurance, communicating with patients, and in some cases making decisions that used to require a human's direct sign-off.

The distinction matters. A basic chatbot responds to a prompt and stops. An agent, by contrast, can take a goal, break it into steps, and execute those steps on its own, often across multiple systems, without a person approving each move. Think of the difference between a GPS that suggests a route and a self-driving car that actually takes the turns. The second version is far more useful. It's also far riskier if something goes wrong along the way.

The Autonomy Problem

Healthcare has spent decades building safety nets around human decision-making: peer review, credentialing, informed consent, malpractice accountability. Those systems assume a person is making the call and can be held responsible for it. AI agents complicate that picture. When software initiates an action, who checks it before it reaches a patient? Who catches the mistake if it doesn't?

Jain's comments point to a practical concern rather than a theoretical one. Unintended actions could mean an agent scheduling the wrong procedure, misinterpreting a patient's response during an automated call, or acting on outdated information it wasn't supposed to use. Unauthorized actions could mean an agent exceeding its intended scope, perhaps making a clinical recommendation it was never designed to offer, simply because nothing stopped it from trying.

Neither failure requires malice or even a bad algorithm. Both can happen quietly, at scale, before anyone notices. That's the nature of automation. It's efficient precisely because it doesn't wait for a human to double-check every step. But that same efficiency means errors can multiply just as fast as successes.

This isn't a fringe worry within the industry. Other sessions from the same HIMSS TV series, including discussions on federal oversight of healthcare AI and cyber resilience, reflect a broader pattern of experts converging on the same message: governance can't be an afterthought bolted onto AI deployment after the fact. It has to be built in from the start, with clear boundaries on what an agent can and cannot do without human review.

Some of this echoes lessons from other high-stakes industries that adopted automation early. Aviation didn't remove pilots when autopilot systems improved. It kept humans in the loop, with clear protocols for when automated systems hand control back to a person. Healthcare AI advocates are increasingly pointing to that model: automation as a partner that operates within defined limits, not a replacement that runs unchecked.

The challenge is that healthcare's stakes are personal in a way that route planning or flight paths aren't. A scheduling error is an inconvenience. A miscommunication during a medication reminder call, or an agent that acts on incomplete patient data, can turn into a genuine safety issue. That's why the calls for safeguards aren't just about preventing embarrassing glitches. They're about protecting people who often have no idea an AI system played any role in their care at all.

What would meaningful safeguards actually look like? Based on the themes emerging across this wave of industry commentary, a few principles keep surfacing. Agents need clearly defined scopes of action, so they can't wander into decisions they weren't built to make. Organizations need audit trails, so when something goes wrong, someone can trace exactly what the agent did and why. And there needs to be a human checkpoint for anything that directly affects clinical outcomes, not as a bottleneck, but as a backstop.

None of this is about slowing down innovation for its own sake. AI agents genuinely help stretched healthcare workforces manage administrative burden, faster insurance verification, quicker scheduling, more responsive patient communication. Infinitus itself operates in that space, building AI agents designed to handle time-consuming tasks like benefits verification, freeing up staff for higher-value work. The goal isn't to strip agents of their usefulness. It's to make sure that usefulness doesn't come at the cost of patient trust or safety.

Why It Matters

The quiet truth here is that most patients will never know an AI agent touched their care. They won't see the code, read the audit log, or understand the difference between a chatbot and a fully autonomous system. What they will notice is the outcome: whether their appointment was scheduled correctly, whether their insurance was verified accurately, whether the call they received made sense.

That invisibility is exactly why oversight matters so much right now. When systems fail quietly and at scale, the damage doesn't announce itself until it's already widespread. Building safeguards today, clear scopes, human checkpoints, real accountability, isn't about distrust of the technology. It's about respecting the fact that healthcare doesn't get a second chance to earn back a patient's trust once it's lost. As AI agents take on more responsibility across clinical and administrative workflows, the organizations that get ahead of this now will be the ones patients can still trust tomorrow.