Hospitals' AI Committees Ask the Wrong Privacy Questions, Missing Key Risks

The rapid adoption of artificial intelligence (AI) in healthcare is transforming how we diagnose and treat patients. Hospitals are increasingly forming AI committees to navigate the complex landscape of integrating these technologies while ensuring patient privacy. However, a growing concern is that these committees are often asking the wrong questions when it comes to privacy risks.

Too frequently, the review process begins and ends with one question: Was protected health information (PHI) used to train the AI model? While this is a crucial starting point, it is far from sufficient. Consider two types of models: a large language model fine-tuned on clinical notes and a computer-vision model trained to return a blur mask or a binary "in body" or "out of body" label. Both may have been developed using PHI, but their potential for exposing patient information varies significantly.

Under the Health Insurance Portability and Accountability Act (HIPAA), business associates can only use PHI as permitted by their agreements with covered entities and must apply appropriate safeguards. This is the legal baseline, but it should not be the end of the privacy analysis. The more critical question is: What realistic pathways exist for the deployed model to expose information about a patient?

One-Size-Fits-All Reviews Fall Short

The problem with a one-size-fits-all review process goes beyond unnecessary paperwork; it can make hospitals less effective at identifying both high-risk and low-risk AI models. For generative models, which can accept arbitrary prompts and generate open-ended responses, a generic checklist can create a false sense of security. A committee might confirm that a business associate agreement is in place, data is encrypted, and access is logged, but fail to examine the model's most meaningful exposure pathways.

These pathways include whether users can enter arbitrary prompts, whether the model can be queried repeatedly and adaptively, whether it retrieves information from live clinical records, whether prompts and outputs are retained, and whether it can reproduce information from its training or retrieval context. Each of these factors can significantly increase the risk of exposing patient data.

On the other hand, a narrow model with a fixed technical task-such as a closed classification or segmentation model-may have no prompt interface, no generative capability, no exposed model weights or embeddings, and only constrained outputs like labels, timestamps, bounding boxes, or redaction masks. Requiring such a model to undergo the same privacy review as an open-ended generative model ignores its architecture and can lead to over-governance.

This approach not only wastes resources but can also increase privacy exposure. For example, a face-detection model must learn from images containing faces, just as an out-of-body detector for minimally invasive video must see the identifiable frames it is intended to flag. Blocking controlled training on PHI can reduce redaction accuracy and preserve manual workflows in which more people view, handle, and retain identifiable data.

The Bigger Picture

The stakes are high. According to a recent analysis of case studies across 12 domains in healthcare, machine learning is expected to hold the majority share in the AI market by 2023. This underscores the urgent need for robust, nuanced privacy reviews that go beyond the basic legal requirements.

Public trust in AI is a critical factor. A survey found that 53% of American adults feel they have little to no control over the use of artificial intelligence in their healthcare. This lack of trust can hinder the adoption of potentially life-saving technologies. Hospitals must balance innovation with transparency and accountability to maintain patient confidence.

To address these challenges, AI committees should adopt a more nuanced approach to privacy reviews. They need to consider the specific architecture and functionality of each model, not just whether PHI was used in training. This involves evaluating how the model is deployed, who can access it, and what data it interacts with on an ongoing basis.

By focusing on these critical exposure pathways, hospitals can better protect patient privacy while harnessing the full potential of AI to improve healthcare outcomes. The goal should be to create a framework that is both rigorous and flexible, ensuring that each model receives the appropriate level of scrutiny based on its unique risks and benefits.