Blog

Overcoming the Risks of AI in Healthcare to Protect Patients and Digital Assets

by iSphere on May 19, 2025 in Blog, Healthcare IT Solutions

“I will do no harm or injustice.” Those seven words from the Hippocratic Oath shape the actions of everyone in the healthcare profession. From a C-suite or administrative standpoint, that line of thinking applies not only to patient care, but also the execution of digital healthcare trends.

There’s no better example than artificial intelligence.

Few technologies present the risks and rewards of AI. Patient intake chatbots, AI diagnosis, clinical decision support, AI transcription, and an abundance of other use cases make this tool enticing. Yet the fervor of breaking new ground sometimes imperils healthcare organizations.

The good news is that many of the risks of AI in healthcare can be mitigated with appropriate action and preparation. Here are four potential pitfalls that can arise without proper planning and some strategies that can increase your ability to enhance care without compromising trust or compliance.

Security Vulnerabilities

Healthcare providers need to be a hard target for hackers. The 734 large data breaches and 276,775,457 records breached last year show that cybercriminals have numerous attack vectors. Unfortunately, poorly trained, designed, or implemented AI tools can create another potential security gap.

A recent Wired story uncovered a major vulnerability created by AI coding tools. When auditing 16 large language models (LLMs), researchers found that 440,000 of the package dependencies (i.e. external components software projects need to function) were AI hallucinations. If developers failed to remove fictional dependencies from the code, hackers could poison software by inserting their own malicious packages at the source.

That’s just one potential threat. From a healthcare perspective, sloppy usage of AI by in-house development teams or even heedless EHR system vendors could result in compromised systems or zero-day backdoors.

Healthcare providers can avoid these vulnerabilities by hiring DevSecOps professionals who take a holistic approach. By prioritizing application and infrastructure security from the start, the industry can create developer-friendly guardrails as well as test, QA, and evaluate applications after the launch. Treating security as a continuous process can catch any AI-generated issues before they result in an expensive breach.

Biased Algorithms

AI decision-making systems aren’t without prejudices. The accuracy, completeness, diversity, and consistency of the training data can and will shape their responses and recommendations. Typically, these biases tend to reinforce existing inequities, resulting in predictions and care outcomes that fall short of individual needs.

AI bias can occur for a variety of reasons, including:

Underrepresentation of certain demographics during data collection
Failure to account for social determinants and their impact on healthcare
Overreliance on historical data reflecting existing systemic inequities
Inclusion of human bias during data annotation
Overreliance on proxy variables that correlate with race, income, or geography

These biases and blind spots will not sort themselves out. Healthcare organizations need to act to remedy and overcome them, otherwise they’ll impair outcomes.

Proper auditing and monitoring of AI model outputs are essential pillars. Often, experts describe this as keeping a human in the loop, using human intuition and judgment to correct AI biases or hallucinations. Including the perspectives of those who consider social determinants and the circumstances facing underrepresented groups can prevent biased analyses and predictions.

Additionally, it’s important for healthcare organizations to make sure that any research used to fuel AI decision-making is composed of a diverse and extensive sample size. Johns Hopkins found that 75% of participants in clinical research studies are white, noting that white people only account for 60% of the U.S. population. Building balance into sample data can reduce bias and improve overall healthcare outcomes.

Regulatory Noncompliance

The potential to improve diagnoses and workflows makes AI appealing, but there are risks to using this technology in a way that does not comply with healthcare industry regulations. Patient privacy and security rights must be prioritized in any instance where artificial intelligence is used, or your organization can risk massive fines.

HIPAA compliance is one of the largest hurdles to overcome. Federal law requires that covered entities, as well as any of the business associates they engage to carry out healthcare functions and activities, stay HIPAA compliant. As a result, generative AI tools like ChatGPT are off limits for tasks that use ePHI. However, the tool can be used to summarize text or streamline appointment scheduling, patient triage, or appointment reminders.

Some AI tools might be able to use ePHI, but they must first de-identify the training data to prevent potential violations. More than just redaction, healthcare companies and vendors can consider aggregating the data or even replacing sensitive data with non-sensitive substitutes (known as masking).

Even for tools and vendors that offer approved AI use cases, it’s important to verify that their processes and security protocols remain compliant with regulations. Ask any potential vendor questions like these:

Do they have a business associate agreement?
What data encryption methods do they use?
How are they de-identifying patient information?
How is data shared across their organization and networks?

These and other questions can get to the heart of whether or not a vendor will keep your patients protected and your business compliant with regulations.

Cognitive Offloading

Some medical professionals might be getting too comfortable with artificial intelligence. Studies of how the University of Pennsylvania Health System deployed AI to predict cancer patients’ deaths and prompt oncologists to have end-of-life conversations show what can go wrong. The algorithm degraded by 7% in prediction quality during COVID, failing hundreds of times to prompt clinicians to start these discussions, many of which didn’t end up happening.

Behavior like this is all too common. Part of the problem is humans automatically defer to machines, treating technology as objective or inerrant. One classic experiment showed that most human participants would ignore their own intuition in an emergency if a robot told them to, even if they observed the robot struggle with navigation tasks and lead them to a dark room with no discernible exit. Though this is an extreme form of cognitive offloading, AI can encourage people to relinquish some critical thinking.

How do we discourage people from becoming over-reliant on mistaken AI output? One way is to regularly audit the tools we implement. AI models can degrade in quality over time, so it’s important to monitor them. This means setting baselines, creating feedback loops, and actually checking whether your model is still doing what it was designed to do.

Additionally, it’s important to keep human subject matter experts involved in regular reviews to scrutinize AI output. These SMEs should not only be masters of the skillsets AI replaces but also firmly understand AI fundamentals.

When a person understands how these systems process data and generate responses, they can make better conjectures about the root of any discrepancies. Be it user error or AI hallucination, people will be needed to remedy the issue.

Next Steps to Enhance Your AI Usage

AI has the potential to revolutionize healthcare—but only if it’s deployed responsibly. Though there are risks to using AI in healthcare, you don’t need to avoid this technology. The key is creating an internal culture of verification, audits, and cybersecurity best practices, as well as seeking out IT staffing and solutions companies that prioritize the same security, accuracy, and compliance.

iSphere partners with healthcare organizations to implement AI responsibly, offering expertise in cybersecurity, data analytics, enterprise application integration, and IT project management. Whether you need to strengthen digital defenses, streamline EHR systems, or manage complex AI rollouts, our team delivers the strategy and talent to protect patients, support staff, and keep innovation aligned with compliance.

Together, we can build a future where innovation not only does no harm but also elevates the quality of life for the patients you serve.