AI-Based Medical Devices: How to Fail in 4 Easy Steps

You’re ready to create your first AI-based medical device. You have the idea, the data, and the right team of experts. But even with all that, success isn’t guaranteed.

In fact, you might already be just a couple of steps away from failing.

Note: this article focuses on predictive AI algorithms, not LLMs nor computer vision.

Step 1: Neglect Data Quality

The foundation of any AI model is the data it’s built on. Using incomplete, unclean, or biased training data can cripple your device’s reliability and lead to inaccurate predictions or recommendations. For example, if your training data lacks diversity, your model might underperform for certain demographic groups, potentially leading to inequitable care.

How to Succeed Instead:

• Invest in high-quality, representative datasets that reflect real-world variability.
• Prioritize data cleaning, annotation, and validation to ensure your model learns from accurate and relevant inputs.
• Regularly assess data quality to identify and address gaps early.

Step 2: Quickly Drop an Algorithm That Has Good Performance

Finding an algorithm with promising performance is an exciting step, but stopping there is a mistake. Without thoroughly examining its weaknesses, you risk deploying a model that overfits the training data, fails in real-life scenarios, or performs poorly for specific subgroups. A deeper analysis and refinement are essential to ensure robustness and fairness.

How to Succeed Instead:

• Look for potential overfitting by testing the model on independent, real-world datasets.
• Conduct subgroup analyses to identify performance gaps and address biases.
• Iteratively refine the algorithm by incorporating additional data, tuning hyperparameters, or adjusting features.

Step 3: Create an Opaque Model

Clinicians and regulators need to trust your AI system, and trust starts with understanding. If your model is a "black box" that provides
predictions without clarity on how it arrived at them, it risks being dismissed, regardless of its accuracy.

How to Succeed Instead:

• Use explainability tools like SHAP or LIME to provide insights into how your model works.
• Develop a “model card” to document the purpose, performance, limitations, and risks of your AI.
• Communicate clearly with end-users (patients, physicians, regulators, …) about what your model can and cannot do.

Step 4: Ignore the End-User Experience

Even the most accurate AI system will fail if it’s not user-friendly. Healthcare professionals need tools that seamlessly integrate into their workflows, with outputs that are easy to interpret and act upon. Poor usability or lack of training can lead to frustration and abandonment.

How to Succeed Instead:

• Conduct usability testing with real end-users to identify pain points and optimize the interface.
• Provide training materials and ongoing support to ensure users understand how to use the device effectively.
• Design outputs that are actionable, concise, and clinically relevant.

The Real Path to Success

Building an AI-based medical device that genuinely improves patient care is both a challenge and an opportunity. Success requires a meticulous and thoughtful approach that avoids common pitfalls and ensures your solution is reliable, explainable, and user-friendly.

Here’s how you can stay on the right path:

• Invest in High-Quality Training Data: Start with diverse, clean, and representative datasets to ensure your model captures real-world variability and serves all patient groups equitably.
• Refine Your Algorithm: Don’t stop at good performance. Analyze potential weaknesses, address overfitting, and optimize for fairness and robustness, especially across subgroups.
• Ensure Transparency: Develop explainable models that clinicians can trust. Use tools like SHAP and LIME, and document your AI with clear “model cards” to communicate strengths and limitations.
• Focus on User-Centric Design: Create intuitive interfaces and actionable outputs that integrate seamlessly into clinical workflows. Engage end-users in usability testing and provide comprehensive training.
• Stay Compliant with Regulations: Align your development process with applicable standards like MDR, FDA guidelines, and the AI Act to meet regulatory requirements and inspire confidence.

By prioritizing these key aspects, you’ll create an AI-based medical device that not only meets the highest standards but also earns the trust of healthcare providers and delivers measurable value to patients.

About the Expert

Dr. Mikaël Chelli is an orthopedic surgeon and the co-founder of Sciencer, a platform empowering healthcare professionals to develop AI-driven solutions for precision medicine.

Specializing in shoulder and elbow surgery, he combines his clinical expertise with a passion for data-driven innovation. Sciencer allows device manufacturers to easily create explainable, evidence-based, and regulatory-compliant machine learning models.