Medical devices containing artificial intelligence (AI) or machine learning (ML) still pose significant challenges for regulatory authorities and manufacturers alike, due to several unknowns about AI and its behaviour, and a lack of solutions for how to fulfil certain regulatory requirements.

The FDA has launched a program to establish methods that will address some of these regulatory science challenges by developing robust AI test methods and evaluation methodologies for assessing AI performance both in premarket and real-world settings.

In parallel, our Head of eHealth Dr Dirk Hüber has compiled 5 practical tips for how manufacturers of medical devices that contain artificial intelligence can fill, or at least mitigate, some of these knowledge gaps.

1. Identify the knowledge gaps

It may seem obvious, but with pressure to meet market deadlines, identifying skills or knowledge gaps is something that is often overlooked until it is too late. For manufacturers of medical devices that contain artificial intelligence, being up front and honest about any knowledge gaps that exist within their team or organisation that are relevant for the AI they intend to utilise and its application, is fundamental to mitigating problems further along in the project.

2. Understand your technology

Whilst the nature of artificial intelligence is such that it does not provide an algorithm that can be analysed to understand exactly how the software makes its decisions, manufacturers need to understand the technology applied – at the least its basic functionality, its limitations, its advantages and disadvantages.

Available AI and ML technologies are vastly varied and are designed for a multitude of different purposes. As such, manufacturers must assess whether the technology they’ve chosen is the best possible to address the problem the AI is intended to solve.

Machine learning looks for correlations in the data provided. But bear in mind, that a correlation does not necessarily mean that there is a cause-effect relationship. And if there is a cause-effect relationship, what is cause and what is effect needs to be identified.

It is important to understand (and calculate or estimate) what information gain and uncertainty of result is provided by the ML software. Or more generally, how you can measure the performance of the chosen AI application.

Some machine learning technologies for data mining use a model with parameters determined by the ML software. When choosing your model, several questions need to be understood:

• Is the model representative for the relationship you want to understand?
• Is the complexity of the model well-chosen with respect to the training data at hand? For example, if the model is too simple, the model will result in a sub-optimal representation of the data, i.e., the model will be biased.
• Is the model too complex? If so, it will become over-adjusted to the data, especially if the data contain random noise. Consequently, this would diminish the predictive power of the model.

3. Evaluate your selection of training & test data

The selection of the training and test data poses several challenges that must be addressed.

Ensuring your training data and test data are representative

Does your data represent the whole entity of space the AI is supposed to cover? Are there any data gaps, either within the entity space or at its borders? It is worth noting here, that it is never a good idea to extrapolate data with an algorithm, nor with AI.

Removing bias from your data

If your data includes any information provided by humans, the presence of bias is inevitable. And whilst bias can be a strength in human decision making, for example the ability to make fast decisions in an emergency, humans are shaped by individual and societal biases that are often unconscious. Non-human data may also be biased.

You can mitigate the presence of bias in your data by using technical solutions such as “zeroing out” the bias in words during development, by ensuring transparency and implementing auditing processes, and by striving for a diverse workforce.

Ultimately, bias will still be present, so a measurement strategy to monitor and omit such bias is a crucial step in your development process.

Controlling your data sources

Are your data sources truly under control? Distinguishing whether your data are valid – i.e., correct, real-world data, and not, for example, “AI-generated” data, is vital. AI trained by AI-generated data becomes “sick” – i.e., the variety of generated results will drastically decline over time, rendering the results meaningless. The term for this effect is “Model Autophagy Disorder” (MAD) – meaning the model eats itself. The acronym has been chosen intentionally; the disorder is named after the mad cow disease for its analogous cause.

Another aspect to consider, is how your AI application would react if it were validated repeatedly against the same set of test data. Over-adjustment to the test data is a common consequence of this, and so continuous optimisation of your test data should be factored into your strategy.

4. Strategize continuous verification & validation

In general, the verification and validation of AI and ML software poses many open questions, as the FDA acknowledges with its AI program. This is even more relevant for AI and ML software that is continuously learning and continuously updated – thereby potentially leaving its validated status (e.g., no longer fulfilling the performance and safety requirements of the medical device containing the AI or ML). Manufacturers of medical devices containing AI or ML must be aware of these challenges and develop strategies for how to address them, even if the underlying questions are not yet fully understood.

Once in the market, manufacturers must have all the required processes in place to support the AI software. These are the processes required for medical devices and software in general, but may pose special challenges for AI, for example, regarding complaint investigation or post market monitoring. Metrics and methods to measure and monitor the performance, safety, and effectiveness of the AI software must be developed and established.

5. Avoid over-reliance on Artificial Intelligence or Machine Learning

When developing medical devices containing AI or ML, it is important to not over-rely on the results of AI or ML. Over-reliance on AI will lead to failure, with one reason being that AI lacks the ability for self-reflection and has no common sense nor intuition. As such, human input that constructively contradicts the AI results should be carefully considered within the overall evaluation.  

In conclusion, artificial intelligence and machine learning open new and fascinating possibilities for applications in the medical device world. They become increasingly adept at solving problems and finding solutions, often, more efficiently than humans. However, the challenges and risks should not be underestimated, and manufacturers must plan for overcoming the limitations. Furthermore, regulators are challenged to give clear and harmonized guidance to the industry on how these technologies can be used in pre-market and post-market applications for medical devices – a topic which we discuss in depth in this Congenius whitepaper.

Should you have a challenge related to AI or ML software, our eHealth team is ready and happy to help. Simply get in touch to start the conversation.