Clinical evaluation is an ongoing process that’s conducted throughout the life cycle of a medical device. It is a structured, transparent, iterative, and continuous process that forms part of a device’s quality management system.

Whilst Software as a Medical Device (SaMD) is subject to the same general clinical evaluation principles as other medical devices, certain key additions apply. In our topical article below, our CEO Jörg Dogwiler takes a look at the requirements and considerations regarding clinical evaluation for SaMD in the EU and the US.

The general principles

When it comes to undertaking a clinical evaluation for SaMD, manufacturers must comply with the same legal requirements as for the clinical evaluation of all other types of medical device. 

As with other medical devices, SaMD manufacturers specify an intended medical purpose which has a clinical benefit, and therefore, their SaMD requires clinical evidence within its own conformity assessment.

Performing an effective clinical evaluation normally requires clinical data from the device itself. Manufacturers can obtain this clinical data through a clinical investigation, or they can utilise existing data from a proven equivalent product. For SaMD however, this evaluation approach is often not appropriate.

Whether you are producing SaMD for the EU or the US markets, the following three major components should be considered:

• Valid Clinical Association (also known as scientific validity) is used to refer to the extent to which the SaMD’s output (concept, conclusion, measurements) is clinically accepted or well-founded
• Analytical / Technical Validation is used to demonstrate that the SaMD correctly processes input data, and generates accurate, reliable, and precise output data
• Clinical Validation is used to show that using the SaMD’s output data achieves the product’s intended purpose related to its clinical benefit, and is evaluated and determined by the manufacturer during the development of the SaMD before it is distributed for use (pre-market) and after distribution while the SaMD is in use (post-market)

Requirements in the EU

General considerations for medical devices

The general requirements for clinical evaluation of medical devices are outlined in Article 61 of the MDR 2017/745 (including Annex XIV).

These methodological principles are depicted in the diagram below:

Specific considerations for SaMD Clinical Evaluation

The MDCG document MDCG 2020-1: Guidance on Clinical Evaluation (MDR)/Performance Evaluation (IVDR) of Medical Device Software outlines a possible alternative path for performing the clinical evaluation for SaMD. 

According to the MDCG, three key elements should be considered when compiling clinical evidence for SaMD:

• Valid Clinical Association
• Technical Performance
• Clinical Performance

Valid Clinical Association

Valid clinical association is understood as the extent to which the SaMD’s output (e.g., concept, conclusion, calculations), based on the inputs and algorithms selected, is associated with the targeted physiological state or clinical condition. This association should be well-founded or clinically accepted. The valid clinical association of SaMD should demonstrate that it corresponds to the clinical situation, condition, indication or parameter defined in the stated intended purpose.

Evidence supporting valid clinical association can be generated through literature research, professional guidelines, proof of concept studies, or a manufacturer’s own clinical investigation studies.

Technical Performance

Technical performance is validated by the demonstration of the SaMD’s ability to generate the intended output accurately, reliably, and precisely, from the input data.

Evidence supporting technical performance can be generated:

• Through verification and validation activities such as unit-level, integration, and system testing
• By generating new evidence using curated databases and registries, and reference databases
• By utilising previously collected patient data

Clinical Performance

Clinical performance is validated by the demonstration of the SaMD’s ability to yield clinically relevant output in line with the stated intended purpose. The clinical relevance of the SaMD’s output is a positive impact:

• on the health of an individual, expressed in terms of measurable, patient-relevant clinical outcomes related to diagnosis, prediction of risk, or treatment response
• related to its function, be that screening, monitoring, or diagnosis
• on patient management or public health

Evidence supporting clinical performance can be generated by testing the SaMD under evaluation, or an equivalent device, with the target population for the intended use. The applied methodology should be appropriate for the device characteristics and intended purpose, and may include pre-clinical testing, a clinical investigation, or a clinical performance study.

Whilst valid clinical association, technical performance, and clinical performance portray a methodological principle for the generation of clinical evidence, they do not represent a distinct stepwise approach.

Determining the level of clinical evidence

To determine and justify the level of clinical evidence, both the amount and quality of supporting data should be evaluated. This assessment can be guided by the following non-exhaustive list of questions:

Sufficient Quantity

• Does the data support the intended use, indications, target groups, clinical claims, and contraindications?
• Have the clinical risks and analytical performance/ clinical performance been investigated?
• Have relevant characteristics of the SaMD, such as the data input and output, the applied algorithms or type of interconnection been considered when generating the data to support the performance of the device?
• What is the grade of innovation/history on the market (how big is the body of scientific evidence)?

Sufficient Quality

• Was the type and the design of the study/test appropriate to meet the research objectives?
• Was the data set appropriate and state of the art?
• Was the statistical approach appropriate to reach a valid conclusion?
• Were all ethical, legal, and regulatory considerations/ requirements considered?
• Is there any conflict of interest?

In some cases, particularly for lower risk class I and IIa medical device software (SaMD), clinical data is often not adequate to demonstrate compliance. But both the MDR (Article 61, Section 10) and MDCG document 2020-1 allow an exception. In such cases, demonstration of compliance with general safety and performance requirements can be based solely on the results of non-clinical test methods including performance evaluation, technical testing (“bench testing”) and pre-clinical evaluation. However, this must be justified, covered in your risk management, and considered in line with your device’s intended clinical performance.  

Methodological principle for generation of clinical evidence

Putting the requirements into practice

Take as an example, SaMD intended for image segmentation. Say a manufacturer develops an independent SaMD intended to allow automatic detection of organs and anatomical structures (such as the aorta) in CT scans with the accuracy of a radiologist.

In this example, the manufacturer claims that the SaMD:

• detects abdominal aortic aneurisms on abdominal CT scans
• detects compression fractures on vertebrae
• detects liver cysts

The following depicts how the three key elements we covered earlier might be realised:

Valid Clinical Association

Method | Literature is reviewed to establish valid clinical association

Evidence | The normal shape and size of anatomy is well established, and the segmentation techniques on cross-sectional images correlates well with the actual size and shape

Result | The valid clinical association is established without any gaps being identified

Technical Performance

Method | Verification and validation tests

Evidence | The basic technical performance such as display, modification, window levelling of images, measurements including confirmation of accuracy, sensitivity, and reliability of the SaMD are as per the expected performance

Result | The technical performance meets the expected performance

Clinical Performance

The undertaking of a usability assessment with the intended user groups, in conjunction with the valid clinical association and validation of technical performance results, is determined as sufficient to demonstrate conformity with relevant GSPRs.

In cases where data is available, a retrospective analysis could be performed. In cases where data does not represent the variability of input parameters, the missing data could be generated in a prospective clinical investigation to establish the clinical performance of the segmentation algorithm.

Continuously updating your Clinical Evaluation

The safety, effectiveness, and performance of your SaMD should be actively and continuously monitored. Such data may include (but is not limited to) post-market information such as complaints, PMCF data, real-world performance data, direct end-user feedback or newly published research / guidelines. The data should be subject to the Clinical Evaluation principles depicted in our first diagram.

The unique level of connectivity of SaMD facilitates access to real-world performance data which can be used for multiple purposes, including, but not limited to:

• timely detection and correction of malfunctions
• detection of systematic misuse
• understanding user interactions
• conducting ongoing monitoring of clinical performance
• improving effectiveness
• developing the claims in the clinical development plan
• facilitation of future releases

SaMD can be released for CE marking with initially claimed and validated clinical benefits. Monitoring of real-world performance data can help to formulate hypotheses about future SaMD functionalities and intended uses.

What about the US?

Together with the IMDRF, the FDA has published a guidance document that is very similar to MDCG 2020-1-1. Software as a Medical Device: Clinical Evaluation describes a converged approach for planning the process for clinical evaluation of SaMD to establish:

• A valid clinical association between the output of the SaMD and the targeted clinical condition
• That the SaMD yields the expected technical and clinical data

The principle is the same as that proposed by MDCG for the EU. Manufacturers are required to verify the clinical validity of their product with the clinical evaluation of medical device software following the known steps as outlined in our EU section above. In doing so, as the manufacturer, you will effectively demonstrate that the intended purpose is fulfilled and that there are no unacceptable risks.

Should you have a Clinical or eHealth related challenge, our team is ready and happy to help. Simply get in touch to start the conversation.