EU AI Act & MedTech: Why Humans Must Keep Control Despite Artificial Intelligence

Why Does Artificial Intelligence Create New Challenges for Medical Technology?

In traditional software development, one key question has always been: Does the software work according to the defined requirements?

With artificial intelligence, this question is no longer sufficient. An AI system can work technically as expected, achieve high accuracy, and still create a risk for patients if its limitations are not understood or if the results are interpreted incorrectly.

This is one of the main challenges for medical technology. AI systems can analyse large amounts of data, identify patterns, and provide recommendations. However, AI does not have clinical understanding and cannot take responsibility for medical decisions.

With the increasing use of AI in medical devices, one important question arises: Who is finally in control – the algorithm or the human?

The EU AI Act provides a clear answer. For high-risk AI systems, the regulation requires effective human oversight (Article 14). The goal is not to limit AI, but to ensure that humans remain able to understand, evaluate, and control the results.

For medical device manufacturers, this creates an important change. Human oversight is not an additional regulatory activity at the end of development. It must already be considered throughout the complete product development process.

Why is AI Different From Traditional Software?

Medical devices have been developed for many years based on established development processes. Standards such as IEC 62304 for software lifecycle processes, ISO 14971 for risk management, and ISO 13485 for quality management provide the foundation for controlled product development.

However, AI systems introduce additional challenges.

With traditional software, failures often happen because of incorrect implementation or missing requirements.

With AI, additional risks can result from:

• insufficient or non-representative training data,
• bias in data,
• unknown use conditions,
• model changes,
• incorrect interpretation by users.

For example: An AI system analyses medical images and detects specific findings with high accuracy. However, the training data was mainly created using devices from one specific manufacturer.

After market release, the system is used in hospitals with different equipment.

Due to differences in image quality or device technology, the performance decreases.

The algorithm itself may still work correctly.

The risk comes from not considering the complete system.

Therefore, AI-based medical devices must not only focus on model performance. The complete interaction between data, software, hardware, users, and the clinical environment must be considered.

What are the Requirements of Human Oversight According to Article 14 of the EU AI Act?

Human Oversight does not simply mean that a person confirms a decision at the end. The user must actively be able to monitor the AI system. This includes three main areas:

1. Users must understand the capabilities and Limitations of AI.
2. Humans must always be able to intervene.
3. The user interface must create the right level of trust.

1. Users Must Understand the Capabilities and Limitations of AI

One of the biggest risks with AI systems is the assumption that an automated result is always correct. Users need to understand:

• For which purpose was the system developed?
• Which patient population was considered?
• Under which conditions was the performance demonstrated?
• What are the known limitations of the system?

An AI-based diagnostic support system should therefore not only provide an output. For example: “Abnormality detected.” This information alone is not sufficient. The user may need additional information:

• probability of the result,
• relevant input data,
• possible uncertainties,
• limitations of the system.

Only when users understand the meaning and limitations of AI results can they make safe decisions. This directly connects to risk management according to ISO 14971. Risks are not only caused by technical failures, but also by predictable human behaviour.

2. Humans Must Always be Able to Intervene

Another key requirement of Human Oversight is the possibility to intervene. The user must be able to:

• reject an AI recommendation,
• manually correct a decision,
• stop the system,
• take alternative actions.

AI must therefore not make independent medical decisions. Example: An AI system analyses an X-ray image and identifies possible signs of a fracture.

An unsafe implementation would be: “Fracture present.”
A safer implementation would be: “The AI system identified patterns that may indicate a fracture. Clinical evaluation by the user is required.”

The physician remains responsible.

The AI supports the decision.

These requirements must already be defined during the design phase:

• Which function is performed by AI?
• Which decisions remain with the user?
• Which information does the user need?
• Which control functions must be available?

3. The User Interface Must Create the Right Level of Trust

A frequently underestimated topic is the human interaction with AI systems. People tend to trust automated recommendations, especially when systems appear objective. This behaviour is known as automation bias. In medical technology, this can become a safety risk.

For example, if software presents a diagnosis with high confidence, users may reduce their own critical evaluation. Therefore, the user interface is an important safety element.

The question is not only: “Can the AI provide a result?”

The question is: “Can the user correctly understand and safely use this result?”

This creates a direct connection to IEC 62366 usability engineering.

The interaction between humans and the system becomes part of product safety.

How is Human Oversight Integrated into the Development Process?

The EU AI Act does not create a completely new development process. The challenge is to integrate AI requirements into existing medical device processes.

Design Control
Human Oversight starts already with user needs. Examples:

• The user must be able to evaluate AI results.
• Uncertainties must be visible.
• Intervention possibilities must be available.

These requirements are then transferred into system and software requirements.

Risk management according to ISO 14971

AI-specific risks must be systematically evaluated. Examples:
Hazard: Incorrect AI recommendation
Possible harm: Incorrect decision during diagnosis or treatment

Risk controls:

• Display of uncertainty,
• Definition of use limitations,
• User information,
• Post-market monitoring.

Software development according to IEC 62304

AI creates new challenges for software development. Examples:

• When is a model change considered a software change?
• When is a new validation required?
• How is performance monitored after market release?
• How is the technical documentation updated?

Change control will become a key activity for AI-based medical devices

Which Stakeholders Need to Work Together?

The implementation of Human Oversight is not the responsibility of one single department. Different functions need to work together:

System Engineering. Defines system architecture and system requirements.
Software Engineering. Implements AI functions and technical controls.
Risk Management. Evaluates new risks caused by AI behaviour.
Clinical Affairs. Assesses clinical relevance and intended use.
Regulatory Affairs. Ensures compliance with MDR, IVDR, and the EU AI Act.
Quality Management. Integrates requirements into the quality management system.
Notified Bodies. Assess conformity within the applicable conformity assessment procedures for relevant products.

Only through this collaboration can AI be safely integrated into medical devices.

Practical Example: AI Supporting Radiologists in Image Analysis

An AI system analyses CT images and supports radiologists in detecting lung nodules. The technical development achieves very high accuracy. However, additional questions must be answered:

• Does the system work with different CT devices?
• How does the AI react to poor image quality?
• Does the system recognise situations outside its intended use?
• How are uncertainties communicated to the user?

A safe solution does not only provide a result. It supports the user in making a well-informed clinical decision.

My Conclusion

AI will not be successful because of the algorithm alone, but because of safe integration.

From my perspective, the biggest challenge of AI in medical technology is not developing a powerful algorithm. Technology will continue to develop rapidly. The real challenge is integrating AI into existing development and regulatory processes in a controlled way.

The EU AI Act does not create a completely new world. Many principles are already familiar in medical technology from MDR, IVDR, ISO 13485, ISO 14971, and IEC 62304.

However, AI introduces new challenges. Data, models, and system behaviour must be considered throughout the complete product lifecycle. The companies that will be successful in the long term will not be those that only develop the most complex AI.

The successful companies will be those that integrate AI safely, transparently, and responsibly intommedical devices. Because, in the end, the algorithm does not decide about the patient.

The human does. And that is exactly why humans must always understand, control, and intervene.

Disclaimer. The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the official policy or position of Test Labs Limited. The content provided is for informational purposes only and is not intended to constitute legal or professional advice. Test Labs assumes no responsibility for any errors or omissions in the content of this article, nor for any actions taken in reliance thereon.