The fundamental misconception plaguing our profession and its impact on the (mis)use of AI
Documentation is not the aim, but merely a means to an end.
From discussions with clients, former colleagues in different fields, and AI evangelists, it is clear that many have a very narrow view of what quality and regulatory work entails. They believe it is all about creating pages and pages of text and see this as overhead and an unnecessary burden that can easily be automated away with AI.
What this view misses is that, for a regulatory strategy document, the actual work is the discussion and the trade-offs that were considered. Writing down the final classification, selected predicate, and regulatory pathway is the easy part, and indeed AI could support this. Risk assessment is about considering how the device is used in clinical practice and identifying the appropriate control measures, not just filling in a large Excel table. In the end, the paperwork is there to document that certain aspects were considered and tests were performed. The old saying remains true: “If it is not documented, it did not happen.” However, we now seem to be moving into the opposite situation: “It is documented, but did it happen?”
My previous article about the use of AI in regulatory affairs resulted in a lot of discussion, both in the comments on that article and elsewhere. I also had in-depth discussions in real life with various people developing these tools, as well as other experts in the field. This article builds on those discussions and explores the impact on both strategic work and the creation of documentation.
Regulatory strategy
As discussed before, in MedTech your regulatory strategy is your business strategy. Small changes in the intended use and claims can have a major impact on the regulatory classification, eligibility of potential predicates, required clinical evidence, and thus time to market. Most of the work is therefore spent defining the scope of the product and the exact wording of the product definition. This requires taking into consideration the business ambitions and constraints, as well as the runway and financing roadmap.
Once this has been worked out and the wording is clarified, it is relatively straightforward to apply the classification rules, relevant guidelines, and identify applicable predicates. In reality, this is often an iterative process. In my discussions with AI regulatory providers, it became clear that their tools work reasonably well for writing down a strategy based on a well-formatted product description, but do not help with making these trade-offs.
The demos use well-formed examples with all the right qualifiers. For example, they include phrases such as: “The software is used in the triage and prioritization of radiological images for clinical assessment.” These qualifiers are not typically wording that founders would come up with by themselves. This can lead to approaches that are not realistic within the available budget and timelines, or positions that are ultimately not defensible during a notified body or FDA review.
Thus, trusting your full regulatory, and therefore business, strategy to these tools is risky without proper expert support. In the hands of actual experts, these tools can save time when drafting the final regulatory strategy and creating an initial draft of applicable standards and GSPRs. We are actively exploring options to make our work better and more effective for our clients.
Drafting technical file documentation
The other area where we see many teams experimenting with LLMs is in the creation of documentation describing their product. This is, of course, very appealing. Most engineers do not like writing reports or documentation. An LLM is therefore a very convenient way to outsource this drudgery.
The output is typically voluminous and plausible-sounding, but its relationship to the product and how it is used in clinical practice depends heavily on how much information the LLM has available to build its response.
This is where major issues can start to occur. Using an LLM to document software design and architecture, where it has access to the full code base, might work reasonably well. However, one would hope that the software design team spent time up front defining the high-level architecture and documenting it so that the complete team, and your friendly code bot, know what the plan is. So why not use that?
The further one moves away from describing the software and its functioning, the more difficult it becomes for the LLM to gain sufficient insight into the product and how it is used. For products that contain a significant hardware component, there may not always be a good basis from which to draft design documentation. Design choices regarding materials and component selection, for example, require input from the people who made those decisions. One can easily have an LLM generate a rationale and write a plausible document, but that may have little to do with the actual trade-offs that were considered.
Having design documentation that does not correctly describe reality is annoying and a non-conformity waiting to happen. However, as long as the actual design is correct and functioning properly, it may not immediately impact patient safety. The consequences often appear later, when this incorrect documentation is used for future updates or, more worryingly, as part of the risk management process.
Risk management documentation
One of the most important processes in medical device product development is risk management. In the end, these devices are used to diagnose or treat patients, and it is critical to understand how potential failures could result in harm. Based on a thorough risk assessment, it is possible to identify risk control measures that mitigate these risks.
ISO 14971 is the state-of-the-art standard, and it requires manufacturers to identify, evaluate, and mitigate risks. Because many different aspects are associated with the use of a device, this requires a multidisciplinary team that jointly understands the technical design, manufacturing process, and clinical use of the device. The process requires building a common understanding and a shared model of how the device works and is used. Based on this mental model, it becomes possible to identify relevant risks and appropriate control measures.
One of the outputs of this process is the risk assessment file, typically a large table with an entry for each identified risk, evaluating the chain of events that can lead to harm, the probability and severity of that harm, and the associated control measures. These files can become large and unwieldy and are therefore often viewed as an ideal target for automatic generation using an LLM.
The challenge is that the LLM has no direct knowledge of how your device is actually used in practice. Furthermore, depending on the quality of your technical documentation, possibly AI-generated, it may also have significant gaps in its understanding of how the device is designed and functions. It will still generate a complete risk file that looks impressive, but it may contain major gaps or control measures that are described but not actually implemented. Since the team is not jointly building the underlying mental model, it becomes difficult to take real ownership of the outcome.
AImnesia
In a recent post, Eran Rubens proposed the term AImnesia with the following definition:
AImnesia (n.) /aim-NEZ-ee-ə/
1. The sudden and total inability to explain, defend, or even recall a document, slide, or work product you technically “wrote” using AI.
2. An occupational hazard of the modern knowledge worker, characterized by a strong sense of ownership over a deliverable paired with a complete absence of understanding of it.
I’ve seen this in a number of places over the past weeks. People claim to be the author of a lengthy document, but when challenged on a clear mistake, they hide behind the argument that the AI did not have the right information.
Observing the workflow of some of these “authors,” it appears that they no longer create or edit in a text editor or Excel file. They interact with the document exclusively through AI prompts. Instead of manually correcting the document, they submit another prompt asking the AI to make the changes. This allows them not only to create but also to edit large documents very quickly.
While this sounds appealing, it has the disadvantage that you are not really building a model of what you are doing. This is not necessarily a problem when the AI is describing how software is structured and functions. However, it becomes problematic when creating something new, such as a risk evaluation. And in all cases, the documentation will be much longer then necessary, costing significant more review time downstream.
Potential solutions
If teams are frustrated because the process is complex and requires documentation that does not add value, explore whether the process itself can be simplified (see previous post). In general, the basic requirements of medical device regulations and standards are not unreasonable, but internal processes sometimes go overboard. This appears to be especially true when you ask your friendly LLM to create processes. They have a tendency to create many different roles, documents, and steps.
Have internal discussions about what it means to be the author of something. What uses of AI are acceptable?
How do you avoid outsourcing all your thinking to AI while still building a mental model of what is going on? For example, create the first draft yourself and then have the AI provide feedback, perform formatting, and suggest minor edits.
How do you deal with the fact that the output still needs to be reviewed by others who must spend time reading it? Respect their time and effort.
AI can certainly play a role in reviewing documents for completeness and consistency. This is an area where AI can be very useful. It can check whether wording is consistent and whether a document referenced for a GSPR actually addresses the relevant requirement.
As an example, Saber GRAF, PhD recently posted a demo showing how AI could be used to assess the completeness of clinical protocols. The discussion we had in the comments was that checking whether something is present or not is relatively straightforward, but assessing completeness and the appropriate level of detail is much more difficult.
Conclusion
Text-generating AI systems are here to stay, and under the right circumstances they can help reduce some of the more menial aspects of quality and regulatory work. Fundamentally, however, documentation exists only to record what was actually done and considered.
Since most companies will not claim that AI was the author of their documents or that it performed all evaluations, humans must remain able to take real ownership of the documentation. This still requires developing a genuine understanding of what is happening, building the mental model, carefully considering the arguments, and then documenting the outcome. Simply responding with “LGTM” to AI-generated output will not be enough.
Getting this balance right is critical in the quality and regulatory space. Unlike coding tasks, where feedback loops are often short, the feedback loops in our domain are measured in years. The wrong strategy can result in companies running out of runway because they require too much evidence or receive rejected submissions. Design documentation that does not accurately represent the product can create significant complications when the product is updated. An underestimated risk or missing risk control measure can ultimately result in patient harm and costly recalls.
This remains an interesting topic, and I look forward to the discussion in the comments about how other experts see these challenges and opportunities.
