Transparency & control of AI Responses in a Clinical Process

Why Observability and Auditability Are Distinct Problems 

Confusion between these two concepts is common, and it leads to architectures that address one while believing they have addressed the other. 

Observability is an operational property. It answers the question of whether the system is working correctly, in real time or near-real time: are queries being processed within expected timeframes, are error rates within acceptable bounds, is perceived response quality degrading, are certain query types consistently producing unsatisfactory results? Observability is what allows a problem to be detected before it becomes an incident. 

Auditability is a regulatory and legal property. It enables the complete reconstruction of how any given response was produced: which documents were retrieved, in which version, with what relevance score, what prompt was submitted to the model, what response was generated, who asked the question, and from which application. Auditability is what allows an organization to respond to an inspector, defend a clinical decision, or investigate a pharmacovigilance incident. 

These two dimensions require different technical mechanisms, even if they share a common logging infrastructure. 

The Regulatory Context: What the Frameworks Actually Require 

The frameworks governing clinical information systems converge on similar requirements, even if the specific language varies. 

The FDA’s 21 CFR Part 11 regulation requires that any computer system used in a regulated context maintain timestamped, tamper-proof audit logs linked to user identity. This requirement applies to AI systems whenever they participate in a documented decision-making process. 

The ICH E6(R3) Good Clinical Practice guidelines introduce traceability requirements for source data and decision-making processes in clinical trials. An AI assistant used by an investigator or data manager in that context falls squarely within scope. 

The European Medical Device Regulation MDR 2017/745, and even more so the AI Act currently being rolled out, impose transparency and traceability obligations on AI systems intended for medical use. Clinical decision support systems are explicitly classified as high-risk systems. 

The GAMP 5 standard defines validation requirements for computerized systems in GxP environments. An AI system deployed in a clinical process must be qualified and supported by documentation demonstrating that its behavior is predictable and controlled. 

Across all of these frameworks, the question is the same: can you demonstrate, for any response generated by your AI system, on what basis it was produced and who was accountable for it? 

The Four Pillars of an Observable and Auditable AI System 

1. Structured Logging of Interactions 

The first pillar is the systematic and structured capture of every interaction with the AI system. This goes beyond simply retaining logs: each query must be recorded with its full context, meaning the user identity, the source application, the precise timestamp, the query as it was formulated, and the response generated in its entirety. 

This logging must be tamper-proof. In a regulated environment, a log that can be altered has no evidentiary value. Logging solutions compliant with 21 CFR Part 11 use electronic signature mechanisms or write-once recording on infrastructure whose integrity is guaranteed. 

The structure of the log also matters. A free-text log is difficult to use during an audit. A log structured in JSON or a standardized format allows audit queries to be automated, reports to be generated, and problematic patterns to be identified. 

2. Traceability of Documentary Sources 

In a RAG architecture, the response generated by the language model is shaped by the documents retrieved by the search layer. Auditability of a clinical AI response requires that these sources be traced precisely: which documents were retrieved, in which version, with what relevance score, and which ones actually contributed to the context submitted to the model. 

This traceability is not trivial to implement. Most RAG frameworks retrieve document chunks without exposing in a structured way the source document identifier, its version, or its date. A system designed for a clinical environment must address this from the outset, maintaining a traceable correspondence between each chunk used and its versioned source document. 

The granularity of traceability is also a design question: should tracking happen at the level of the whole document, the section, or the paragraph? The answer depends on the use case, but in a pharmacovigilance or regulatory submission context, section-level precision is generally required. 

3. Versioning of the Corpus and Models 

An AI response is only reproducible if the document corpus and the model used to generate it are identified and versioned. This is a requirement that non-technical teams rarely anticipate, and that technical teams sometimes overlook in favor of operational simplicity. 

A document corpus is constantly evolving: new documents are ingested, existing ones are updated, some are archived or invalidated. If a response was generated from a version of the corpus that no longer reflects the current state, reproducing that response during an audit requires restoring the corpus to its state at the corresponding date. Without version management, this is simply not possible. 

The same logic applies to models. An LLM is updated, fine-tuned, replaced. A response produced by an earlier version of the model is not necessarily reproducible with the current version. An auditable clinical system must therefore maintain a registry of model versions used and link them to the corresponding interaction logs. 

4. Identity-Linked Access Control 

Auditability requires knowing who asked what. In a multi-user clinical environment, this means strong authentication and identity management integrated into the AI layer itself, not just in the front-end application. 

Logging a user identifier is not enough: that identifier must be linked in a non-repudiable way to a verified physical identity, with the corresponding rights and role at the time of the interaction. In GxP processes, electronic signature in the sense of 21 CFR Part 11 may be required for certain categories of interactions. 

Granular access control, as defined in source systems, must propagate all the way through to the AI layer: a user must not be able to obtain via an AI assistant information they cannot access directly. This property, sometimes called end-to-end security in RAG architectures, is difficult to guarantee without explicit design. 

The Most Common Failure Points in Practice 

In the clinical AI deployments observable today, failures on observability and auditability follow recurring patterns. 

The first is the absence of logging at the RAG layer. Teams log calls to the LLM but not queries to the search layer or the documents retrieved. The result is a response that is retained without any traceable documentary context. 

The second is the confusion between application logs and audit logs. An application log is designed for debugging: it is often rotated, sometimes compressed, and not necessarily tied to identity. An audit log is designed for proof: it must be complete, immutable, and usable by a third party. Conflating the two leads to having a large volume of data but no usable evidence. 

The third is the absence of corpus versioning. Organizations update their document base without maintaining a history of successive corpus states. They then find themselves unable to reproduce a response generated six months earlier when an audit or incident investigation requires it. 

The fourth is the use of third-party model APIs without SLA guarantees on log retention. When an LLM is consumed via an external API, the API logs belong to the vendor, not the organization. This creates an unacceptable regulatory dependency in a clinical context. 

What a Correctly Designed Architecture Guarantees 

A properly designed clinical AI system must be able to answer, for any response generated in the past 24 months, the following questions: who asked this question, from which application, at what time, which documents were used to construct the response, in which version of those documents and of the model, and what exactly was the response. 

This capability cannot be retrofitted onto an existing system. It must be architected from the outset, with structured logging, corpus versioning, source traceability, and identity-linked access control treated as first-order components, on the same level as response quality or system performance. 

Organizations that approach these topics in reverse order, starting with response quality and deferring compliance for later, typically discover that an architectural overhaul is required before clinical production deployment becomes possible. That is an avoidable cost. 

FAQ

01

Does a clinical AI system necessarily need to be deployed on-premise to be auditable?

No. A cloud architecture can be auditable if logs are retained under the organization’s own responsibility, if vendor contracts guarantee SLAs on log retention and access, and if data does not transit through shared infrastructure that fails to meet applicable regulatory requirements.

02

What is the recommended retention period for AI audit logs in a clinical environment?

The duration depends on the applicable framework. Under ICH E6, retention of trial data is tied to the product lifecycle. In practice, a minimum retention of 15 years is often applicable for clinical data. These requirements should be defined with regulatory and legal teams before deployment.

03

How should observability be managed when multiple LLMs are used in parallel?

Each model must be identified in the logs with its precise version. A centralized orchestration middleware that routes queries to the different models and aggregates logs is the most robust approach for maintaining consistent observability in a multi-model environment.

04

Does auditability apply to multi-turn conversational interactions?

Yes, and this is one of the more complex cases. In a multi-turn conversation, the context of each response depends on the history of prior exchanges. Full auditability requires retaining not only each individual response but the entirety of the conversation session, including the documentary context drawn on at each turn.

05

What is the relationship between auditability and explainability?

Auditability is a system property: it concerns the traceability of sources and processes. explainability is a response property: it concerns the ability to explain why the model produced a particular line of reasoning. The two are complementary but distinct. A system can be auditable without being explicable, and a model can provide explanations without the system hosting it being auditable.