Audit Trails: What Does the System Actually Prove?

In the previous Article, we drew an important distinction: 

Traceability asks: “What happened?” Evidence asks: “What can be proved?” 

Now let us take the next step. 

Imagine that a hospital is required to prove that Instrument Set A1746 completed a compliant decontamination cycle on 16 March 2023. 

The Track & Trace System says that it did. 

It may be nothing of the kind. 

Counsel looks at the Record and asks: 

•  Who created the underlying Record? 
•  Who had the ability to alter it? 
•  If somebody did alter it, would the intervention itself be recorded? 
•  Does the Audit Trail preserve what somebody actually did within the System, or merely record that they logged into it? 
•  Is the information routinely displayed by the System the same information contained within its underlying Audit Data? 

And then comes the question which matters: “What does your Audit Trail actually prove?” 

Those questions expose a difficulty which sits at the heart of Digital Evidence. The mere existence of an Audit Trail does not establish what it records, what it omits, or ultimately what it is capable of proving.

The Record has been preserved. It is produced in Court. Perhaps, after argument, the Judge admits it into Evidence. 

On the face of it, therefore, the Record appears to have cleared the Admissibility hurdles we discussed in the earlier Articles. It is in Evidence. The organisation might reasonably think that the difficult part is over. 

That may sound like success. 

What Does an Audit Trail Actually Prove?

Audit Trails #1 

Faced with questions about the reliability of a Digital Record, there is an answer which sounds immediately reassuring: 

“But we have an Audit Trail.” 

But what, exactly, does that prove? 

That question was explored on an extraordinary scale in Bates & Others v Post Office Ltd (No 6: Horizon Issues) [2019] EWHC 3408 (QB).  

This was major High Court litigation concerning the operation and reliability of the Post Office’s Horizon computer system. Mr Justice Fraser (a senior High Court Judge) heard extensive factual and expert Evidence concerning the System, including its bugs and defects, remote access, transaction data and the Audit Data which could be used to reconstruct what had actually happened. 

For lawyers, the importance of the case extends far beyond the terrible facts of the Post Office scandal. 

Horizon had Audit Data. Indeed, Fraser J described accurate Audit Data as “vital to the proper operation of a system such as Horizon”. 

That sounds encouraging. 

But Bates demonstrates why the words “we have an Audit Trail” cannot be the end of an evidential enquiry. 

In one example, ordinary management data indicated that a sub-postmaster had reversed a transaction. Examination of the underlying Audit Data demonstrated that he had not. Fraser J’s conclusion was direct: 

“This shows that the management data is not entirely reliable.” 

The distinction matters. 

The information routinely presented by a Digital System may not be the same thing as the underlying Audit Data required to establish what actually occurred. 

Fraser J described the management information available to the Post Office as “confusing, contradictory, has been shown to be wrong”, whereas the underlying Audit Data was “far superior and the best evidence available of what has occurred on Horizon.” 

The problem was not an absence of Data. There was plenty of it. The problem was that the information ordinarily relied upon could be wrong, while the better Evidence lay deeper within the System. 

So when somebody says – perhaps, from their manner, exclaims – “But we have an Audit Trail!” the evidential enquiry has only begun.

What Does an Audit Trail Record?

Audit Trails #2

That question takes us deeper into Bates – the shorthand lawyers commonly use for the case by referring to its first-named party. 

Horizon did not simply have Audit Data. It also had privileged users: people with powerful permissions capable of intervening in the System and, in certain circumstances, altering transaction data. 

If somebody with elevated privileges can intervene in a Digital Record, the evidential question is not merely whether the System records access, but whether it records what they actually did. 

In Bates, the two IT experts agreed that, where technical users possess privileged access with wide-ranging capabilities, each action taken while logged in should be recorded and audited. Fraser J regarded that requirement as “entirely conventional” and “technically justified”. 

But Horizon had fallen a long way short of it. 

From 2009 to July 2015, the privileged-user logs recorded whether a user had logged on or logged off, but not what actions that user had taken while logged in. 

The System could establish presence without establishing activity. 

Even after July 2015, the Judge recorded the experts’ agreement that the logs were “not a useful source of evidence about remote access” because of their lack of content. His conclusion was direct: 

“In my judgment, this amounts to a deficiency in controls.” 

Now shine the light of Bates upon an example healthcare or medical-device environment.  

Suppose an Administrator accesses a Track & Trace System at 10:17 and logs out at 10:43. During that period, the Administrator has sufficient privileges to amend records associated with Instrument Set A1746. 

The Audit Trail proves that the Administrator entered the System. 

But if it does not record the actions performed during that session, can it prove whether Record A1746 was altered? 

Bates exposed another difficulty. 

Fraser J found that users with sufficient access permissions could inject additional messages — i.e. Data — into Horizon. Those messages contained information identifying the user responsible. But that identifying information “would not be visible in the standard audit extracts”(said Fraser J). It became visible only through “a detailed examination of the raw audit data”. 

An Audit Extract may therefore be authentic yet omit information within the underlying Audit Data which is directly relevant to establishing who did what. 

The Court later wanted to know how frequently one particularly powerful privileged-user role had actually been used. Neither the Post Office nor the experts could provide a clear answer to what Fraser J described as “a very simple question”. The inability to answer it arose from Fujitsu’s “plainly inadequate records”. 

The alarming and terrible fact is that a sophisticated System may generate enormous quantities of Data and possess something all involved are satisfied to call an Audit Trail. Neither fact establishes that the Audit Trail records the events which will matter when a particular transaction, intervention or Clinical Record is challenged years later. 

The Digital Record Is Not the Real-World Event

Audit Trails #3 

There is, however, a still deeper problem. 

Suppose the Audit Trail accurately establishes what the computer recorded. 

What does that prove about what actually happened in the real world? 

The distinction became painfully important in another Post Office case, this time before the Criminal Division of the Court of Appeal. We have moved from the High Court to one of the most senior Courts in England and Wales. And the consequences could scarcely have been more serious. 

The case is Hamilton & Others v Post Office Ltd [2021] EWCA Crim 577. In the relevant Horizon cases, the Court of Appeal found there was “no independent evidence of an actual shortfall”. It accepted that defects within Horizon created a material risk that an apparent shortfall “did not in fact reflect missing cash or stock”. 

Remember, when we talk about cash and stock here, the evidential principle is not confined to cash and stock! In Digital Healthcare, the Record might concern an Endoscope, or a device used to seal an artery, or another item upon which patient safety depends. 

The computer showed a shortfall. But that did not prove that the money was actually missing. 

The Court stated the consequence in terms which could scarcely be clearer: “If the Horizon data was not reliable, there was no basis for the prosecution.” 

For Digital Healthcare, Medical Devices and Laboratory Systems, the equivalent distinction is straightforward. 

A Track & Trace System may record: Instrument Set A1746 — Decontamination Cycle Complete — PASS. 

But what has actually been proved? 

That the Digital System contains a Record saying that the cycle was completed? Or that Instrument Set A1746 actually underwent the required decontamination process and successfully completed it? 

Those propositions may look almost indistinguishable on a screen. Evidentially, they are not. An Audit Trail may help establish what happened within the Digital System. The Court may still have to determine whether what happened within that System reliably proves what happened outside it. 

The System says X happened. But did X actually happen?  

Between those two propositions lies the evidential foundation upon which everything else may depend. 

When a Computer Record Becomes the Fact

Audit Trails #4 

The danger becomes greater when the distinction between Record and Event is forgotten. 

In Hamilton, the Court of Appeal said that the Post Office had “treated what was no more than a shortfall shown by an unreliable accounting system as an incontrovertible loss”.  Notice that word: “incontrovertible”. Has your organisation begun to treat what its Systems record as incontrovertible too? 

Let’s remember: Defendants were “prosecuted, convicted and sentenced on the basis that the Horizon data “must be correct” when “in fact there could be no confidence as to that foundation.” 

A proposition generated by the computer had become, institutionally — including amongst those responsible for the organisation at executive level — an Undeniable Fact. Has your organisation reached exactly the position Fraser J observed? 

Fraser J described the Post Office’s approach as “simple institutional obstinacy” and, memorably, as “the 21st  century equivalent of maintaining that the earth is flat.” 

The institutional danger — and it is a danger for any heavily computerised organisation — is that the organisation begins to believe the Record and then to defend it, until incontrovertible contrary Evidence is treated not as a reason to question the System, but as a reason to disbelieve the person challenging it. 

Now apply that danger to a regulated healthcare environment. The same Track & Trace Record may be relied upon by a Quality Department, a hospital, an auditor and a Regulator, each successive reliance reinforcing confidence in the Record, before a witness eventually tells a Court: 

“The Instrument was properly decontaminated. The System proves it.” 

Does it? Or has the Digital Record quietly ceased to be Evidence of the Fact and become the Fact itself? 

What Does an Audit Trail Actually Prove?

An Audit Trail is not an evidential magic wand. 

Its existence does not prove that it records the actions which matter.  

Nor does a standard Audit Extract prove that it contains everything which matters within the underlying Audit Data. 

And even an entirely accurate Record of what happened inside a computer does not, without more, prove that the corresponding Event actually occurred in the physical world. 

For organisations operating Digital Healthcare, Medical Device, Laboratory and Track & Trace Systems, that changes the questions which should be asked when those Systems are designed, acquired or upgraded. 

It is not enough to ask whether the System you’re acquiring or the upgrade you’ve commissioned has Audit Trail functionality. 

Ask instead: 

•  Whether the Audit Trail records privileged interventions at sufficient granularity; 
• Whether the underlying Audit Data can be preserved and retrieved; 
• Whether standard reports omit information needed to understand that Data; and  
• Whether somebody independent of the System could reconstruct what happened years later. 

Because one day those questions may no longer be asked by an auditor or a Quality Manager. They may instead be asked by Counsel in a Court of Law — when the answers matter rather more. 

And Counsel will neither be reassured nor accept the answer: 

“But we have an Audit Trail.” 

Counsel will ask: 

“What does it actually prove?”

Missed the earlier parts of the series?

Read the first three articles by Dr. Brian Bandey:

When Compliance Data Enters the Courtroom – and Fails

Why Most Compliance Systems Cannot Prove Data Integrity in Court

The Forgotten Purpose of Evidential Friction

Disclaimer. The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the official policy or position of Test Labs Limited. The content provided is for informational purposes only and is not intended to constitute legal or professional advice. Test Labs assumes no responsibility for any errors or omissions in the content of this article, nor for any actions taken in reliance thereon.