Clinical Study Audit

Today, audits are a familiar and integral part of clinical studies, especially when the results are submitted to official authorities for drug registration and market authorization. Under clause 1.6 of the ICH Good Clinical Practice Guideline, an audit is “a systematic and independent examination of trial-related activities and documents to determine whether the evaluated activities were conducted, and the data were recorded, analyzed and accurately reported, according to the protocol, the sponsor’s standard operating procedures (SOPs), Good Clinical Practice (GCP) and applicable regulatory requirements.”

An audit is performed by personnel “…who are independent of clinical-trial/system[s], not involved in the routine monitoring and quality control of the clinical study” (clause 5.19.1). Otherwise — especially when errors are found — a conflict of interest may arise. Depending on the objective, an audit may be performed by one person or by a team. Involving several people is very effective because it allows group expertise. Most pharmaceutical companies and contract research organizations (CROs) have a Quality Assurance (QA) department, one of whose duties is conducting clinical-study audits.

As in any other specialty, an auditor must receive appropriate training before taking up their duties. A common practice is to staff the QA department with the most experienced employees — those with clinical-trial coordinator experience. Even so they must take additional training, as the work has its specifics. What must a qualified auditor know? They must know the regulatory requirements, the ICH GCP rules, the sponsor’s procedures and the study protocol, have sufficient knowledge of the investigational product and the content of the Investigator’s Brochure, have the skills to assess the state of work at the research site and the study documentation, and be able to develop an audit plan, identify and resolve trial-related problems and give business advice. The auditor must be independent of the project under review and unbiased.

Audit goals and objectives. Clinical-study audit goals are defined in advance and described in the sponsor’s SOPs. They include a range of activities to ensure that the trial is conducted safely, that subjects’ rights are protected, that the investigator and site staff are qualified and trained, that the protocol and procedures are followed, that the work complies with ICH GCP and regulatory requirements, and that the trial results are credible and suitable for submission. The work of clinical monitors is also reviewed — whether monitors are performing their work correctly and in a timely manner, in compliance with ICH GCP, sponsor procedures, protocol, local legislation and the relevant SOPs. One objective of the audit is preparing for possible inspections by regulatory authorities. Based on the audit results, recommendations are given to correct the findings and improve work quality. A “good” audit also aims to train investigators on trial conduct and documentation — something done during the collaborative work itself.

Types of audits. Audits can be split into several large groups, first of all by who initiates and conducts them. If QA staff of the sponsoring pharmaceutical company or CRO — the ones directly organizing and overseeing the study — perform the audit, it is called internal. If the audit is performed by an independent third party, i.e. QA staff of another CRO that has had no other involvement in the study, the audit is called external.

An audit may focus on different aspects of a clinical study. It may review the work of sponsor or CRO staff directly involved in organizing, conducting and monitoring the study — a qualification audit. Such audits are typically carried out before a joint project begins or when considering further cooperation between a pharma company and a CRO. The audit may also focus on quality assurance at the research site, where the clinical study is actually conducted — an on-site audit. Statistics show that on-site audits account for over 55% of all audits. Historically, companies sponsoring clinical trials audit pivotal trials, whose results are decisive for continuing the development program and subsequent registration.

An on-site audit can be performed at various points in the study — at the very earliest stages, when patient enrollment is just beginning (and many companies prefer to audit as early as possible), or after the clinical-study report has been written, when questionable statistical findings appear, or when it is necessary to prepare the site for a regulatory inspection. The same site may be audited several times. Some sponsors initiate an audit when 25% or fewer of the planned patients have been enrolled. At this stage, with much already done “but much more ahead,” it is important to check the quality of the work being performed so that issues can be identified and corrected in time.

Accordingly, on-site audits can be divided into “planned” and “unplanned” (for some cause); an expected regulatory inspection is often cited as a separate reason for an audit.

Planned audits. Today, a well-controlled trial will always be subject to audit because audits are part of the trial’s quality-assurance system. For Phase II–III studies (we deliberately leave aside Phase I and IV studies) the audit plan is, as a rule, drawn up before the trial starts (recall the “monitoring plan” — many analogies). The minimum requirement is at least one audit. The percentage of sites to be audited “on plan” depends on the sponsor’s/CRO’s SOPs, study complexity and project budget. 10–25% is a typical benchmark. Even a routine planned audit requires the right site selection — criteria similar to those used for for-cause audits apply.

For-cause audits. Site selection. Criteria for selecting a site for an audit are varied but predictable. When an investigator hears that their site will be audited, they sometimes ask the monitor, “why is my site being chosen?” The main criteria usually point to sites with the highest patient enrollment and/or with very fast enrollment — the more patients enrolled, the greater the impact of the site’s data on the overall results, so the data must be checked thoroughly. An investigator enrolling many patients quickly may also make more mistakes — with inclusion/exclusion criteria, protocol procedures or simply collecting complete and accurate data.

Simultaneous participation of the investigator in several studies — especially with similar inclusion/exclusion criteria — may also attract auditors’ attention, since compliance with inclusion/exclusion criteria is one of the main ICH GCP requirements (clause 4.5.2) and the investigator “should have sufficient time to properly conduct and complete the trial” (clause 4.2.2).

A site with frequent protocol deviations, GCP and regulatory violations is, of course, one of the first candidates for an audit — failure to meet these requirements (clauses 4.1.3 and 4.5) is a direct route to invalid data.

Limited study experience or significant changes in the investigator team during the trial also attract QA attention.

A site with an unusually high proportion of serious adverse events — or, conversely, one where the investigator reports no or very few adverse events compared with other sites — may also be audited. Such situations may suggest that the investigator is inadequately recording patient data or prescribing the investigational product incorrectly. The audit aims to verify compliance with the inclusion/exclusion criteria, adequate medical care (clause 4.3) and subject safety.

Poor quality/completeness of source documents identified during monitoring. The monitor identifies shortcomings, records them in the report, and an audit decision may follow from that information.

Another cause may be questionable efficacy data obtained during an interim or final analysis that differ significantly from other investigators’ results, or a large number of queries on submitted data.

And, of course (fortunately rarely), suspicion of data falsification or fraud may trigger an audit.

Some terms and examples. A lot in clinical trials is built on trust. Data published in journals is often accepted as presented, though it may be disputed or revised. Audits can uncover deception and various forms of misconduct.

Falsification — presenting incorrect information (e.g. false data) or, conversely, failing to report important information with the intent to mislead another person or group.

Misconduct — non-compliance with clinical-study standards. Misconduct may be intentional or unintentional (e.g. carelessness). When misconduct in the research process is deliberate and unlawful, it becomes a criminal act.

Fraud — deliberate alteration of a small amount of data leading to full falsification of an entire study.

For example, an audit of one site revealed significant discrepancies between the data in the CRF and source documents (lab forms, patient diaries, ECGs and X-rays).

Significant attention to spotting false data is paid in Europe and the US. In the US, a major public scandal involved a study in breast-cancer patients where a new therapeutic strategy was based on partially false data; fortunately, excluding 16% of the study population did not affect the final conclusions.

Bailey (1997) reported a preclinical study with data-analysis problems in one lab out of four; after it was pointed out to management, the results “became comparable” with those of the other labs. During an on-site audit inspectors were able to confirm the data had been falsified — for example they found no radioactivity signs in the animal tissue even though radioactive isotopes had been administered.

Cases are described where an audit found the absence of source data and its retrospective entry. For instance, in a randomized placebo-controlled 3-month study in heart-failure patients, 16 of 61 investigators were selected for audit. One investigator who had enrolled 18 patients kept postponing the audit visit, just as he had postponed monitor visits. When the auditor arrived, the investigator presented computer printouts as source documentation, claiming that data were entered directly into the computer and that no other source documentation existed. Comparing the printouts with the CRF, the auditor found that the printouts contained only patient initials and numbers. There was no information identifying the patients (date of birth, initials). The structure and content of the printouts were identical to the CRF. Corrections were made only in the CRF, while the printouts retained the old values. The correction dates matched the patient visit dates, indicating that the data had been entered retrospectively — a violation of ICH GCP rules (clause 4.9.2).

Adequate preparation and training of site personnel guarantee that investigators will follow the protocol and collect data correctly and on time. During investigator training before the study starts (kick-off meeting, initiation visits), quality-assurance methods must be presented and explained. If investigators understand the role and tasks of the QA department, they will better understand and accept the audit process. An additional “check” is often perceived negatively by those about to be checked, so auditors are often unwelcome at the CRO, investigator or site level. A well-designed communication system and SOPs help investigators properly perceive the quality-assurance system — including audits — of clinical studies.

Investigators must understand that auditing some sites is the sponsor’s obligation. Survey data show that 6 companies audit 15–20% of sites for pivotal trials, another 6 audit 25–45%, and 2 audit 75–100%.

Audit preparation. Preparation must start when the clinical study itself is being prepared, not when the audit notice arrives. The philosophy of a clinical trial is compliance with GCP. Auditors must be able to confirm that the trial is being run in accordance with those rules — something that cannot be achieved in the few weeks after an audit notice. From the start of the trial, from protocol signing and submission to the Ethics Committee, the investigator must comply with all ICH GCP requirements, sponsor procedures and local legislation.

The right choice of sites, their adequate training on the protocol, procedures, ethical aspects, handling of the investigational product and other clinical-trial materials, documentation, and AE registration and reporting before and during the study, plus a sensible distribution of responsibilities among team members — all largely determine the fate of the study and the validity of its data, and therefore also the audit outcome.

The choice of clinical-study monitor, their professional qualities, good preparation before the study starts and ability to cooperate with site personnel, and the quality of their work, also influence study quality and audit results.

Audit preparation must be based on understanding what an audit is, its objectives and the activities taken by the auditor, sponsor and monitor before, during and after the audit.

The sponsor’s representative must notify the principal investigator in writing of the upcoming audit. After receiving the written notice, the appointed auditor — directly or through the sponsor — agrees on a mutually acceptable time. Auditors usually keep a flexible schedule to avoid disrupting the site’s normal workday. The date and time should be agreed at least 2 weeks before the visit, with written confirmation.

Such contacts are useful for other reasons as well. The audit should be positive and educational for the investigator, staff and monitor. The auditor therefore prepares a work plan and shares it with stakeholders. The audit-notice letter includes the auditor’s name(s), a brief description of objectives, and a list of documents to be reviewed. To save time and avoid misunderstandings, the investigator must know exactly which documents to prepare. Additional time may be needed to retrieve source documents from the hospital archive, especially if some patients have already completed or prematurely discontinued the study. Source documents must be available for every patient who has participated in the study at that site.

The letter also lists the facilities to be inspected (patient examination rooms, drug storage areas, local lab, drug-administration area, etc.) and the site personnel whose attendance is expected, plus workspace arrangements for the auditor.

Before the audit, the monitor/CRO may conduct preparatory activities. The monitor may also attend the audit in a support role — handling logistics, introductions, explaining roles, assisting with translation. It helps if the auditor meets the monitor before the site visit to learn about the site. The monitor may have recent information that helps the auditor avoid misinterpreting data.

Audit preparation is usually handled by QA in cooperation with the monitor and/or project lead. Preparation starts in the monitor’s office (checking/preparing study files) and continues at the site. During the pre-audit visit, all aspects of the study that will be audited are reviewed — so this visit is essentially a “pre-audit” in scope. The pre-audit visit does not always happen; it may not be in the CRO’s SOP or the study budget. Its purpose, besides checking, is to prepare the site for the auditor — confirming the schedule, meeting rooms, document accessibility, etc. Issues found are discussed with site staff and resolved immediately or documented in detail.

Still, probably the best approach is to run the study as carefully and conscientiously as possible from the start — then no “special” audit preparation is needed.

Auditor preparation for the visit. The auditor must prepare for the visit, especially if it is the first on this study. There will already be a large number of study-specific documents: protocol, the overall scheme of interactions (sponsor — investigator — central lab, etc.), product and safety information, and possibly sponsor/CRO SOPs. The auditor may request copies of documents from the sponsor’s file — SAE reports, monitor reports, information letters/protocol amendments, team list. The auditor then notifies the investigator in writing.

Audit of the site. Work starts with introductions. Ideally, the Principal Investigator introduces the team, describes the site structure and the organization of work. The auditor in turn explains the goals, tasks and procedure. You can help the auditor — for instance by suggesting the most convenient work order from your perspective: labs and other facilities first, then documentation. The intro may include an active interview of the PI and colleagues — both to get acquainted and to begin actively checking their knowledge and understanding of the protocol and GCP rules.

The order of document review may vary but is not crucial.

Even after thorough monitor review and a pre-audit visit, there has never been an audit with zero findings. We’ll focus on audit-specific “findings.”

The auditor may start with the Investigator’s File, paying special attention to the sections with regulatory authorizations, protocol approvals and amendments, and the Ethics Committee section.

The auditor may also visit the local Ethics Committee. Questions of interest can also be verified from the documentation in the Investigator’s File. What interests the auditor at the EC? Existence of EC SOPs describing meeting frequency, EC composition and approval, informing investigators of changes, document submission, meeting minutes and storage, decision-making options, approval templates, criteria for repeat meetings. Availability of documents confirming EC review of this study (submission letter, meeting minutes, approval letter, EC correspondence). Findings include missing copies of EC submission letters, missing safety information. Local ECs are often based at the same institution as the trial, and the PI or team member may be an EC member — this is allowed, but they cannot influence the EC’s decision. Auditors sometimes find missing notes in meeting minutes that the investigator-member did not vote.

Adherence to ethical principles by the investigator is thus checked in monitoring and in audits. Informed-consent forms (100%) are reviewed: not only the fact of the signed consent for every patient currently in the study, but the correctness of the process. Key points: corresponding entry in the source documentation; the date and signature of the patient are written by the patient; the section with contact persons (and their phone numbers) for the patient in case of questions about their rights or urgent health complaints is filled in.

All these aspects are monitored by monitors, but auditors often find related errors.

Another important point is timely communication of new information about the investigational product to patients through updated consent forms. The investigator may have done everything on time, the monitor may have verified it, but the auditor is “clairvoyant” — a common finding is the absence of source-document entries showing that the patient was informed of the new information and signed the new consent. Sometimes the patient originally signed a version that was already/not yet in force when they enrolled.

CRFs and query responses (if any) are verified against source documents — essentially a repeat of the monitor’s verification. The percentage checked depends on the auditing company’s SOP and enrollment (e.g. 25% if ≥10 patients enrolled), though up to 100% may be checked. Typically the first-enrolled patients are checked (first patients, first experience, highest error probability). Particular attention is paid to patients with SAEs and/or protocol deviations. Typical errors include missing data in the CRF — on concomitant therapy, concomitant diseases, adverse events that are in the source document.

Some AEs are rated “non-serious” by investigators when a closer look shows otherwise. Investigators still may not interpret certain events as “other important medically significant events” — one category of Serious Adverse Events (SAEs) — with all the attendant reporting requirements. Helpful criteria to prevent more severe consequences: drug therapy was prescribed (e.g. for an asthma attack) or surgical intervention was performed (e.g. repeat laparotomy for purulent peritonitis) — such may fall under “other important medically significant event.” These cases require individual discussion; we want to highlight this SAE type, but not imply that every asthma attack is an SAE.

There have been cases where investigators, wanting to look “picture-perfect,” completely rewrote CRF pages and destroyed originals before the audit. It is better to have a CRF page with several visible corrections — which truly reflects documentation rules — than to “clean-copy” pages if individual CRF pages serve as source documents.

As for SAE reports, besides accuracy and completeness, the auditor must confirm that the severity and causality information in the source document, SAE report and CRF match. Compliance with SAE reporting procedures is also checked — reporting timelines (most SAEs — 24 hours from awareness or the next business day), and the availability of contact lists for SAE reporting (phone and fax).

A key area in any study is investigational-product handling — starting from receipt and storage. The product must be stored under appropriate conditions (temperature, humidity, light), in a secure, access-controlled location. Example: the investigational product was stored at −70°C; the fridge was in a room with a metal door, complex locks and alarm, accessible by two people. Still, the auditor made a comment on storage safety — at the moment of the auditor’s visit the investigator had left the lock on the fridge open. The pharmacist explained the room was the safest in the hospital and the fridge was unlocked to show the contents quickly, but the auditor recorded the finding anyway — the fridge lock must be closed.

Auditors verify the investigational-product prescription (doses, schedule). If the product requires special preparation, that is checked too. Example: the product had to be prepared in a laminar-flow hood meeting BSL-2. Neither the monitor nor the sponsor noticed during site selection or study conduct that the hood on site was only BSL-1. Only during the audit did this come out, and the study was paused until a compliant hood was purchased.

Every step of drug handling must be thoroughly documented. Sometimes sponsors provide less-than-convenient forms/journals. It is better to understand how to fill them out before the study starts. If questions arise later, discuss them with the project monitor promptly.

The investigator’s knowledge of the randomization procedure and the correct assignment of patients to treatment groups is verified, as well as access to and integrity of randomization envelopes.

Clinical trials must use certified equipment that is periodically serviced and calibrated. Investigators often “overlook” these procedures.

Documentation related to laboratory services (central and/or local labs) is audited — normal values, licenses, external quality-control program certificates — along with the fact of investigator review of lab results (date and signature on the lab form). Errors include date mismatches or missing lab documents.

The monitor’s work is separately reviewed — visit frequency, documentation of study status, document verification, issue identification and resolution — i.e. whether the study was properly overseen. Timely and correct documentation of every training/meeting with investigators is important. Its absence signals insufficient continuity between monitor and investigator.

Detecting false data. Avoiding all errors in a clinical study is hard. With enough effort they can be minimized. The greatest danger is “false” data — something sponsors fear most, especially if it significantly affects safety or efficacy conclusions, regardless of intent. The FDA’s 1993 “Guide to Detecting Fraud in Clinical Trial Inspections” lists three types of false data:

1. Altered data — obtaining inaccurate data or altering previously obtained data, e.g. unblinding treatment codes or altering lab data.

2. Omitted data — failure to report data that may influence results, e.g. failure to report or underestimation of SAEs.

3. Fabricated data — presenting fictitious information or results without doing the actual work, e.g. entering BP values or physical-exam findings without actually performing them.

We remember the “presumption of innocence.” The auditor seeks to prove the investigator’s “innocence,” must be benevolent and unbiased, but also aware that unintentional errors occur. A core task is to confirm that data are credible, accurate, complete and obtained per protocol. One example of an “unintentional” error: a patient missed a visit, so pulse and BP were not measured; the investigator entered values identical to the previous visit, since they had been stable. This reflects insufficient understanding of GCP.

There are no strict rules for spotting fraud. Even after analyzing prior cases, simple logic does not always give a clear answer. As Sherlock Holmes put it, “A very careful analysis of all circumstances and details is required.”

Some useful approaches:

— define what the source document is and what it should contain;

— check for inconsistencies in data collected by different site staff;

— verify whether there is documentation not presented for review, and why;

— pay more attention to corrected/flagged data;

— check for trends or deviations from expected results;

— compare handwriting and ink characteristics;

— compare event sequences, visit dates, examination schedule;

— check whether patients could actually have visited on the stated dates (e.g. weekend, 9 PM on Dec 31);

— verify whether the site has the equipment and trained staff for certain tests.

Close-out meeting with investigators. The auditor’s final step at the site is the close-out meeting — the auditor explains what was done during the audit, what findings were identified, and gives recommendations. Matters requiring immediate action (e.g. an SAE discovered during the audit) are discussed separately. Timing for the full report is set; often a full report is not sent to the site, but a “list of issues” and recommendations are provided. During the audit and at the close-out meeting, active dialogue is recommended — not passive acceptance of the findings. Sometimes the auditor did not find a document or data and should be helped. On the other hand, do not keep arguing if your arguments were not accepted — the auditors are qualified people striving for objective, unbiased assessment; one audit goal is to train investigators in ICH GCP.

Audit report. The audit conclusion states whether the conduct of the clinical study at the site is adequate and acceptable against sponsor, protocol, ICH GCP and local legislation requirements.

ICH GCP states the audit report is “a written evaluation by the sponsor’s auditor of the results of the audit” (clause 1.8) and that “audit findings shall be formalized” (clause 5.19.3.c). Broadly, an audit report is a fact-based document describing the observations and shortcomings of the clinical study identified during its assessment against ICH GCP.

Although audit reports vary with audit type and sponsor, all conform to a common standard. The sponsor must have an SOP describing the form and content. The report fully describes procedure, objectives, methods and tasks, all findings and recommendations, and the distribution list. Findings in the report must match those presented at the close-out meeting. Each finding is described in detail to demonstrate why it is a deviation from ICH GCP, with a reference to the relevant clause.

Classification of findings by importance:

— major findings — significant non-compliance with ICH GCP and major data errors — requiring urgent action. Examples: no EC approval of the study, missing patient source documents, enrolling a patient without signed informed consent, invalid results, etc.;

— minor findings — requiring action. Examples: missing date on a team member’s CV, verification findings (e.g. inaccurate date of prescription of a concomitant therapy), etc.;

— recommendations — advice that does not require action from the investigators, aimed at improving future trial quality.

The report sets a deadline for the investigator’s response on actions taken to address the findings. These actions are then included in the final audit report, which thus describes both the findings and the actions taken.

Finally, the site receives an audit certificate — a useful document that can support a site’s experience in clinical trials and play a decisive role when a sponsor considers the site for a new project.

Conclusion. An audit is an important part of clinical-trial quality assurance — a complex, interconnected process.

Source: N.V. Klimkovskaya, I.S. Firsov