The COVID-19 pandemic presented a serious disruption in the conduct of clinical trials in 2020 and beyond. If not addressed appropriately, the statistical integrity of trials could be compromised. In this briefing, we discuss the challenges and risks to statistical integrity and how they can be mitigated. In particular, we discuss the major statistical principles that need to be considered when assessing the impact of COVID-19 on trials, the role of Data Monitoring Committees (DMCs) and interim analyses for risk assessments and how to handle missing or delayed assessments resulting from the pandemic in analyses and reports.

In 2020 the FDA¹ and EMA² provided guidance to help sponsors in assuring the safety of trial participants, maintaining compliance with good clinical practice (GCP) and minimizing risks to trial integrity during the COVID-19 pandemic. One of the recommendations from the EMA2 is a risk assessment of the impact of COVID-19 on trial integrity and interpretability of potential study results. In particular, this includes the major statistical principles when considering the impact of COVID-19 on trials, the role of DMCs, interim analysis for risk assessments and how to handle missing or delayed assessments resulting from the pandemic.

Major statistical considerations
1.  Take appropriate action quickly to ensure that trial data and results can remain robust.

2.  Maintain the integrity of the data already collected and document clearly how and why data are missing, along with the reasons for discontinued patients.

3.  Review the Statistical Analysis Plan (SAP) to check if it needs to be amended or at least appended to address changes such as:

• Handling missing data;
• Protocol deviations;
• Visit window definitions;
• Covariates; and
• Changes to the timing and process of data collection.

4.  Document any changes to study conduct along with the rationale for the changes:

• It is important to consult with regulatory agencies to agree to the amendments and revise the SAP and/or protocol accordingly.
• If the impact of the pandemic results in critical data being collected by alternative means (e.g. local vs central laboratories), this may result in biased estimates and increased variability in the outcomes, which may necessitate a sample size re-estimation.

Missing data
The disruption of clinical trials due to COVID-19 is likely to result in missing visits and thus some critical data is expected to be missing. This is important because missing data are a potential source of bias when analyzing clinical trial data. The extent to which missing data may lead to biased study results is influenced by many factors. Among these are the relationship between the mechanism causing the missing data, treatment assignment and outcome. A framework and options for dealing with missing data are presented below.

Missing data are quite common in clinical trials. However, only a small proportion of the total data are generally expected to be missing. If a significant proportion of the data are missing, as may occur during and after the COVID-19 pandemic, amendments may be required to introduce more sophisticated statistical techniques to address the issue. This is particularly critical if the value of the primary outcome for a subject is missing or there has been a change in the timing of data collection (a delay or advance). This may result in a reduction in the power* of the study and therefore a need to revisit the sample size (see DMCs and Interim Analysis below).

Minimizing the impact of missing data
It is important to capture, in a consistent and systematic manner, all relevant information during the study that explains the reasons for the missing data, including any relationship to COVID-19, for missing protocol-specified visits, procedures or assessments. With the appropriate information collected during the trial, a variety of different statistical methods can be applied to assess the impact of COVID-19 on the trial results. The more detailed the metadata (i.e. the reason for missingness), the more informed the assumptions behind the mechanism leading to the missing data will be. As such, these assumptions can be incorporated into the statistical model and thus improve the credibility of the study results. The following steps can be taken to mitigate the impact of missing data:

• Collect as much data as possible. When and where possible, continue to collect data, even if it is out of sync or at the wrong time points. This may have to be done through virtual visits or through the employment of telemedicine.
• Ensure that you have a mechanism for identifying missing data at the visit level and record the reasons for any subjects discontinuing the trial (or simply discontinuing the investigational medical product [IMP]). This will likely require modifications to the CRF.
• If the trial involves patient reported outcomes (PROs) or diaries, consider contacting patients and encouraging them to continue recording even if clinic visits are discontinued or delayed.
• If a patient discontinues a trial, every effort should be made to obtain the participant’s consent for the data to be used for subsequent treatments and outcomes, as per industry best practice. This preserves the ability to analyze endpoints for all participants who underwent randomization, and thus to make possible intention-to-treat inferences. This is particularly necessary when premature discontinuation of the study drug is expected to be at higher rates due to challenges with clinic visits and study drug distribution during the COVID-19 pandemic.

How much is too much missing data?
There are, unfortunately, no guidelines regarding how much missing data is too much missing data. There may be cases where 3% of missing data may generate substantial bias and reduce precision whereas satisfactory results have been observed with more than 90% missingness in simulation studies.³ Therefore, there is no proportion of missing data whereby valid results and the preservation of study power can be guaranteed. The proportion of missing data should not be used to guide decisions. The cardinal quantity is the fraction of missing information (FMI), which combines the proportion of missing data and the degree to which the missing variable is correlated with observed values.⁴

The answer to the question, how much missing data is too much missing data, is that the proportion of missing data alone is not sufficient to judge the integrity of a trial. A team experienced in the use of missing data methods is needed to address this question on a study-by-study basis. Additionally, in accordance with EMA guidance,² sponsors may consider convening an independent DMC to assess the scientific integrity of the study (see DMCs and Interim Analysis below).

Missing data framework
Assuming a trial continues, it is necessary to define the plan for dealing with the increase in missing data. Formal planning is needed in order to avoid introducing bias into treatment estimates, which could happen if the methods are chosen after seeing the data.

There is no single correct method of analysis when data are missing. All of the methods that have been developed and used to address missing data require strong assumptions that are often unverifiable. The best approach to deal with missing data in the COVID-19 situation (where the missing data are related to external circumstances beyond the control of the trial) will depend on what is missing and the assumed mechanism responsible for the missing data.⁵

There are several different methods for dealing with missing data**:

• Missing At Random (MAR): when a recorded characteristic about the participant can account partly or entirely for the differences in the data for the observed and missing cases.  For example, suppose that the trial requires diary data on daily exercise. In the current situation, it may be that older patients, who are considered to be at greater risk from COVID-19, will be less willing to take exercise outdoors than younger patients. The reluctance is related to their age and being more concerned about COVID-19 and is not related to their actual level of fitness. In statistical terms, you are conditioning on age. If it is reasonable to believe that within each age group the pattern of missing diary exercise data is likely to be similar to the pattern of observed diary exercise data, then these data can be assumed to be MAR.
• Missing Not At Random (MNAR): when the data are missing due to a factor related to the unobserved outcome measure. This is also known as non-ignorable missingness and the probability of an observation being missing depends on unobserved measurements. For example, the treatment is an immunosuppressant so subjects for whom the treatment is working are more susceptible to contracting COVID-19. Their outcome value is missing as they were hospitalized and dropped out, whereas without the COVID-19 pandemic they would have had successful treatment. In other words, the reason for being missing is related to the outcome they would have had and thus cannot be considered to be missing at random. In this case, any analysis of the study endpoints has the potential to be biased by missing data.

Methods for dealing with missing data that rely on statistical models will be required. Both MAR and MNAR methods will likely be utilized. Methods for dealing with COVID-19 related missing data include:

Assuming MAR

• Mixed Model Repeated Measures (MMRM) might be assumed to have functional distribution with a specified covariance matrix.
• Multiple Imputation (MI), in which multiple sets of plausible values for missing data are created from their model-based predictive distribution and estimates and standard errors are obtained with the use of multiple-imputation combining rules.
• Bayesian methods, in which inferences are based on a statistical model that includes an assumed prior distribution of the parameters.

Assuming MNAR

• Pattern Mixture Models8 specify the conditional distribution of the variables X 1, …, Xv given that XV is observed or missing respectively and the marginal distribution of the binary indicator for whether or not Xv is missing.
• Jump to Reference Method6 uses an MI framework to impute missing values using plausible data generated from the observed values of the reference arm (usually, although not always, the control group).

Time-to-event endpoints

• Time-to-event endpoints deal with missing data through their censoring rules. Sensitivity to new occurrences, such as the pandemic, can be assessed by varying whether such occurrences are considered events or censored observations. The choice may vary depending on whether the missing data is considered MAR or MNAR.

The choice of strategy should be fully described and justified in the statistical section of the protocol and the assumptions underlying any mathematical models employed should be clearly explained. Analysis methods that are based on plausible scientific assumptions should be used. Ultimately, a strategy that works well for one study may not be appropriate for another.

Sensitivity analysis to support robust inference
These analyses are critical to evaluate the sensitivity of the conclusions for violations of missing data assumptions regardless of the method(s) used to address the issue. This is especially the case when the number of missing values is substantial. One approach is to use pattern mixture models, which examine subgroups of participants with different patterns of missingness. Another method that is viewed favorably by some regulatory authorities is a tipping point analysis, with the goal of identifying and discussing the clinical plausibility of assumptions under which inferences change.

Estimands
ICH E9 (R1) presents an Estimands Framework to align the planning, design, conduct, analysis and interpretation of a clinical trial.⁷ This framework requires that consideration be given to how intercurrent events will be handled in the analysis. Intercurrent events are post-randomization incidents that either preclude observation of a variable and/or affects the interpretation of a variable. The variable can be an expression of how well the treatment works and can also include treatment safety and tolerability.

The COVID-19 pandemic is such an intercurrent event, as are associated sub-events such as:

• Concomitant medications;
• Treatment withdrawal without study withdrawal;
• Study discontinuation;
• Hospitalization; and
• Death.

Items 3–5 are a missing data problem as per the ICH guideline. The mechanisms and/or assumptions can be different for different intercurrent events.

DMCs and interim analysis
One of the EMA recommendations² is a risk assessment of the impact of COVID-19 on the integrity of the trial and the interpretability of potential study results. The risk assessment should be part of the trial monitoring activities and performed on aggregate and blinded data wherever possible. In some cases, a more thorough analysis may be warranted. If so, it is recommended that such analyses are reviewed by an independent DMC. This ensures that Sponsors preserve the integrity of a trial.

An unblinded data review by an independent DMC may be considered to assess the impact of missing data and changes in patient populations. In addition to the primary responsibility of subject safety, the charter of the committee should include provisions for advising on changes to trial design, sample size re-estimation, formal management of missing data and the potential for an interim efficacy or futility analysis. This may require an update to the charter.

Unplanned interim analyses and adaptations may be considered for ongoing trials to evaluate the predictive power of the study and to assess whether the trial should be stopped early rather than being modified or prolonged.² An interim analysis that was not originally planned will have an impact on the power of a study and should be carefully planned with appropriate statistical support to maintain scientific validity and credibility. The proposed analysis must be fully documented prior to being undertaken. A separate statistical team would be required to maintain blinding for clinical teams and/or CRO staff if unblinding is considered. 

* Power is the probability of rejecting the null hypothesis when it is, in fact, false.
** Missing Completely At Random (MCAR) is unlikely to be the case in the context of COVID-19 and also unlikely to be accepted by regulatory authorities so is not considered here.

References

1. US Food and Drug Administration. FDA Guidance on Conduct of Clinical Trials of Medical Products during COVID-19 Public Health Emergency. Guidance for Industry, Investigators, and Institutional Review Boards. March 2020. Updated June 3, 2020. Available at: https://www.fda.gov/media/136238/download. Accessed 28 April 2020.
2. European Medicines Agency. Guidance on the Management of Clinical Trials during the COVID-19 (Coronavirus) pandemic. Version 3. 28/04/2020. Available at: https://ec.europa.eu/health/sites/health/files/files/eudralex/vol-10/guidanceclinicaltrials_covid19_en.pdf. Accessed 28 April 2020.
3. Madley-Dowd P, Hughes R, Tilling K, Heron J. The proportion of missing data should not be used to guide decisions on multiple imputation. J Clin Epidemiol 2019;110:63–73.
4. Schafer JL. Analysis of incomplete multivariate data. Chapman & Hall, London. 1997.
5. European Medicines Agency. Guideline on missing data in confirmatory clinical trials. EMA/CPMP/EWP/1776/99 Rev. 1 2 July 2010. Available at: https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-missing-data-confirmatory-clinical-trials_en.pdf. Accessed 28 April 2020.
6. Carpenter JR, Roger J, Kenward M. Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible assumptions, and inference via multiple imputation. J Biopharm Stat 2013;23:1352–71.
7. European Medicines Evaluation Agency. ICH E9 (R1) addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical 6 principles for clinical trials. EMA/CHMP/ICH/436221/2017. 30-August-2017. Available at: https://www.ema.europa.eu/en/documents/scientific-guideline/draft-ich-e9-r1-addendum-estimands-sensitivity-analysis-clinical-trials-guideline-statistical_en.pdf. Accessed 28 April 2020.
8. Little, RJA. A class of pattern-mixture models for normal incomplete data. Biometrika, 1994; 81(3), 471 – 483

---

Timothy Victor, Ph.D. is the Global Head of Biostatistics at ICON where he provides executive biostatistics oversite of Phase I – III clinical trials. Tim has nearly 30 years industry (sponsor and CRO) and academic research experience. He is a member of the tenured faculty at the University of Pennsylvania and clinical faculty at the Philadelphia College of Osteopathic Medicine. He completed his doctoral work in clinical psychology at the University of Connecticut and applied statistics at the University of Pennsylvania. Dr. Victor therapeutic experience includes oncology, central nervous system, metabolic disorders, and vaccines. He has published on a variety areas including statistics, psychometrics, and ethics.