Gary C. Messplay and Adele M. Kaplan Gilpin04.05.10
Among a Critical Path Initiative suite of five planned clinical trials guidances, the FDA Centers for Drug Evaluation and Research (CDER) and Center for Biologics Evaluation and Research (CBER) have released a draft guidance document on adaptive design clinical trials. FDA intends the draft guidance to inform industry on what aspects of adaptive design trials call for special consideration, when to interact with FDA in planning and conducting these trials, what to submit to FDA for review, and issues to consider when evaluating a completed study.
The Critical Path Initiative, launched in 2004, is FDA's effort to stimulate and assist modernization of the scientific process through which FDA-regulated products are developed, assessed and manufactured. Encouragement to develop, use and explore new clinical trials paradigms has been prompted by Agency recognition of the negative aspects of traditional clinical trial paradigms used in drug development. These deficits have been acknowledged by senior FDA officials with a broad variety of descriptions, such as "outdated," "cumbersome," and "often insensitive to timely detection of safety concerns." One such paradigm shift, being cautiously approached, is adaptive study design.
Advantages of adaptive designs may be improved study efficiency (e.g., shorter duration trials with fewer participants, a higher likelihood of finding an effect if one exists), or more informative trials (e.g., the acquisition of broader dose-response information, or more accurate pinpointing of what kind of patient may benefit from a therapy). Nevertheless, there also are recognized pitfalls associated with adaptive designs. These may include the potential to limit identification of gaps in knowledge, and the elimination of time to thoughtfully explore study results from earlier studies that may no longer be separate studies, but part of seamless transition studies. Additionally, the more adaptations that are planned, the more demanding such a trial may be. In a 2006 FDA conference on adaptive design, a senior Agency official acknowledged "Adaptive procedures are more complicated to design and to analyze, and in some settings are more difficult to implement," and added, "In many cases, researchers are still unaware of the option to use adaptive designs because standard statistical courses and packages do not include them, or FDA has not laid out clear guidelines on how to use them. Another reason for the trepidation may be that product developers believe that FDA has not yet demonstrated its own readiness for adaptive trial designs, endpoints and approval criteria based on data from adaptive trials." The new guidance points out the need to cautiously balance all considerations, and broadly identifies circumstances where adaptive designs should be attempted, and when it might be best to avoid them.
The new guidance defines an adaptive design clinical study as one that "includes a prospectively planned opportunity for modification of one or more specified aspects of the study design and hypotheses based on analysis of data (usually interim data) from subjects in the study." The timepoints at which the analyses are to be performed are prospectively chosen as part of the plan. These analyses can be performed either in a blinded or unblinded manner, and with or without formal statistical testing. The key is to prospectively plan the modification opportunities. Prospectivity, in this context, means that not only is the adaptation opportunity to be planned, but that it must be planned in detail, prior to the unblinded examination of data by any persons involved in planning the adaptation. It is noteworthy that this encompasses the possibility of planning a modification after the trial has begun, as long as everyone involved in the planning has been unequivocally blinded prior to, and throughout, the planning of the modification. The Agency has signaled that it intends to require documentation that such blinding was, in fact, maintained.
Also, in accord with the definition of an adaptive design clinical study, if study design is revised based on information obtained entirely from sources outside of the specific study, then this does not qualify as an adaptive design strategy. To qualify as an adaptive design clinical trial, at least some information supporting the adaptation must be derived from the study itself. This does not mean that revisions based on external data cannot be planned prospectively, only that such a revision is not considered an adaptive design modification. Finally, it should be noted that when the guidance document uses the term "interim analysis," this may have a much broader meaning than the term as used in the ICH E9 guidance on statistical principles for clinical trials. The adaptive design guidance defines the term "interim analysis" as any examination of the data obtained in a study while that study is still ongoing. It is not used to describe only cases in which there are formal between-group comparisons. Therefore, even an examination of the study data, without intent to allow the result to be used to modify the study; i.e., an "administrative look," is still considered an interim analysis for purposes of the adaptive design guidance.
The guidance demarcates situations leading to heightened versus routine levels of concern about study integrity, according to the type of data analysis used to support an adaptation. The least level of concern is related to noncomparative analyses based on blinded data. Examples of these are analyses of discontinuation rates and baseline characteristics, but may also include study endpoint aggregate event rates and variances. The highest level of concern is related to comparative analyses of unblinded results of a controlled trial that would involve study endpoints or other outcomes that might be correlated with study endpoints, i.e., the traditionally defined "interim analysis." When the analysis leading to the planned modification opportunity is an unblinded analysis, a very high level of care, scrutiny, and documentation must be applied to avoid risking irresolvable uncertainty regarding the interpretation of study results. Unplanned modifications are not addressed by the new guidance, as these are not in keeping with the planned nature of the adaptive design.
A wide range of possible study design revisions may be planned prospectively, either in the protocol or in a separate statistical analysis plan. These include changes to:
study eligibility criteria for future enrollment,
subset selection for analysis,
randomization procedure,
treatment regimens, including dose level, dose schedule, and duration of regimen,
total sample size, including the prospect of early termination,
concomitant treatments,
data collection schedule, including number of intermediate timepoints, timing of last patient observation, and duration of patient study participation,
primary endpoint, including selection among several types of outcome assessments, change of the timepoint for the endpoint, use of simple or composite endpoint, or components to be included in a composite endpoint,
selection and/or order of secondary endpoints, and
methods to analyze endpoints, including covariates included in the final analysis, statistical methods, and means of Type I error control.
The guidance notes a few important factors in determining whether to use an adaptive design. The first is whether the study is an adequate and well-controlled study ("A&WC"), intended to support a product approval, or whether it is an "exploratory study." Another is whether significant information about the treatment is already known, as this may lead to a less risky trial. A third relates to whether the adaptive design involved is of a generally well-understood type or whether it is of a less well-understood type. In general, the guidance recommends that adaptive design be used in exploratory studies, but not in A&WC studies that are designed to provide substantial evidence of effectiveness. A&WC studies may still have some adaptive elements, but not related to those elements that will support a marketing application. The guidance identifies some adaptive designs that are considered to fall within the "well-understood" category and those that fall within the "less well-understood" category.
Protocol content requirements for A&WC adaptive design studies are described in the new guidance. FDA intends to require complete documentation regarding the rationale for the design selected (including its place in the general development plan for the product), detailed justification for each and every design feature, plans to assure study integrity when unblinded analyses are planned, and Data Monitoring Committee (or other entities involved in the process) rules of operation - with description of responsibilities. The guidance further describes what constitutes adequate documentation for adaptive designs. This may involve, among other things, documentation of models and simulations quantifying statistical uncertainty in each adaptation and its impact on Type I error, study power or bias, both for single adaptations and for combinations of adaptations. Computer programs used in simulations are to be included, along with graphical flowcharts depicting different potential adaptive pathways, the probabilities of their occurrence, and the various choices for combining information from the choices. Full details of statistical methods must be provided.
Additionally, in-depth documentation related to the entities and persons conducting analyses and adaptation selection must be provided, along with written agreements with those entities and persons. The Agency indicates that a well-trusted firewall established for trial conduct beyond those established for conventional group sequential clinical trials can help provide assurance that statistical and operational biases have not been introduced.
FDA indicates that adaptive study design will likely require closer communications between sponsors and the Agency, especially for designs that are less well understood. In the early and middle period product development, the use of Type C meetings is recommended. The established communication vehicles for later stages of product development, End-of-Phase 2 (EOP2) meetings and Special Protocol Assessments (SPA), may not be appropriate in the context of adaptive design trials. SPA timelines are too short for the intensive review FDA must conduct. If a study uses less well-understood methods, then Type C or End-of-Phase 2A meetings may be more appropriate than EOP2 meetings.
Purpose of the Adaptive Design Guidance
The Critical Path Initiative, launched in 2004, is FDA's effort to stimulate and assist modernization of the scientific process through which FDA-regulated products are developed, assessed and manufactured. Encouragement to develop, use and explore new clinical trials paradigms has been prompted by Agency recognition of the negative aspects of traditional clinical trial paradigms used in drug development. These deficits have been acknowledged by senior FDA officials with a broad variety of descriptions, such as "outdated," "cumbersome," and "often insensitive to timely detection of safety concerns." One such paradigm shift, being cautiously approached, is adaptive study design.
Advantages of adaptive designs may be improved study efficiency (e.g., shorter duration trials with fewer participants, a higher likelihood of finding an effect if one exists), or more informative trials (e.g., the acquisition of broader dose-response information, or more accurate pinpointing of what kind of patient may benefit from a therapy). Nevertheless, there also are recognized pitfalls associated with adaptive designs. These may include the potential to limit identification of gaps in knowledge, and the elimination of time to thoughtfully explore study results from earlier studies that may no longer be separate studies, but part of seamless transition studies. Additionally, the more adaptations that are planned, the more demanding such a trial may be. In a 2006 FDA conference on adaptive design, a senior Agency official acknowledged "Adaptive procedures are more complicated to design and to analyze, and in some settings are more difficult to implement," and added, "In many cases, researchers are still unaware of the option to use adaptive designs because standard statistical courses and packages do not include them, or FDA has not laid out clear guidelines on how to use them. Another reason for the trepidation may be that product developers believe that FDA has not yet demonstrated its own readiness for adaptive trial designs, endpoints and approval criteria based on data from adaptive trials." The new guidance points out the need to cautiously balance all considerations, and broadly identifies circumstances where adaptive designs should be attempted, and when it might be best to avoid them.
Adaptive Trial Design and Its Key Requirements
The new guidance defines an adaptive design clinical study as one that "includes a prospectively planned opportunity for modification of one or more specified aspects of the study design and hypotheses based on analysis of data (usually interim data) from subjects in the study." The timepoints at which the analyses are to be performed are prospectively chosen as part of the plan. These analyses can be performed either in a blinded or unblinded manner, and with or without formal statistical testing. The key is to prospectively plan the modification opportunities. Prospectivity, in this context, means that not only is the adaptation opportunity to be planned, but that it must be planned in detail, prior to the unblinded examination of data by any persons involved in planning the adaptation. It is noteworthy that this encompasses the possibility of planning a modification after the trial has begun, as long as everyone involved in the planning has been unequivocally blinded prior to, and throughout, the planning of the modification. The Agency has signaled that it intends to require documentation that such blinding was, in fact, maintained.
Also, in accord with the definition of an adaptive design clinical study, if study design is revised based on information obtained entirely from sources outside of the specific study, then this does not qualify as an adaptive design strategy. To qualify as an adaptive design clinical trial, at least some information supporting the adaptation must be derived from the study itself. This does not mean that revisions based on external data cannot be planned prospectively, only that such a revision is not considered an adaptive design modification. Finally, it should be noted that when the guidance document uses the term "interim analysis," this may have a much broader meaning than the term as used in the ICH E9 guidance on statistical principles for clinical trials. The adaptive design guidance defines the term "interim analysis" as any examination of the data obtained in a study while that study is still ongoing. It is not used to describe only cases in which there are formal between-group comparisons. Therefore, even an examination of the study data, without intent to allow the result to be used to modify the study; i.e., an "administrative look," is still considered an interim analysis for purposes of the adaptive design guidance.
The guidance demarcates situations leading to heightened versus routine levels of concern about study integrity, according to the type of data analysis used to support an adaptation. The least level of concern is related to noncomparative analyses based on blinded data. Examples of these are analyses of discontinuation rates and baseline characteristics, but may also include study endpoint aggregate event rates and variances. The highest level of concern is related to comparative analyses of unblinded results of a controlled trial that would involve study endpoints or other outcomes that might be correlated with study endpoints, i.e., the traditionally defined "interim analysis." When the analysis leading to the planned modification opportunity is an unblinded analysis, a very high level of care, scrutiny, and documentation must be applied to avoid risking irresolvable uncertainty regarding the interpretation of study results. Unplanned modifications are not addressed by the new guidance, as these are not in keeping with the planned nature of the adaptive design.
What Can Be Modified?
A wide range of possible study design revisions may be planned prospectively, either in the protocol or in a separate statistical analysis plan. These include changes to:
When To Use/Not Use Adaptive Design
The guidance notes a few important factors in determining whether to use an adaptive design. The first is whether the study is an adequate and well-controlled study ("A&WC"), intended to support a product approval, or whether it is an "exploratory study." Another is whether significant information about the treatment is already known, as this may lead to a less risky trial. A third relates to whether the adaptive design involved is of a generally well-understood type or whether it is of a less well-understood type. In general, the guidance recommends that adaptive design be used in exploratory studies, but not in A&WC studies that are designed to provide substantial evidence of effectiveness. A&WC studies may still have some adaptive elements, but not related to those elements that will support a marketing application. The guidance identifies some adaptive designs that are considered to fall within the "well-understood" category and those that fall within the "less well-understood" category.
Documentation of Adaptive Design and Communications with FDA
Protocol content requirements for A&WC adaptive design studies are described in the new guidance. FDA intends to require complete documentation regarding the rationale for the design selected (including its place in the general development plan for the product), detailed justification for each and every design feature, plans to assure study integrity when unblinded analyses are planned, and Data Monitoring Committee (or other entities involved in the process) rules of operation - with description of responsibilities. The guidance further describes what constitutes adequate documentation for adaptive designs. This may involve, among other things, documentation of models and simulations quantifying statistical uncertainty in each adaptation and its impact on Type I error, study power or bias, both for single adaptations and for combinations of adaptations. Computer programs used in simulations are to be included, along with graphical flowcharts depicting different potential adaptive pathways, the probabilities of their occurrence, and the various choices for combining information from the choices. Full details of statistical methods must be provided.
Additionally, in-depth documentation related to the entities and persons conducting analyses and adaptation selection must be provided, along with written agreements with those entities and persons. The Agency indicates that a well-trusted firewall established for trial conduct beyond those established for conventional group sequential clinical trials can help provide assurance that statistical and operational biases have not been introduced.
FDA indicates that adaptive study design will likely require closer communications between sponsors and the Agency, especially for designs that are less well understood. In the early and middle period product development, the use of Type C meetings is recommended. The established communication vehicles for later stages of product development, End-of-Phase 2 (EOP2) meetings and Special Protocol Assessments (SPA), may not be appropriate in the context of adaptive design trials. SPA timelines are too short for the intensive review FDA must conduct. If a study uses less well-understood methods, then Type C or End-of-Phase 2A meetings may be more appropriate than EOP2 meetings.