Ben Locwin, Healthcare Science Advisors01.28.16
Metrics are a field like nothing else. As a tool they are so widely used that they are regarded as a sine qua non, or indispensable element, of business, and at the same time are incorrectly applied and misunderstood almost everywhere. As a result of the Food and Drug Administration Safety and Innovation Act (FDASIA) of 2012, the FDA was authorized to collect manufacturing quality data from pharmaceutical companies and obtain certain records from a drug manufacturer in lieu of, or in advance of, an inspection. As a part of this act, FDA also was directed to replace the previous two-year drug production inspection frequency with a risk-based inspection approach.
FDA wrote in its Request for Quality Metrics Guidance for Industry that, “Quality metrics are used throughout the pharmaceutical industry to monitor quality control systems and processes and drive continuous improvement efforts in drug manufacturing,” (2015, 3). As one of the reviewers of this document, I had witnessed and heard firsthand some of the consternation from various companies in the industry regarding the selection of good quality metrics, and how the FDA would mandate the new guidances.
Within their Request for Quality Metrics Guidance for Industry, the FDA developed a tetra-scale ordinal list of qualifications as to what companies should consider to constitute relevant and valuable quality metrics: “Metrics should be (1) objective, (2) subject to inspection under section 704 of the FD&C Act, and (3) valuable in assessing the overall state of quality of the product and process, commitment to quality by the manufacturer, and the health (i.e., effective functioning) of the associated PQS [author’s note: pharmaceutical quality system], while (4) avoiding any undue reporting burden.”
Additionally, the metrics don’t tell the FDA the full story about organizations’ approaches to proper quality practices, they also have identified that “senior management commitment to quality is an important factor in evaluating the overall health of the PQS and quality culture.”
Difficulty Building Consistency in the Industry
One of the ways I have seen the discussions about proper quality metrics go awry is that there is an assumption made that all members of every organization’s quality unit are equivalently-qualified to discuss, determine, and report on quality metrics.
Unfortunately, with few qualification programs to ensure that all quality representatives understand actual quality to the same level of competence, there is wild disagreement over what sorts of metrics should be tracked and to what level. This leads to an incredible amount of variation in the industry in how quality units establish their principles and demonstrate that they are adhering to them. In fact, a significant source of industry push-back at the third-quarter 2015 FDA-industry meeting on this topic was that collecting and reporting the new metrics to standard would be costly and time-consuming. While at some level changing operational metrics definitions necessarily requires effort to be applied and can highlight until-now unreconciled production issues, to not agree to consistency in approach and reporting of quality metrics is being satisfied that the status quo is acceptable.
Moving the Industry Forward As An Evolution
The changes to proper metrics requirements is a way in which FDA can put into practice an approach to allow inspectors to assess a site’s ability to develop and/or produce high-quality medicinal products. More subtly than this, and something which the FDA never directly realized they were unifying, is that these approaches to metrics, when operationalized consistently across all of the industry, would directly encourage the use of quality-by-design and continuous process improvement. The FDA’s plan also included a stated goal that organizations with positive inspection reports would merit less frequent future re-inspections and a reduced reporting requirement for post-approval manufacturing changes. This results-based inspections paradigm is another overt nod to a larger principle of quality risk assessment and management (QRM). In this way, the spirit of ICH Q9 is formalized because the levels of effort applied to businesses would in fact be commensurate with the relative levels of risk identified. It is not in a business’ best interest—nor is it best quality practice—to try to mitigate every risk identified; individual risks need to be properly prioritized, and those that pose the greatest ordinal potential hazard are the focus of improvement activities.
Another substantial effort to help align quality in the industry that I have been a part of firsthand is an approach being developed through Xavier University and PricewaterhouseCoopers. Marla Phillips, director of Xavier Health at Xavier University commented that, “The Xavier University/PwC Metrics Initiative identified proactive metrics that assess risk to product quality across the Total Product Lifecycle in order to drive the right-first-time mindset as close to the initial days of development as possible.” There is also a focus on the behavioral elements of quality—those Critical-to-Quality behaviors that can lead to, or prevent, cultural risks to the process of drug production.
Beginning in mid-2014, but tracing a history back to FDASIA 2012, Xavier University launched an initiative with a team of 25 industry representatives that was characterized with the following goals:
When in the time series depicted in in the chart above are you going to act? Do you always act on quality defects when you should? Or refrain from acting at the right times? Do you have the proper standards in place to guide this? Joseph Juran correctly observed that for proper response, “Without a standard there is no logical basis for making a decision or taking action.”
Metrics should be a way to quickly appraise the health of a system and be designed in such a way that nonconforming results can easily be traced back to their source in the process. If your metrics don’t do this for you, or are measuring something else that is tangentially related to actual quality, you need to get new quality metrics.
As Ovid said, “No one desires what is unknown.” If you don’t have good quality metrics, you can’t predict what your systems and processes are going to produce next.
References and Further Reading
Ben Locwin
Healthcare Science Advisors
Ben Locwin writes the Clinically Speaking column for Contract Pharma and is an author of a wide variety of scientific articles for books and magazines, as well as an acclaimed speaker. He is an expert media contact for the American Association of Pharmaceutical Scientists (AAPS) and a committee member for the American Statistical Association (ASA). He also provides advisement to many organizations for a range of business, healthcare, clinical and patient concerns.
ben.locwin@healthcarescienceadvisors.com
FDA wrote in its Request for Quality Metrics Guidance for Industry that, “Quality metrics are used throughout the pharmaceutical industry to monitor quality control systems and processes and drive continuous improvement efforts in drug manufacturing,” (2015, 3). As one of the reviewers of this document, I had witnessed and heard firsthand some of the consternation from various companies in the industry regarding the selection of good quality metrics, and how the FDA would mandate the new guidances.
Within their Request for Quality Metrics Guidance for Industry, the FDA developed a tetra-scale ordinal list of qualifications as to what companies should consider to constitute relevant and valuable quality metrics: “Metrics should be (1) objective, (2) subject to inspection under section 704 of the FD&C Act, and (3) valuable in assessing the overall state of quality of the product and process, commitment to quality by the manufacturer, and the health (i.e., effective functioning) of the associated PQS [author’s note: pharmaceutical quality system], while (4) avoiding any undue reporting burden.”
Additionally, the metrics don’t tell the FDA the full story about organizations’ approaches to proper quality practices, they also have identified that “senior management commitment to quality is an important factor in evaluating the overall health of the PQS and quality culture.”
Difficulty Building Consistency in the Industry
One of the ways I have seen the discussions about proper quality metrics go awry is that there is an assumption made that all members of every organization’s quality unit are equivalently-qualified to discuss, determine, and report on quality metrics.
Unfortunately, with few qualification programs to ensure that all quality representatives understand actual quality to the same level of competence, there is wild disagreement over what sorts of metrics should be tracked and to what level. This leads to an incredible amount of variation in the industry in how quality units establish their principles and demonstrate that they are adhering to them. In fact, a significant source of industry push-back at the third-quarter 2015 FDA-industry meeting on this topic was that collecting and reporting the new metrics to standard would be costly and time-consuming. While at some level changing operational metrics definitions necessarily requires effort to be applied and can highlight until-now unreconciled production issues, to not agree to consistency in approach and reporting of quality metrics is being satisfied that the status quo is acceptable.
Moving the Industry Forward As An Evolution
The changes to proper metrics requirements is a way in which FDA can put into practice an approach to allow inspectors to assess a site’s ability to develop and/or produce high-quality medicinal products. More subtly than this, and something which the FDA never directly realized they were unifying, is that these approaches to metrics, when operationalized consistently across all of the industry, would directly encourage the use of quality-by-design and continuous process improvement. The FDA’s plan also included a stated goal that organizations with positive inspection reports would merit less frequent future re-inspections and a reduced reporting requirement for post-approval manufacturing changes. This results-based inspections paradigm is another overt nod to a larger principle of quality risk assessment and management (QRM). In this way, the spirit of ICH Q9 is formalized because the levels of effort applied to businesses would in fact be commensurate with the relative levels of risk identified. It is not in a business’ best interest—nor is it best quality practice—to try to mitigate every risk identified; individual risks need to be properly prioritized, and those that pose the greatest ordinal potential hazard are the focus of improvement activities.
Another substantial effort to help align quality in the industry that I have been a part of firsthand is an approach being developed through Xavier University and PricewaterhouseCoopers. Marla Phillips, director of Xavier Health at Xavier University commented that, “The Xavier University/PwC Metrics Initiative identified proactive metrics that assess risk to product quality across the Total Product Lifecycle in order to drive the right-first-time mindset as close to the initial days of development as possible.” There is also a focus on the behavioral elements of quality—those Critical-to-Quality behaviors that can lead to, or prevent, cultural risks to the process of drug production.
Beginning in mid-2014, but tracing a history back to FDASIA 2012, Xavier University launched an initiative with a team of 25 industry representatives that was characterized with the following goals:
- Identify metrics that would enable industry and FDA to proactively understand risk to product quality.
- Assess risk to product quality across the total product lifecycle to drive a mindset of designing quality-in at the source.
- Provide FDA information to assess while on inspection, and therefore, review in-context. It was felt that if these goals could be accomplished, then the resulting metrics would prove to be meaningful to both industry and FDA.2
When in the time series depicted in in the chart above are you going to act? Do you always act on quality defects when you should? Or refrain from acting at the right times? Do you have the proper standards in place to guide this? Joseph Juran correctly observed that for proper response, “Without a standard there is no logical basis for making a decision or taking action.”
Metrics should be a way to quickly appraise the health of a system and be designed in such a way that nonconforming results can easily be traced back to their source in the process. If your metrics don’t do this for you, or are measuring something else that is tangentially related to actual quality, you need to get new quality metrics.
As Ovid said, “No one desires what is unknown.” If you don’t have good quality metrics, you can’t predict what your systems and processes are going to produce next.
References and Further Reading
- FDA, Draft Guidance, Request for Quality Metrics (CDER, CBER, July 2015). http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM455957.pdf
- Xavier Health (2016). Pharma Quality Metrics Initiative. http://xavierhealth.org/pharma-quality-metrics-initiative
- Xavier University/PwC Metrics Initiative (2015). http://xavierhealth.org/wp-content/uploads/Cannon_Carri_How-to-Measure-Quality-Performance.pdf presented at PharmaLink 2015.
- The Brookings Institution (2016). Measuring pharmaceutical quality through manufacturing metrics and risk-based assessments. http://www.brookings.edu/events/2014/05/01-measuring-pharmaceutical-quality
- ISPE (2015). Quality metrics initiative. http://www.ispe.org/quality-metrics-initiative
- Parenteral Drug Association (2015). Pharmaceutical quality metrics conference. https://www.pda.org/global-event-calendar/event-detail/2015-pda-quality-metrics-conference-session-recordings
Ben Locwin
Healthcare Science Advisors
Ben Locwin writes the Clinically Speaking column for Contract Pharma and is an author of a wide variety of scientific articles for books and magazines, as well as an acclaimed speaker. He is an expert media contact for the American Association of Pharmaceutical Scientists (AAPS) and a committee member for the American Statistical Association (ASA). He also provides advisement to many organizations for a range of business, healthcare, clinical and patient concerns.
ben.locwin@healthcarescienceadvisors.com
