Dr. Ben Locwin, Healthcare Science Advisors01.29.15
Process Analytical Technology (PAT) has been a valuable paradigm for many years, and Quality-by-Design (QbD) has been in the vernacular for about a decade in our industry. However, one of the reasons why QbD remains a hot talking point with limited actual practice-level exposure is because to perform it properly requires a high level of fluency with statistical process control, Design of Experiments (DoE), and similar methodologies. While there are still arguments about the implementation of PAT and QbD, one topic which likely has had just as much weight in ink printed about it (and pixels alight in online text) is Risk Management. The very simple reason for this is that appropriate risk management frameworks generally are fairly easy to conceptualize: Identify, assess, and document the relative risks within a facility or production process. The outputs then are ordinal ‘risks’ that can be addressed at the facility’s discretion, and per ICH Q9, have a level of resolution effort applied to them, which is commensurate with their relative risk priority.
Thus has launched hundreds of Quality Risk Management programs and paradigms all with the very simple value proposition: Assess and catalog our individual risks so that we can resolve them properly. Sounds easy, doesn’t it? Unfortunately, the tactical and practical reality couldn’t be further from that.
The purpose of my column in this issue is to specifically alert you to potential Quality Risk Management pitfalls and gaffes that you could, or might already, fall prey to. At the end of the article, it should give you pause to consider what your organization is specifically doing, and if any of the elements fall into these categories.
The pitfalls listed herein are captured from observing many sites representing large, medium, and small pharma, biopharma, cell development sites, and contract research organizations (CROs). While each type of business model provides a different platform to perform Risk Management, surprisingly, many of the pitfalls observed even across different business types are very similar. I have arranged the list in frequency order, from highest to lowest.
The Quantified Reasoning Fallacy
In order to explain this pitfall in proper application of Risk Management, I will refer to an example. If you have ever been admitted to an emergency room or visited a physician and have been asked about any pain that you’re in, they’ll tacitly or explicitly refer to a Mosby Pain Scale. An example of this instrument is shown in Figure 1.
Pain Rating Scales
Now notice that it is an interval-level scale, which has stratifications from 1-10. Perhaps, you may explain to the physician or nurse, your pain is currently at a 7 upon admission. But what does that mean exactly? And how does that really differ from a 6 or an 8? And is your perception and reporting of pain similar to another individual being asked the same question in different circumstances? Clearly there is a great deal of ambiguity in these scores, and in fact research has shown that environmental stressors, mood, and even circulating serotonin levels in individuals can influence the perception of pain. So we’re not just asking about the particular discrete injury, we’re actually getting a response that has all of the above factors confounded within it, as well as differential opinions of pain. In Figure 2, see how another iteration of the pain scale has been stratified by number pairs from 11 available selections (0 – 10) down to 7 in an attempt to capture some of the ambiguity in perception and respondent feedback.
In Figure 3, I have shown the revised universal pain scale. It has different levels translated into several languages for use around the world. Notice something far more subtle now. The interval scale is now grouped into bins of uncertainty, suggesting that the interpreter of your response should not take a 1 or a 2 to be meaningfully different, or a 3 from a 5, both moderate.
This is clear deference to the quantified reasoning fallacy where when we apply numbers (a score, rating, gauge, etc.) to something, it appears to give it more weight and importance. Charles Seife explores some of this in his excellent book Proofiness1. In the marketing world, there has been an incredible uptick in articles phrased like, “Read about the 7 ways you’re managing your money wrong.” Just think about all the similar articles you’ve seen recently, or by way of example, by me having called your attention to these articles you’ll be more likely to notice them in the future owing to the priming effect. Clearly in the example that I listed, there aren’t precisely 7 ways, nor 4, and not 15; it’s a continuum of sorts, but the number of click-throughs of online articles titled in these ways shows unequivocally that readers can’t get enough of the fake precision that these articles suggest.
And so it is when you are developing or performing work within your Quality Risk Management program. There have been countless arguments across industries and businesses when performing FMEAs or a quantified risk assessment as to whether an individual line-item should be scored a 6 or a 7, or some other falsely-important number. The human brain processes numbers differently than letters and words, and we just can’t seem to get enough of the importance that a scored element appears to give to us. But let’s remember an important fact: Although we develop these quantified rankings, whether 1-10, 1-5, or anything remotely similar, we are subjectively applying labels to each as to what they mean, and there is no law which suggests these rankings are universally meaningful. And so it is that many people, meetings, and companies are being fleeced by the concreteness fallacy into thinking that each of these discrete scores actually means anything. They are a useful guide as to relative ranking of priority—and hence the acronym RPN for risk priority number in FMEA—not a final arbiter as to the actual value of any item.
In fact, the takeaway for this pitfall is solved in the following manner. The level of measurement discrimination of the risk assessment ranking should be determined based on the quantity, quality, and fidelity of the available data for the particular risk assessment.
For example, when choosing to use a L,M,H scale, a 1-5 scale, a 1-10 scale, or anything else of increasing resolution, ask yourself, What is the quality of the data I am making these determinations on? If you have n=3 or some other (very) small and unreliable data set, or your data are likewise unreliably based on observer testimony and witness descriptions of events, you should never allow yourself to be duped into working within a high resolution ranking scale (1-10 or something similar); you have no repeatable or mathematically meaningful way of discriminating between a 1 and a 3, a 5 and an 8, or anything else for that matter. And to do so is only fooling yourself and your business. In Quality Risk Management, there is absolutely nothing wrong with a qualitative ranking scale and in fact, that’s about the best you can actually do in many cases with the available evidence. Please do not fall into this trap again. Now you are armed with science and logic on your side.
Proper Logical Fallacies – Inductive vs. Deductive
This pitfall is also very widespread in businesses performing or attempting to perform risk management. The way in which we improperly assess the risk of items based on how we misperceive their actual pervasiveness in the business or probability of occurring, is due to their being insufficiently understood with evidence.
Inductive reasoning. This is reasoning where a set of occurrences are observed in order to construct a model, or mental picture, of what is actually occurring. So for example, perhaps you’ve observed there to be traffic on a particular stretch of highway during your commute for the last 10 mornings between 7:00 and 8:00 a.m. Then tomorrow you take a vacation day, and while you’re in your kitchen sipping on a coffee (or tea) you, through course of habit, check the clock, and it’s about 7:15 a.m. You think to yourself, and immediately and reflexively conjure up a visual model of what the traffic might be like that you’d be sitting in at just that very moment. And you could likely be wrong. You are basing your inductive assumption on a finite data set and trying to extrapolate it to fit any given day—which it may not at all.
Deductive reasoning. This reasoning, in contrast to the above, is where you would take an overarching principle or maxim and assume that it holds sway for all observable cases. Here’s an example of this. Your company uses HEPA filters for all incoming air in controlled rooms and spaces. Therefore all the air is of sufficiently high quality. The pitfall here is that you are trying to extend your assumption to areas where you have no empirical data, and are therefore doing what we call extrapolating outside of your data’s visibility into a prediction space. As with regression plots, prediction intervals get very wide very quickly, and therefore modeling the uncertainty, because without data beyond a certain range, all we can do is make an educated guess; and look what the second word of that phrase is. Then acquaint yourself with the Space Shuttle Challenger’s likelihood of elastomer o-ring failure on the day of launch given what the actual launch-day temperatures were3. Flaws in deductive reasoning are also usually the way in which the Root Cause Analysis technique of ‘5 Whys’ is performed improperly. Without considering an array of possibilities for each successive element, it’s easy to linearly always arrive at a trivial, and incorrect, root cause such as human error. For example:
A valve leaked
Why? Valve elastomer was found pinched.
Why? Installation issue.
Why? Human error.
It’s so easy to draw these false conclusions that they’re ubiquitous in the industry and beguiling for investigators to end their work there. But did they gather evidence as to where exactly the elastomer damage was, which may help in trying to interpret the failure mode? Was it damage due to a sticking actuator? Were the steam pressures and temperatures within specification or did temperature and pressure excursions lead to failed elastomer material? What alternative theories could explain the observations? Clearly good evidence brings the scientific method into the discussion and averts choosing the wrong logic path.
For these logical fallacies, ensure that you have a balance of scientifically-sound hypotheses and principles for your decision making, as well as a sufficient amount of robust evidence, i.e. data, to make your decisions, so you aren’t squarely at the mercy of one or the other logical pitfalls. The Scientific Method itself is a synthesis of logic of both a deductive nature and an inductive nature.
Choosing CAPA
At least once there is a list of prioritized risks, the selection of mitigation activities such as some form of CAPA brings about the ability to reduce the risks to the business. After all, the point of Risk Management isn’t to catalog risks, but to resolve those risks, which are significant to the business. So based on the fallacies in reasoning discussed above that lead to erroneous root causes and improper CAPA, if we instead bring the focus squarely on the right sorts of CAPA, you can at last deliver the business value that Risk Management is supposed to deliver.
Let’s say you go to the ER with arm pain. A few superficial questions are asked of you, where you recount that it could be because you raked the lawn yesterday and strained your arm. Even though the actual root cause is that there is an underlying cardiac risk at-play, you address the superficial symptom by taking a pain reliever or ice your arm to reduce the pain. Significantly, you haven’t actually addressed anything, and could be faced with immediate and serious consequences. And so it is with improperly-scoped CAPA. When they are developed based on the wrong hypotheses because of incorrect and unscientific assumptions, they don’t do what CAPA are first and foremost supposed to do, which is prevent recurrence. So when it comes time to apply your valuable resources to address those risk items at the top of your prioritized list, make sure you’re doing so in a way that’s relevant and not just a superficial fix based on a linear and incorrect investigation and analysis of the situation.
The very concept of Risk Management assumes in its underlying framework that there are unknown risks of low probability. So you make corrections to your processes and systems in such a way that they become more robust and less susceptible to the unexpected. Think for example about the glove boxes used for radioactive or highly infectious material. These solutions eliminate the need for layers of procedural complexity to maintain containment and also reduce the potential for unintended error and potentially very serious consequences. In that way, they design-out many potential failure modes in one elegant solution and make it easier to do the job right than to do it wrong. And that’s how we set employees up for success and begin focusing on the real risks to our facilities.
References
Dr. Ben Locwin
Healthcare Science Advisors
Ben Locwin, PhD, MBA is President of Healthcare Science Advisors and writes the Clinically Speaking column for Contract Pharma. He is an author on a wide variety of scientific topics. He is also a frequent speaker and consultant for a variety of industries including behavioral and psychological, food and nutrition, pharmaceutical, and academic. Follow him at @BenLocwin.
Thus has launched hundreds of Quality Risk Management programs and paradigms all with the very simple value proposition: Assess and catalog our individual risks so that we can resolve them properly. Sounds easy, doesn’t it? Unfortunately, the tactical and practical reality couldn’t be further from that.
The purpose of my column in this issue is to specifically alert you to potential Quality Risk Management pitfalls and gaffes that you could, or might already, fall prey to. At the end of the article, it should give you pause to consider what your organization is specifically doing, and if any of the elements fall into these categories.
The pitfalls listed herein are captured from observing many sites representing large, medium, and small pharma, biopharma, cell development sites, and contract research organizations (CROs). While each type of business model provides a different platform to perform Risk Management, surprisingly, many of the pitfalls observed even across different business types are very similar. I have arranged the list in frequency order, from highest to lowest.
The Quantified Reasoning Fallacy
In order to explain this pitfall in proper application of Risk Management, I will refer to an example. If you have ever been admitted to an emergency room or visited a physician and have been asked about any pain that you’re in, they’ll tacitly or explicitly refer to a Mosby Pain Scale. An example of this instrument is shown in Figure 1.
Pain Rating Scales
Now notice that it is an interval-level scale, which has stratifications from 1-10. Perhaps, you may explain to the physician or nurse, your pain is currently at a 7 upon admission. But what does that mean exactly? And how does that really differ from a 6 or an 8? And is your perception and reporting of pain similar to another individual being asked the same question in different circumstances? Clearly there is a great deal of ambiguity in these scores, and in fact research has shown that environmental stressors, mood, and even circulating serotonin levels in individuals can influence the perception of pain. So we’re not just asking about the particular discrete injury, we’re actually getting a response that has all of the above factors confounded within it, as well as differential opinions of pain. In Figure 2, see how another iteration of the pain scale has been stratified by number pairs from 11 available selections (0 – 10) down to 7 in an attempt to capture some of the ambiguity in perception and respondent feedback.
In Figure 3, I have shown the revised universal pain scale. It has different levels translated into several languages for use around the world. Notice something far more subtle now. The interval scale is now grouped into bins of uncertainty, suggesting that the interpreter of your response should not take a 1 or a 2 to be meaningfully different, or a 3 from a 5, both moderate.
This is clear deference to the quantified reasoning fallacy where when we apply numbers (a score, rating, gauge, etc.) to something, it appears to give it more weight and importance. Charles Seife explores some of this in his excellent book Proofiness1. In the marketing world, there has been an incredible uptick in articles phrased like, “Read about the 7 ways you’re managing your money wrong.” Just think about all the similar articles you’ve seen recently, or by way of example, by me having called your attention to these articles you’ll be more likely to notice them in the future owing to the priming effect. Clearly in the example that I listed, there aren’t precisely 7 ways, nor 4, and not 15; it’s a continuum of sorts, but the number of click-throughs of online articles titled in these ways shows unequivocally that readers can’t get enough of the fake precision that these articles suggest.
And so it is when you are developing or performing work within your Quality Risk Management program. There have been countless arguments across industries and businesses when performing FMEAs or a quantified risk assessment as to whether an individual line-item should be scored a 6 or a 7, or some other falsely-important number. The human brain processes numbers differently than letters and words, and we just can’t seem to get enough of the importance that a scored element appears to give to us. But let’s remember an important fact: Although we develop these quantified rankings, whether 1-10, 1-5, or anything remotely similar, we are subjectively applying labels to each as to what they mean, and there is no law which suggests these rankings are universally meaningful. And so it is that many people, meetings, and companies are being fleeced by the concreteness fallacy into thinking that each of these discrete scores actually means anything. They are a useful guide as to relative ranking of priority—and hence the acronym RPN for risk priority number in FMEA—not a final arbiter as to the actual value of any item.
In fact, the takeaway for this pitfall is solved in the following manner. The level of measurement discrimination of the risk assessment ranking should be determined based on the quantity, quality, and fidelity of the available data for the particular risk assessment.
For example, when choosing to use a L,M,H scale, a 1-5 scale, a 1-10 scale, or anything else of increasing resolution, ask yourself, What is the quality of the data I am making these determinations on? If you have n=3 or some other (very) small and unreliable data set, or your data are likewise unreliably based on observer testimony and witness descriptions of events, you should never allow yourself to be duped into working within a high resolution ranking scale (1-10 or something similar); you have no repeatable or mathematically meaningful way of discriminating between a 1 and a 3, a 5 and an 8, or anything else for that matter. And to do so is only fooling yourself and your business. In Quality Risk Management, there is absolutely nothing wrong with a qualitative ranking scale and in fact, that’s about the best you can actually do in many cases with the available evidence. Please do not fall into this trap again. Now you are armed with science and logic on your side.
Proper Logical Fallacies – Inductive vs. Deductive
This pitfall is also very widespread in businesses performing or attempting to perform risk management. The way in which we improperly assess the risk of items based on how we misperceive their actual pervasiveness in the business or probability of occurring, is due to their being insufficiently understood with evidence.
Inductive reasoning. This is reasoning where a set of occurrences are observed in order to construct a model, or mental picture, of what is actually occurring. So for example, perhaps you’ve observed there to be traffic on a particular stretch of highway during your commute for the last 10 mornings between 7:00 and 8:00 a.m. Then tomorrow you take a vacation day, and while you’re in your kitchen sipping on a coffee (or tea) you, through course of habit, check the clock, and it’s about 7:15 a.m. You think to yourself, and immediately and reflexively conjure up a visual model of what the traffic might be like that you’d be sitting in at just that very moment. And you could likely be wrong. You are basing your inductive assumption on a finite data set and trying to extrapolate it to fit any given day—which it may not at all.
Deductive reasoning. This reasoning, in contrast to the above, is where you would take an overarching principle or maxim and assume that it holds sway for all observable cases. Here’s an example of this. Your company uses HEPA filters for all incoming air in controlled rooms and spaces. Therefore all the air is of sufficiently high quality. The pitfall here is that you are trying to extend your assumption to areas where you have no empirical data, and are therefore doing what we call extrapolating outside of your data’s visibility into a prediction space. As with regression plots, prediction intervals get very wide very quickly, and therefore modeling the uncertainty, because without data beyond a certain range, all we can do is make an educated guess; and look what the second word of that phrase is. Then acquaint yourself with the Space Shuttle Challenger’s likelihood of elastomer o-ring failure on the day of launch given what the actual launch-day temperatures were3. Flaws in deductive reasoning are also usually the way in which the Root Cause Analysis technique of ‘5 Whys’ is performed improperly. Without considering an array of possibilities for each successive element, it’s easy to linearly always arrive at a trivial, and incorrect, root cause such as human error. For example:
A valve leaked
Why? Valve elastomer was found pinched.
Why? Installation issue.
Why? Human error.
It’s so easy to draw these false conclusions that they’re ubiquitous in the industry and beguiling for investigators to end their work there. But did they gather evidence as to where exactly the elastomer damage was, which may help in trying to interpret the failure mode? Was it damage due to a sticking actuator? Were the steam pressures and temperatures within specification or did temperature and pressure excursions lead to failed elastomer material? What alternative theories could explain the observations? Clearly good evidence brings the scientific method into the discussion and averts choosing the wrong logic path.
For these logical fallacies, ensure that you have a balance of scientifically-sound hypotheses and principles for your decision making, as well as a sufficient amount of robust evidence, i.e. data, to make your decisions, so you aren’t squarely at the mercy of one or the other logical pitfalls. The Scientific Method itself is a synthesis of logic of both a deductive nature and an inductive nature.
Choosing CAPA
At least once there is a list of prioritized risks, the selection of mitigation activities such as some form of CAPA brings about the ability to reduce the risks to the business. After all, the point of Risk Management isn’t to catalog risks, but to resolve those risks, which are significant to the business. So based on the fallacies in reasoning discussed above that lead to erroneous root causes and improper CAPA, if we instead bring the focus squarely on the right sorts of CAPA, you can at last deliver the business value that Risk Management is supposed to deliver.
Let’s say you go to the ER with arm pain. A few superficial questions are asked of you, where you recount that it could be because you raked the lawn yesterday and strained your arm. Even though the actual root cause is that there is an underlying cardiac risk at-play, you address the superficial symptom by taking a pain reliever or ice your arm to reduce the pain. Significantly, you haven’t actually addressed anything, and could be faced with immediate and serious consequences. And so it is with improperly-scoped CAPA. When they are developed based on the wrong hypotheses because of incorrect and unscientific assumptions, they don’t do what CAPA are first and foremost supposed to do, which is prevent recurrence. So when it comes time to apply your valuable resources to address those risk items at the top of your prioritized list, make sure you’re doing so in a way that’s relevant and not just a superficial fix based on a linear and incorrect investigation and analysis of the situation.
The very concept of Risk Management assumes in its underlying framework that there are unknown risks of low probability. So you make corrections to your processes and systems in such a way that they become more robust and less susceptible to the unexpected. Think for example about the glove boxes used for radioactive or highly infectious material. These solutions eliminate the need for layers of procedural complexity to maintain containment and also reduce the potential for unintended error and potentially very serious consequences. In that way, they design-out many potential failure modes in one elegant solution and make it easier to do the job right than to do it wrong. And that’s how we set employees up for success and begin focusing on the real risks to our facilities.
References
- Seife, C. (2011). Proofiness: How you’re being fooled by the numbers. Penguin Books.
- Locwin, B. (2013). Quality risk assessment and management strategies for biopharmaceutical companies. BioProcess International, 11(11), 52-57.
- Vaughn, D. (1996). The Challenger launch decision: Risky technology, culture, and deviance at NASA. University of Chicago Press.
Dr. Ben Locwin
Healthcare Science Advisors
Ben Locwin, PhD, MBA is President of Healthcare Science Advisors and writes the Clinically Speaking column for Contract Pharma. He is an author on a wide variety of scientific topics. He is also a frequent speaker and consultant for a variety of industries including behavioral and psychological, food and nutrition, pharmaceutical, and academic. Follow him at @BenLocwin.
