Dr. Melvyn Langford, formerly of the UK National Health Service, has written an article debunking the “5 x 5” risk matrix approach and, by extension, the concept of Risk Priority Numbers (RPN). The US aerospace industry, in particular, reinforces the use of 5 x 5 matrices and RPNs despite multiple mathematicians, researches and scientists consistently decrying the practice as mathematically fallacious.
Writing in an article published in Health Estate Journal (signup required), Dr. Langford discusses the mathematical flaws in the approach of assigning numbers for risk likelihood and consequence, and then applying a mathematical function such as multiplication to them. While the 5 x 5 approach appears to have been developed in the 1980’s by the US Dept. of Defense, it was then denounced by the very same DoD in the early 2000’s:
There is a common tendency to attempt to develop a single number to portray the risk associated with a particular event. This approach may be suitable if both probability/likelihood (probability) and consequences/impacts have been quantified using compatible cardinal scales or calibrated ordinal scales whose scale levels have been determined using accepted procedures (e.g., Analytical Hierarchy Process). In such a case, mathematical manipulation of the values may be meaningful and provide some quantitative basis for the ranking of risks.
In most cases, however, risk scales are actually just raw (uncalibrated) ordinal scales, reflecting only relative standing between scale levels, and not actual numerical differences. Any mathematical operations performed on results from uncalibrated ordinal scales, or a combination of uncalibrated ordinal and cardinal scales, can provide information that will at best be misleading, if not completely meaningless, resulting in erroneous risk ratings.
Oxebridge has argued that such methods are flawed, but this has not stopped ISO 9001 and AS9100 auditors from demanding that certified companies adopt them as part of “risk-based thinking.” Oxebridge has stated that while the methods are mathematically fallacious, they are “better than doing nothing” regarding risk. Dr. Langford’s article shows, however, that such 5 x 5 calculations can actually lead to “rank reversal,” or instances where an RPN may result in the opposite conclusion when compared to calculations using calibrated data. Dr. Langford concludes that approaches such as the 5 x 5 matrix are, therefore “misleading and completely meaningless.”
Given that RPN methods are used to rank risks that may impact on human health or the likelihood of disastrous product defects, “rank reversal” is particularly chilling.
Mathematician Dr. Dennis Wheeler wrote against RPN usage in an 2011 article in Quality Digest. In that piece, Dr. Wheeler wrote:
…any attempt to use RPNÂ values is an exercise in absurdity. Their use in the same room with a mathematician will tend to produce a spontaneous explosion. They are utter and complete nonÂsense.
Such mathematical realities have not stopped ISO certification body auditors and supply chain engineers from imposing the approaches on companies building aircraft, medical devices, or automobiles. While the ISO 9001 standard does not call out RPN or 5 x 5 explicitly, the AS9100 standard implies they are required. In clause 8.1.1 “Operational Risk Management,” AS9100 requires:
The organization shall plan, implement, and control a process for managing operational risks to the achievement of applicable requirements, which includes as appropriate to the organization and the products and services … definition of risk assessment criteria (e.g., likelihood, consequences, risk acceptance.)
Some aerospace prime manufacturers mandate the use of RPN and/or 5 x 5 risk analysis matrices on their suppliers, further exacerbating the problem.
The solution is not simple, however, and requires that data used in risk analysis be comprised of calibrated, quantitative data, rather than qualitative guesses of likelihood and consequence. This is feasible in industries where such data is readily available or routinely generated, such as pharmaceutical trials or aircraft wind tunnel testing, but not in the vast majority of applications where no such data is possible.
Accreditation bodies and the International Accreditation Forum have provided no guidance on this issue, and appear content to allow CB auditors to continue to mandate such approaches, despite having been mathematically debunked.
Oxebridge utilizes the RPN concept in its “COTO Log” approach to satisfying ISO 9001’s “risk-based thinking” requirements, but warns clients the results may be misleading. ISO 9001 merely requires the user to “think” about risk, but then publishes the standard for use in third-party objective auditing, where “thinking” must be supported by evidence, creating the need for some tool. Oxebridge argues that the inclusion of risk-based thinking in ISO 9001 was done purely for marketing purposes, and not developed by any serious risk management professionals, resulting in meaningless requirements that create risk rather than reduce it.
