TC 176 Isn’t Following Its Own Rules on Design Validation for ISO 9001

[This article was originally published in June but still remains current.]

As we watch TC 176 “design” the next revision of the standard, it’s interesting to see if they are following their own proposed methods. As you know, design validation is a part of clause 8.3 of ISO 9001:2015, so if TC 176 is designing a product, it should validate that design, right?

What Is Design Validation?

First, though, let’s understand the difference between design verification and validation within ISO 9001.

Design verification is the comparison of the design outputs (drafts, drawings, etc.) against the design inputs (requirements.) In this case, it would be the comparison of the ISO 9001 draft against the Design Specification, which is where the requirements are defined. Example: if the design requirement is to build a birdhouse for eagles, design verification would be a check to see if the dimensions of the birdhouse indicated on the drawing are big enough to fit an eagle.

Design validation is the comparison of an actual product produced by the design against the design inputs. This is typically done by creating a prototype or first piece and then performing inspections and tests on that product to see if the design can actually result in a functioning product. Example: you built the birdhouse from the drawing, and then shove a real eagle in it to see if it fits.

TC 176 claims it performs design validation on ISO 9001 drafts in its recent Design Specification (which you can read about here). That document then quotes the official definition of “design validation” as follows:

Validation: confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled.

ISO 9001’s Intended Use(s)

So if validation is assuring your design will result in a product that meets requirements for “intended use,” what is the intended use of ISO 9001?

ISO 9001:2015 presents its intended uses in two clauses: Clause 0.1 and Clause 1.0.

In Clause 1.0 it says the following:

This International Standard specifies requirements for a quality management system when an organization:

(1) needs to demonstrate its ability to consistently provide products and services that meet customer and applicable statutory and regulatory requirements, and

(2) aims to enhance customer satisfaction through the effective application of the system, including processes for improvement of the system and the assurance of conformity to customer and applicable statutory and regulatory requirements.

Clause 0.1 then adds some additional flavors to this, recapping the same points (1) and (2) above, but then adding two additional possible uses: (3) that the user organization should be able to address risks and opportunities, and (4) the user organization should be able to prove conformity, implying to third parties — meaning, of course, audits.

So, putting this all into Human English, we get:

1. To ensure the quality of products and services
2. To ensure the improvement of customer satisfaction and conformity to requirements
3. To ensure risks and opportunities can be addressed
4. To ensure the user organization can prove conformity when asked (such as during audits)

We can probably argue a bit over my summary, but I want to remind you: I’m taking this directly from ISO 9001:2015 itself. I didn’t write that, they did. But stick with me, any minor quibbles over this are not going to matter much.

Validating ISO 9001 

So, if design validation is to be done to ensure the product can meet its intended use, then any validation efforts must be done to “test” the product against those use cases above. But let’s look at what the official Design Specification calls out for their method of design validation for ISO 9001 (emphasis added by me):

Validation will be conducted at the Draft International Stage (DIS) through the review of comments from member bodies and liaisons. Member bodies and liaisons are responsible for obtaining feedback from users. Priority of comments will be given to member bodies and liaisons.

Yikes.

What TC 176 described isn’t design validation at all. As I said above, comparing the design (in this case, the draft text) against the requirements (the Design Spec) is verification, not validation. At no point does TC 176 ever perform any testing of the draft against its actual, intended uses.

TC 176’s approach only makes sense if the intended use is for people to read ISO 9001 like a book — for enjoyment and nothing else. But no matter how you interpret the use cases defined in ISO 9001 clause 0.1 or 1.0, the purpose is not to just read the document, but to implement it. Nothing about TC 176’s validation protocol addresses the implementation or actual use of the standard.

It gets worse. In the Design Spec, they discuss a “tool” that must be used to perform this validation effort, but again conflate the concept with verification:

Any validation tool that is developed will be focused on validation of any key elements of the design specification. It will not be a clause-by-clause review of changes. A simple validation tool will be provided to member bodies during the comment period. This tool will confirm whether the [draft] meets the intent of the design specification. The tool will also confirm that the changes meet agreed criteria for acceptable levels of benefit and effort.

The Working Group will assign experts who are assigned responsibility for ensuring that changes that are incorporated meet the intent of the design specification. They will do this through participation in the drafting process.

So it’s very, very clear that they are talking about design verification, not validation. They are comparing the draft against the requirements, not against its ability to produce end-use results.

Worse still, by only polling members for their “comments” on the draft text, they are inviting subjective opinions. Remember, the definition of “validation” requires that the designer produce objective evidence of validation against the intended use. Opinion polls on text don’t come close to meeting that requirement.

The Authors Don’t Know What Validation Is

I’m pretty sure the Design Specification was written mostly by Lorri Hunt, who doesn’t understand the difference between design verification and validation. In fact, in her book The ISO 9001:2015 Handbook, Hunt and her co-authors Jose Dominguez and Craig Williams don’t define design validation at all.  Instead, Hunt has only four, short paragraphs on design review, design verification, and design validation, all lumped together. She jumbles them up and then suggests they can all be done by “meetings.” She is not even interested in trying to help the reader understand the concepts and writes that “based on the planning and the complexity of the project, these review, verification and validation activities may vary.”

“May vary?” Ya think?

Hey, Lorri, isn’t that why people are buying your book, so you can explain those variances, and how to address them? Instead, she just restates the obvious and changes the subject before anyone can ask a follow-up question. (This is what she does during actual meetings, too.)

In her book, there is nothing specific to design validation: no mention of “intended use,” no discussion of prototyping or validation testing. Nothing.

So it’s not an accident that the people running the revision process for ISO 9001 contradict themselves and don’t know what design validation is. They’re also lazy, so it’s easy to understand why they took the easiest route to “tick the box” on design validation, but without actually doing it.

How To Really Validate ISO 9001

So what would real design validation of ISO 9001 look like?

First, let’s start by only addressing the two end-uses defined in ISO 9001 Clause 1.o, and ignore those in Clause 0.1. This means that the standard would have to be implemented by “test companies,” who would then assess whether or not their implementation of the draft actually resulted in some level of “assurance” that product quality was being produced under the ISO 9001 system, and that customer satisfaction and product conformity with requirements were all improved. And, per the definition of “validation,” they would have to provide TC 1767 “objective evidence” of this, not just subjective opinions.

This would require the test companies to do the following:

1. Capture information on product quality, customer satisfaction, and conformity to requirements before implementation.
2. Implement the draft ISO 9001 requirements
3. Let the system cook for an established amount of time
4. Check the data to see if there were improvements in quality, customer satisfaction, and conformity to requirements.

Now what if we add in the other two use cases defined in clause 0.1? Then, the test companies would also have to assess if after ISO 9001 implementation they saw improvements in their ability to identify risks and opportunities. That would actually be fairly simple.

But that last criterion? It means they’d have to verify the standard can produce a system that passes audits. This would be huge, since it would now require ISO 9001 to be auditable, which — in its current state — it isn’t. Things like vague requirements, ghost clauses, shoving requirements into non-auditable notes — all those would have to be fixed before publication. Auditing of the final released standard would be much, much easier, and much, much more consistent. Requests for interpretation would drop to near zero, and auditors would not have the leeway to invent shit from thin air, as they do now.

This would be a massive effort, and I get that. It would just delay the publication of a draft ISO 9001 standard by years, at least. If problems arose, they’d have to update the draft and do a new round of validation. This would not be child’s play at all. But it would be true design validation. TC 176 wouldn’t be lying to the world, as they are now.

I remember raising the issue of “risk-based thinking” with Nigel Croft. I argued that a standard should present proven, known, and pre-validated concepts, rather than invent new ones from thin air, like RBT, that have never been tested. Croft argued that we’d find out if RBT worked after the standard was published. In short, RBT was being validated by the consumers, after it was already produced. The Croft approach is to put whatever he wants in the standards, and let the world figure it out later. That’s not how design validation works.

So it’s no wonder that TC 176 doesn’t do real design validation. It’s simply too hard for their peanut brains to wrap their head around, much less execute. Plus, folks like Hunt and Dominguez would lose a lot of money while they waited additional years for the standards to be published, unable to write any books about it during the validation phase! And ISO’s executive isn’t about to let something as dull as design validation stand in the way of their annual sales.

Which sends a clear message to the ISO 9001 user community: the rules are for you unwashed peasants, not for the wealthy landowners.

Shut Up and Do It

So, yes, design validation of ISO 9001 would be a heavy lift, but it’s doable. They cannot say otherwise. And, remember: I didn’t write ISO’s requirements or definitions for design validation, they did. I’m just holding them accountable to their own text.

Meanwhile, I can prove it. The Oxebridge Q001 standard has been undergoing real validation for the past three years. We have been implementing it in test companies to work out the kinks. These include Horberg Manufacturing (Bridgeport CT), Diamond Defence (Victoria Australia), and 5X Precision (Southington CT). The latest pilot client (pardon the pun) is none other than Piper Aircraft out of Vero Beach FL.

And we’ve found kinks, too. For example, Oxebridge Q001 requires both a Quality Policy (to remain consistent with “legacy” ISO 9001) and a definition of the “Quality Culture.” Most companies have been entirely confused on what is the difference. Also, where ISO 9001 just ignores the design of processes, Oxebridge Q001 tackles this, but clients have (again) been confused about it.

Another kink is the associated audit evidence requirements, something the ISO 9001 scheme doesn’t even try to implement. Generally it’s been found to be a fine level of detail, but it’s just way too confusing for auditors. I think we can address this through training, because I’m not keen on reducing the level of audit evidence required. But a lot of things are on the table.

We hope the next versions of Q001 will try to clear these matters up.

But Oxebridge doesn’t have the financial conflicts of interest inherent in the ISO and TC 176 bubbles. We can take our time to produce a standard (and certification scheme!) that exceeds the bar of ISO 9001, and while it takes additional time, it’s not that hard since ISO’s bar is so low. We’re not in the rush to make millions like ISO is. And we don’t have pirates like Hunt and Dominguez trying to tip the scales in their personal favor.

(Before you say it, no, we don’t make any money on the standard. It’s literally given away for free.)

So performing validation of ISO 9001 is possible, yes. And it would produce a much better standard. But ISO will never adopt it, since it will directly impact its sales, and piss off dummies like Lorri Hunt who (I guess) have nothing better to do, so have to fill their days with TC 176 committee projects.

But you need to know that TC 176 is outright lying when they say they perform design validation on ISO 9001. They do not, and that’s one reason why their end product is such junk.