As recently discussed, self-proclaimed ISO 9001 expert David Seear revealed his idea of a “process audit” is to merely select a handful of active orders, chosen during a pre-audit shop floor tour, and then audit those. Unfortunately this practice isn’t something reserved to wannabe Demings who never studied statistics, it is pretty much practiced by every auditor out there. We’ve all seen it. But it is not only NOT a process audit, it’s also a pretty horrible way to gather evidence. Here’s why.

Sampling By Divining Rod

Mr. Seear alleges his approach is a “sampling method.” Of course, it’s not, because it doesn’t look anything like this. ISO 19011 requires the sampling plan be documented, approved, and communicated to the client; “active order” auditors never do this. They think it’s so obvious, that it doesn’t even need to be defined. Since accreditation bodies never cite CBs for failing to comply with sampling rules, and apparenty they do the same thing when auditing CBs, this whole mess gets a green light, leading auditors like Seear to believe it’s just fine.

But let’s be very clear. Simply choosing a “handful” of active orders — and therefore relinquishing all non-active orders from any scrutiny — is not sampling. It is not an organized selection of candidates from a set pool for the purposes of determining compliance of the whole. It is sticking one’s finger in the air and hoping the plant air conditioner is on.

Luck Be A Lady During This Audit

This method also requires something that should never enter an auditor’s sampling plan: luck. By merely hand-picking orders that happen to be running that day, the auditor could accidentally skew the results in one direction or another. If, by luck of the draw, that day’s orders happen to be for products that are well-established, with few problems ever arising, the resulting decision on conformity of the entire QMS may be a false positive. Likewise, if — by bad luck — the process is running a difficult product that day, the resulting decision may be a damnation of the QMS without a sense of context.

If a process happens to be shut down that day — say, for maintenance or office Christmas party — then the auditor is left with nothing at all to audit, and may just skip it.

Home Off the Range

If a given manufacturing process could run different products, at different process settings or conditions, then merely auditing “active orders” fails to assess the process across its potential range.

For example, if the product is a powder, it may enter a given process as a slurry mixture, and undergo a series of drying steps, coatings, baking, sifting, and finally dumping into tote-bins. For one particular product, the settings for temperature, air flow, conveyor rate, agitation and pH may be different than for another product. Assessing only what happens to be running that will only assess if the process is effective at the particular settings for that day. It will not assess products run at higher (or lower) temperatures, speeds, etc. If the equipment or process struggles at the high end of the range, the auditor might never know.

Service Sector, Shmervice Shmector

The method entirely ignores the service sector, and assumes every process is a manufacturing process run by machines with setpoints and using “job orders” to define work. Try applying this to a hotel, or software development company, or a municipal services provider. In these industries, sampling multiple process outputs under varying conditions is critical, because those conditions may change dramatically from one to another, even day to day.

Ghillie Suits and Ninja Moves

Finally, this method allows the client to intentionally skew the results of an audit by populating processes with “easy” jobs, thereby ensuring a positive result. While this is difficult, it’s not impossible, and I have seen it done more times than I would like. The clueless auditor, full of self-importance and armed with his “solid sampling method” has no idea the entire thing is a joke, played off his ignorance. Everyone loses.

Sometimes you have to drink the soured milk.

The Un-Seear Method

Audits must consider both active orders, and inactive ones. Active orders allow for easy-to-see objective evidence, and can highlight process deficiencies easily. But a review of the records of inactive orders is critical to ensure the process is assessed across its full range, and to ensure that no funny business has been engaged to skew the audit. Sampling should be based on a defined method. Inactive orders should be examined as far back as the previous audit, not merely recent jobs and not very, very old ones.

This requires reviewing records, conducting more interviews, and trying to piece together what happened without the benefit of seeing things underway at the time. But it’s easy, once you get the hang of it.

For clients, if they see an auditor conducting “sampling” in the David Seear method, they should stop the audit and have a quick counseling session with the auditor. If the auditor pushes back, show them the door. You won’t get a good assessment of your processes, and may even get some false nonconformities.

Photo credits: NBC Universal, Hamilton Elementary via Docstoc, Wikipedia, Dreamworks.

About Christopher Paris

Christopher Paris is the founder and VP Operations of Oxebridge. He has over 30 years' experience implementing ISO 9001 and AS9100 systems, and is a vocal advocate for the development and use of standards from the point of view of actual users. He is the author of Surviving ISO 9001 and Surviving AS9100. He reviews wines for the irreverent wine blog, Winepisser.


ISO 17000 Series Consulting