In September of last year, the IAQG held discussions on revisions to AS9104/3, an obscure standard called “Requirements for Aviation, Space and Defense Auditor Training, Development, Competence and Authentication.” The presentation is a stunning look at some of the problems facing the AS9100 certification scheme, and the backward thinking that plagues the IAQG when confronted with having to develop solutions.
First, let’s name the actors involved. From the aerospace industry, this included Eric Jefferies of Bell Helicopter, Deneige Fitzpatrick of Bombardier, Pete Cracknell of BAE, Yuta Kumada of MHI, Markus Krooss of Airbus, Brian Geer of Lockheed, Eric Saillard of Thales. Representing the registrar community were Brendon Hill of BSI, Jeanette Preston of Smithers, Mike McRandall of NSF-ISR. And representing Probitas, the aerospace auditor authentication body, was Rich Demary.
Keep those names in mind, because these people and companies own this.
Users = Invisible
According to the presentation, they had previously conducted surveys to uncover and rate problems in the AS91xx auditor scheme (involving AS9100, AS9110 and AS9120.) As usual, no AS9100 user organizations were polled, despite those being the companies that buy audits and have to pay to fix nonconformities. The IAQG’s official position is that its members are the “users” of the standard, despite not paying for it, and in many cases (such as Boeing) not certifying themselves to it. So the IAQG limited the polling to its members, alongside registrars, accreditation bodies and some AS auditors themselves. So they ensured the data was skewed before they even began.
And they came up with some interesting, if contradictory, statistics. The IAQG claims it found that auditors are 3.4 times more likely to write nonconformities when accreditors are watching, during witness audits. We all knew this — heck, auditors boast about this on social media — but it’s interesting to see some data behind it.
The polling of auditors gave them a more formal platform for their incessant whining about being overworked and underappreciated. That poll found that 51% of auditors — more than half — downgraded or “softgraded” nonconformities due to external pressures, such as “time management, of-site workload, client pressure.”
To recap: the CBs said their auditors were hardgrading when the accreditation bodies were looking, but auditors said they were softgrading because of workload pressure and client griping. No one seems to take accountability, but instead is blaming everyone else. In the end, we can’t trust nonconformities of any stripe, because we can’t know if they artifically hard- or soft-graded for one reason or another. The system is broken.
But then, when asked about their own auditors’ competence, the CBs and ABs ranked themselves “just fine, thank you” with a score of “high perceived competence.” Wait… what?
So the various numbers don’t quite add up.
Solutions = Problems
What’s worse, of course, is the IAQG’s planned solution. Their “big idea” to solve these conflicting problems — which really come down to auditors being terrible, and CBs not caring so long as the checks clear — the IAQG is going to develop metrics for “Auditor Performance Monitoring.” Notice it won’t be “CB Performance Monitoring,” because the CB reps are on the committee, and don’t want anyone looking over them. So they just blame their auditors.
The actual metrics are scary, and likely to get IAQG sued. For one, the IAQG intends to hardcode monitoring criteria that include “nonconformities per audit day.”
If that sounds like quotas, you are right. Worse, the IAQG intends that this data gathering will be done by the CBs themselves, all but ensuring they fuck it up. Registrars are not likely to be honest in such activities, since they could reflect poorly on their own audit pools, opening their clients up to registrar-shopping. Next, as the BSI scandal shows, registrars can tie “NCN’s per audit day” to financial goals, thus potentially opening them up to widespread civil and class action lawsuits, if not full-on FTC fraud charges.
Remember: ISO 17021-1 and AS9101, the rules for conducting such audits, demand that nonconformities be based on evidence. Those rules specifically prohibit audit results from being swayed by “threats to impartiality” such as financial goals or outside influence. Here, the IAQG appears to be the latter, while pushing CBs to engage in the former.
So, no: the IAQG cannot impose nonconformity quotas or it will invite the entire aerospace certification industry to violate the accreditation rules governing them.
So, again,… everyone’s gonna get sued.
Damn Lies = Damned Statistics
But for a moment, let’s give the IAQG the benefit of the doubt. It could be that they were trying to institute a scheme similar to round-robin proficiency testing schemes, which compare results between bodies to establish patterns and note trends. If nonconformity-per-audit metrics were gathered, and then compared between the CBs as a whole set, this might identify useful patterns. If, for example, NCN’s suddenly decreased industry-wide upon release of an updated standard, it might mean the auditors don’t understand the new standard and need training. If NCNs were wildly different between CBs, it may mean one CB is imposing quotas for financial gain — again, a possibly illegal practice — while another might be systematically soft-grading in order to gain market share. If NCN’s only go up during accreditation oversight audits, and then drop down again afterward, you know your auditors don’t know how to write nonconformities to begin with.
But that’s not what IAQG is saying. By measuring NCNs per auditor, per audit day rather than per audit (or over another portion of time), it implies quotas. The practical effect will be an artificial increase in nonconformities issued during audits, at the expense of the AS9100 user. Again, now you’re back in court or answering to fraud regulators. If an AS9100 users has to spend five minutes of labor on a bogus nonconformity driven by an artificial quota, he can argue his costs were due to fraud.
Making things worse, the IAQG then claims it will establish “targets” for such NCN metrics, again, all but admitting that this will be a quota scheme, and wholly illegal.
Upholding Complaints = Dropping Them
Another interesting metric proposed by IAQG makes sense, but will never work because of who is involved. The presentation includes two additional metrics related to CB performance: “upheld complaints” and “upheld client nonconformity appeals.”
Now let’s go back and look at that list of architects for this scheme. Recently, Probitas’ Rich DeMary was presented with a complaint that showed a certified AS9100 Lead Auditor had falsified his aerospace experience by claiming he was a “director” of a major aerospace company, when in fact he was unemployed. DeMary, covering up for the auditor, ruled that so long as the auditor didn’t lie when he submitted his application for Probitas certification, it didn’t matter what he did afterward. DeMary ignored the entire Probitas Code of Conduct in order to ensure the complaint was not “upheld.” The issue was then escalated to the IAQG, but they have refused to even acknowledge the complaint, presumably to avoid to rule against DeMary.
Because, of course, DeMary is on the IAQG committee, so fuck you and your complaints.
So the metric of measuring “upheld complaints” will never work, because the guy who dreamed it up is working to cover up complaints, and then has the support of the IAQG in doing so.
Ditto for Smither’s Jeanette Preston. It was Preston — who has no aerospace experience, by the way — who personally colluded with ANAB, a private consultant and a publishing company, to create the “SN9001” certificate scheme for “snow management companies.” (Her lack of “snow management” experience didn’t stop her there, either.) It was Preston who worked to shut down complaints related to that scheme under allegations that they violated a host of ISO 17021-1 and ISO 17011 rules.
Then there’s Brendon Hill of BSI. While Hill isn’t personally responsible, BSI is the registrar which recently ruled their internal NCN quota program was just fine, but didn’t address the evidence presented to it. From BSI’s perspective, despite emails from a Vice President openly tying the NCN quotas to BSI’s financial performance, the complaint was never “upheld,” so would simply disappear from the IAQG’s dashboard.
So, again, it’s not quite honest to talk about “upheld” complaints as a potential measurement if you are involving people who actively work to ensure any complaints against their companies are never “upheld.”
Reducing Competence = Improving Competence
The final blow for AS9104/3 is the apparent attempt by IAQG to address the fact that nobody wants to audit under their bullshit, low-paid scheme. The remaining auditors are largely ancient, post-retired folks literally in the 70s, whose only other employment options is as a Wal-Mart greeter. The younger people quit for better-paying gigs years ago, and now the remaining auditor pool is literally dying off.
When AS9100 Rev D was rolled out, even die-hard auditors gave up trying to navigate the mysteries of the revamped OASIS database, and just quit. This left a vacuum of auditors that IAQG can’t fill.
So IAQG’s answer to this is, of course, to reduce the requirements to become an aerospace auditor, because fuck you again.
This invites the perfect storm: opening the doors to unqualified auditors while imposing potentially fraudulent nonconformity quotas. This means companies will be spending more time and money addressing bogus nonconformities, and eventually drop AS9100 altogether.
Already, companies are reaching out to their suppliers and getting waivers for AS9100 demands. Ironically, the very same companies that control the IAQG — like Boeing and Lockheed — are granting these waivers, too. Even they know the IAQG is off the rails, and AS9100 descending into madness.
But, again, the IAQG doesn’t treat users as the customer, and only considers itself the customer. With that backward thinking, it can never improve, and will always resort to shady practices developed by conflicted, shady characters.
In the meantime, if your CB starts issuing bogus nonconformities driven by quotas, it may be time to call a lawyer.
[Nearly all of the members of the committee responsible for this new rollout were contacted for comment, but did not reply. Some viewed my LinkedIn profile, though, so I know they got the emails. Once again, the IAQG is ignoring stakeholders.]