CMMC Assessment Process Released – Initial Concerns

The CyberAB (formerly the CMMC Accreditation Body) has released a draft of their CMMC Assessment Process (or “CAP”), which defines the official methods for conducting CMMC assessments. It’s being met with some level of concern, as it seems to simultaneously fail to address prior feedback given to the CyberAB, as well as injects new uncertainties. The document is big, and too much to unpack in one sitting, so let me take a few initial passes.

(You can grab a copy here.)

First, though, it’s worth reading the comments by Leslie Weinstein, a US Army Major and cybersecurity consultant with Deloitte. She has posted two well-reasoned commentaries on LinkedIn (here and here), but I’ll pull out a few main points:

There are so many mistakes, logical incongruencies, and references to appendices and paragraphs that do not exist that I question why this was released without undergoing a QA process.

• The CAP says that CMMC practices are to be “scored” according to the DoD NIST 800-171 self-assessment methodology, however, each applicable control receives a binary determination (MET or NOT MET)
• There seems to be A LOT of work that goes into the planning and preparing phase of a CMMC assessment, however, the cost of an assessment is supposedly determined after the majority of that activity is completed. How much money will OSCs need to pay a C3PAO just to get a quote for a CMMC assessment?
• The CAP appears to be a guide to business practices for C3PAOs more so than a guide for HOW to conduct an actual assessment. There are no actual assessment processes outlined in the CAP, only processes for how to negotiate a business contract.
• The CAP allows DIB companies to have up to 22 gaps/POA&Ms and still achieve a “conditional CMMC” certification… [then] the CAP contains a list of 52 “Limited Deficiency Correction Consideration” controls that have “limited or indirect effect on the security of the network and its data”. These controls are the DOD NIST assessment methodology controls valued at 1 point. I don’t believe the The Cyber AB realizes that these 1-point controls are the most fundamental, simple, and cost effective ways to protect #CUI. These controls should NOT be allowed to have a gap because they are the building blocks for every other control. These controls DO have a serious effect on the security of a network and its data-contrary to what the CAP says.

The latter point made by Weinstein is that a company can have major gaps in the most basic cybersecurity controls, and still achieve CMMC certification. That’s not confidence-inspiring.

Evidence Gathering

In reviewing the CAP from an assessment viewpoint (not a cybersecurity one, since I’m not a cyber expert), I have additional concerns.

First, the CyberAB ignored feedback concerning the collection of evidence. The NIST method — which has been used for self-attestation or assessments not resulting in a certification — allows for three methods of evidence evaluation: Examine, Interview, and Test. The problem is that NIST allows the Assessor to decide which to use at any given moment.

Again, the NIST approach was developed before anyone thought of piling on a certification scheme. From our decades of experience with ISO certifications, we know the industry always becomes a race to the bottom: the least restrictive (meaning “easiest & cheapest”) certification body will gain market share. The CyberAB has its head in the sand on this subject, and has created the exact nightmare scenario that will reward the poorest C3PAOs for bad auditing.

If you don’t think costs come into the picture when deciding which evidence method to use, you’re wrong (emphasis added):

This determination is made based on how the organization can accomplish the Assessment objectives in the most cost-effective mannerand with sufficient confidence to support the determination that the CUI requirements have been satisfied.

For the “Examine” method, assessors will be expected to check records; for “Interview,” they will talk to people and gather verbal testimony; and for “Test,” they will observe actual tests being run in real-time during the CMMC assessment.

Now it doesn’t take a lot of thought to know which of these will win the day. To run literal tests during the assessment is both complicated and expensive. The CAP makes this worse by layering in a lot of detail on how such tests are to be run, thus adding more costs. If the decision of whether to utilize “Test” to verify a given CMMC control is left to the Assessor, then no C3PAO will ever choose “Test.” Doing so will make their assessments more expensive than the C3PAO who opts for one of the other controls.

So you can scratch “Test” entirely. That leaves the last two.

To summarize, for “Examine,” the assessor needs to look at records and procedures, etc.; in short, documentary evidence. This one will likely be opted for to a great extent, but not exclusively. Under the ISO scheme, we see where ISO’s “New Age” departure from requiring documentation and records has got us: weak management systems that don’t ensure robust training and consistent performance. Thus, a CMMC company may not have procedures for crucial cybersecurity aspects at all, pushing the CMMC Assessor to skip “Examine” entirely. Or maybe the Assessor will review documents and records thoroughly, and all will be well. It’s not clear, but it also won’t be consistent.

But this leaves the third method, “Interview.” This is the easiest and least expensive option, since it just requires the Assessor to sit and chat with the client’s representatives. So I suspect that more than half of a CMMC assessment will be verbal interviews. That introduces the risk that client companies will simply lie during assessments, or create some spin.

There’s no requirement to use two methods, either, so that — say — Interview results would be required to be supported by either Test or Examination results. Only one is required. And it’s up to the Assessor to decide.

Therefore, the C3PAO that promises to “go easy” and rely almost entirely on Interview, ignoring Examination and Test, will gain market share. The C3PAO that tries to go hardcore, and demand Test-based evidence, will be out of business in a week.

Anonymity of Interviewees

The Interview method has another issue, made much worse by the CAP. The CyberAB, for reasons that can only be explained as rookie mistakes, insists that the interviewees remain anonymous. This is repeated throughout the CAP, but here’s one example:

The Assessment Team … takes steps to ensure and verify that confidentiality and non-attribution is addressed for interviewees so that they can speak openly without fear or concern about retribution from any member of the [client]

It’s clear that The CyberAB does not understand the most basic concepts of assessments or audits, and that certification decisions rely on objective evidence. ISO defines this as “information which can be proven true, through observation, test, or other means.” In short, all assessment evidence must have a component to it that can be verified by a third party. Preferably, by someone not present during the assessment itself. This ensures trust, valid results, and capability.

The CyberAB threw out evidence related to verbal testimony. Now, an Assessment report or (worse) an identified nonconformity, will not be traceable to the evidence. People can say whatever they want, and it won’t be verifiable later.

Under best practices, if an assessor wants to accept verbal testimony, it must be processed as objective evidence, and be traceable — that means writing down the name of the people who issued it. Just like a courtroom. If the Assessor indicates a control is “Met” based on verbal evidence, there won’t be any proof. Worse, if they write a finding saying the control is “Not Met,” the client may not have any way to fix it, since they may not know who told the Assessor this. Sometimes, an interviewee can simply give a wrong answer, and asking someone else can result in the correct answer.

I suspect this decision comes from paranoia and (as I said) rookie mistakes. The CyberAB is terrified of being sued, so they don’t want a single employee at a client coming at the C3PAO or, worse, the AB itself for defamation. Or heaven forbid the Assessment discovers someone mucking about on the job, and that person gets fired.

Next, I think the Jeff Dalton gang learned all they know about assessments from CMMI, which is not an audit or assessment approach, but an “appraisal,” and which does not generate a traditional certification. Dalton has never understood the difference between ISO-style certifications and CMMI-style appraisals. So the AB has leaned into CMMI rules which allow for non-attribution of interviewees, but that doesn’t work under CMMC.

It’s Not a Maturity Model

The CAP reinforces the fact that CMMC is not a maturity model, despite the name. A maturity model would assess evidence and issue a decision on a graded slope, based (literally) on the “maturity” of the controls examined during the assessment. A poor-performing company might get a “low” rating, while a high-performing one would be deemed “highly mature.”

Instead, the CMMC approach does a lot of dancing around with scores and grades, but in the end, issues a single certificate. It’s still either pass or fail. That’s not a maturity model. That’s a binary certification scheme, like ISO certifications.

Under CMMC 2.0, sure, there are three possible levels. But the client company decides beforehand which level to pursue, based on customer requirements. But that is still not a maturity model.  A true maturity model assessment determines the maturity during the assessment… they are assessing maturity in real time! The client doesn’t pick beforehand, and the result is not a binary pass/fail.

And, as Maj. Weinstein pointed out above, they added points for things that should be mandatory requirements.

Monstrous Bureaucracy

This article is already long, so I’ll cut it short here. The last point I want to make is that the CAP ignores the reality of the typical defense industrial base (DIB) company, and layers in such massive bureaucracy, it will cripple the ability of small to medium-sized enterprises to ever get certified. The planning stages are huge, the assessments long-winded affairs, and the costs are going to be unimaginable.

The authors’ credit page hints as to why this is. The team of authors includes only cybersecurity consultants, without a single end-user being included in the drafting. Read that again: 100% of the authors sell consulting services. Convenient.

At least one of the authors put her hand on the US flag and promised to do CMMC assessments entirely for free, “any damned day of the year,” so maybe there are some costs to be saved after all.