Last updated: June 2025
[IMAGE: A macro photograph of a fractured crystalline lattice structure with cyan light refracting through the cracks, set against a deep black background — representing the structural weaknesses in cryptographic framework design]
Key Takeaways
- Researchers assessed 3 industry-driven crypto agility maturity models referenced by NIST and found recurring structural weaknesses across all of them
- The most critical flaw: none of the models provide measurable, reproducible assessment criteria — meaning two auditors using the same model can reach different conclusions about your cryptographic posture
- If your PQC migration roadmap is built on one of these models, you may be planning against an unvalidated baseline — a risk that compounds as NIST’s post-quantum deadlines approach
The Framework You’re Trusting May Not Be Trustworthy
Picture this: your security team completes a crypto agility assessment using an industry-recognized maturity model. The report lands on your desk showing a score of “Level 3 — Managed.” Six months later, an external auditor runs the same model against the same environment and scores you at Level 2. The gap isn’t your controls — it’s the model itself.
This isn’t a hypothetical. Research published on arXiv (arXiv:2611.17003) assessed three industry-driven crypto agility maturity models that NIST references in its post-quantum cryptography guidance. The finding is direct: all three models carry recurring weaknesses that undermine their reliability as planning tools for cryptographic transition.
For organizations now building PQC migration roadmaps — under real regulatory pressure and real adversarial timelines — this matters immediately.
What Crypto Agility Actually Means (And Why It’s Hard to Measure)
Crypto agility is an organization’s demonstrated ability to rapidly detect, replace, and update cryptographic algorithms, keys, certificates, and protocols without disrupting dependent systems. It is the operational prerequisite for surviving a post-quantum transition — or any future cryptographic deprecation event.
A crypto agility maturity model attempts to score that capability across dimensions like asset inventory, algorithm lifecycle management, key management automation, and organizational readiness. The higher your maturity level, the faster and more reliably you can swap out a compromised or deprecated algorithm across your entire stack.
The problem is that “maturity” is only useful if the measurement is consistent. A ruler that stretches depending on who holds it isn’t a ruler.
Three Models, Three Sets of Gaps: What the Research Found
The arXiv study evaluated three industry-driven crypto agility maturity models against established design principles for maturity frameworks. The researchers identified both individual strengths — which can help practitioners choose between models for specific use cases — and systemic weaknesses that cut across all three.
The Reproducibility Problem
The most operationally dangerous weakness identified: lack of measurable and reproducible assessment criteria.
When a maturity model’s criteria are qualitative and loosely defined, two assessors evaluating the same organization will produce different scores. That inconsistency destroys the model’s core value — giving you a reliable baseline from which to measure progress and justify investment.
For a CISO presenting a PQC migration business case to a board, an unreproducible score is worse than no score. It creates audit exposure and undermines the credibility of the entire program.
The Prescriptive Guidance Gap
The second recurring weakness: limited prescriptive guidance for improving crypto agility.
Most maturity models tell you where you are. The better ones tell you how to advance. The models assessed largely fail on the second count. An organization that scores at Level 2 receives little actionable direction on what specific controls, processes, or architectural changes would move them to Level 3.
This leaves security teams in the position of knowing they have a gap but lacking a validated path to close it — precisely when the clock on quantum-vulnerable cryptography is running.
“Our analysis reveals recurring weaknesses, particularly the lack of measurable, reproducible assessment criteria and limited prescriptive guidance for improving crypto agility.” — arXiv:2611.17003, Maturity²: Assessing the Maturity of Crypto Agility Maturity Models
Comparing Framework Quality: What a Robust Model Requires
| Criterion | What a Strong Model Provides | Weakness Found in Assessed Models |
|---|---|---|
| Assessment reproducibility | Quantitative scoring rubrics with defined evidence requirements | Qualitative criteria that vary by assessor interpretation |
| Prescriptive improvement paths | Specific controls and architectural changes per level transition | General descriptions of target states without implementation guidance |
| Measurable outcomes | Defined metrics (e.g., % of certificates with automated rotation) | Capability descriptions without attached KPIs |
| Algorithm lifecycle coverage | Explicit mapping to NIST PQC algorithm families (ML-KEM, ML-DSA, SLH-DSA) | Limited or absent mapping to current standardization outputs |
| Organizational scope | Covers people, process, and technology dimensions with equal rigor | Uneven coverage weighted toward technical controls |
Table 1: Crypto agility maturity model quality criteria vs. weaknesses identified in arXiv:2611.17003
Why NIST’s Endorsement Doesn’t Equal Validation
NIST references these maturity models in its PQC guidance — but referencing is not endorsing, and endorsing is not validating. The research makes this distinction operationally significant.
Organizations that interpret “NIST-referenced” as “NIST-validated” are building migration plans on an assumption that the research does not support. NIST’s own assessment or endorsement level of these models beyond citation is not documented in the available literature.
This creates a specific compliance risk: if your PQC readiness documentation cites a maturity model score as evidence of due diligence, and that model’s methodology is later challenged, your compliance posture is exposed — not because your controls are weak, but because your measurement framework is.
The gap between “referenced by NIST” and “validated by NIST” is where audit findings live. Organizations treating these as equivalent are carrying undocumented framework risk in their PQC compliance posture.
The Regulatory Timeline Makes This Urgent Now
NIST finalized its first three post-quantum cryptographic standards in August 2024: ML-KEM (FIPS 203), ML-DSA (FIPS 204), and SLH-DSA (FIPS 205). Federal agencies face migration mandates, and regulated industries — financial services, healthcare, critical infrastructure — are watching those mandates as leading indicators of their own compliance requirements.
The near-term implication (within 1-2 years): organizations using flawed maturity models to assess their PQC readiness are generating compliance documentation that may not survive regulatory scrutiny. The medium-term implication (3-5 years): as PQC migration governance matures, the frameworks with identified weaknesses will either be revised or replaced — and organizations that built roadmaps around them will need to re-baseline.
The cost of re-baselining mid-migration is not just financial. It’s the organizational credibility of the security function that championed the original framework.
How BeQuantum Approaches Crypto Agility Assessment
The weaknesses identified in the research — non-reproducible criteria, absent prescriptive guidance — are exactly the failure modes that BeQuantum’s PQC Layer was designed to address.
Rather than relying on self-reported maturity scores, BeQuantum’s approach centers on continuous, automated cryptographic asset discovery and algorithm classification. Every certificate, key exchange protocol, and signing operation in scope is catalogued against a live algorithm risk registry that maps directly to NIST’s standardized PQC algorithm families.
This produces a measurable, reproducible baseline — the same scan run by two different operators against the same environment produces the same output, because the assessment criteria are encoded in the platform, not in an assessor’s interpretation of a qualitative rubric.
Where the assessed maturity models lack prescriptive improvement paths, BeQuantum’s Digital Notary provides prioritized remediation queues: assets are ranked by algorithm vulnerability severity, certificate expiry proximity, and business criticality, so security teams know exactly which migration actions to execute first and in what sequence.
For organizations evaluating IceCase hardware deployments, the same cryptographic inventory feeds directly into hardware security module (HSM) configuration planning — ensuring that PQC algorithm support is validated at the hardware layer before migration begins, not discovered as a gap during rollout.
What You Should Do in the Next 90 Days
Step 1 — Audit your current framework’s methodology (Days 1-30) Identify which crypto agility maturity model your organization currently uses or plans to use. Request the scoring rubric documentation and test it: have two independent assessors score the same control domain and compare results. If scores diverge by more than one level, your model has a reproducibility problem that will undermine your migration planning.
Step 2 — Map your model’s guidance to NIST PQC algorithm families (Days 30-60) Verify that your maturity model explicitly addresses ML-KEM, ML-DSA, and SLH-DSA — the three finalized NIST PQC standards. If your model predates August 2024 and hasn’t been updated, its algorithm coverage is incomplete by definition. Document this gap before your next compliance audit surfaces it.
Step 3 — Establish a reproducible cryptographic asset baseline (Days 60-90) Before advancing your maturity level, establish a quantitative baseline: total count of RSA and ECC certificates in your environment, percentage with automated renewal, percentage of TLS connections using forward-secret cipher suites, and number of systems with hardcoded cryptographic dependencies. These numbers give you a migration starting point that no qualitative rubric can provide — and they’re defensible in front of a regulator.
Frequently Asked Questions
Q: If NIST references these maturity models, why can’t we rely on them for compliance documentation?
A: NIST referencing a model means it was considered relevant enough to cite — not that NIST has validated its methodology against formal design principles. The arXiv research (arXiv:2611.17003) applied that validation and found structural weaknesses. For compliance documentation, the distinction matters: a regulator examining your PQC readiness evidence will scrutinize the methodology behind your maturity score, not just the score itself.
Q: What makes a crypto agility maturity model’s criteria “reproducible”?
A: Reproducibility requires that assessment criteria be defined with sufficient specificity that two independent assessors examining the same evidence reach the same conclusion. In practice, this means quantitative thresholds (e.g., “90% of certificates have automated rotation enabled” rather than “certificate management is largely automated”), defined evidence requirements per criterion, and explicit scoring logic. Without these, maturity scores reflect assessor judgment as much as organizational capability.
Q: How do we know when a maturity model has been sufficiently strengthened to rely on?
A: The research derived concrete recommendations for strengthening future crypto agility maturity models. When evaluating a revised or new model, check for three things: quantitative scoring rubrics with defined evidence requirements, explicit level-transition guidance specifying what controls and processes must be implemented to advance, and direct mapping to current NIST PQC algorithm standards. A model that satisfies all three is structurally sound enough to anchor a migration program.
Sources: Maturity²: Assessing the Maturity of Crypto Agility Maturity Models, arXiv:2611.17003