Two teams use exactly the same testing practices.
One receives a positive capability assessment. The other does not.
Is the assessment inconsistent?
Not necessarily.
One team develops a relatively simple internal application. The other develops a safety-critical subsystem.
The practices may be identical, but the expectations should not be.
This raises an interesting question.
How can a generic capability reference model remain useful when adequate capability depends on context?
From Generic to Specific
A capability reference model describes what an organization should be capable of doing, without necessarily prescribing how that capability must be realized.
Consider Quality Assurance.
A generic reference model might describe capabilities related to quality strategy, verification, validation, test design, defect management and quality evaluation.
These concepts can apply across many organizations.
But their realization depends on context.
An organization developing medical devices faces different risks, obligations and quality expectations from one developing internal business applications.
Even within the same organization, different products and subsystems may require different approaches.
A safety-critical component might demand extensive traceability and formal verification. A user interface might place greater emphasis on usability and exploratory testing. A data integration component might require particular attention to data integrity and failure handling.
The underlying capability concepts remain recognizable.
What constitutes adequate capability changes.
Contextualizing the Reference Model
This suggests a progression:
Generic capability → Organizational context → Product or subsystem context
At the generic level, the reference model provides a common vocabulary and a basis for reasoning about capability.
At the organizational level, we consider the organization’s objectives, responsibilities, operating environment, risks and constraints.
At the product or subsystem level, we consider the specific characteristics and quality expectations of the subject being assessed.
Each level adds context.
But that doesn’t necessarily mean creating an entirely new capability model at every level.
The reference model can remain conceptually stable while the expectations, evidence and appropriate realization become increasingly specific.
For example, risk-based testing may be a relevant capability in both a business application and a safety-critical subsystem.
But what constitutes sufficient risk analysis, test coverage and supporting evidence may differ substantially.
The capability is recognizable in both contexts.
Its adequate realization is not necessarily identical.
Assessing Capability or Checking Compliance?
This distinction matters because reference models can easily become checklists.
Does the organization have a test strategy?
Does it use automated testing?
Does it perform reviews?
Does it maintain traceability?
These questions may be useful, but the existence of a practice doesn’t necessarily establish that the capability is adequate.
An organization can perform all the expected activities while remaining unable to achieve the intended outcomes.
Conversely, an organization may realize a capability effectively through practices that differ from those anticipated when the reference model was created.
If we assess compliance with prescribed practices, we risk confusing the means with the capability.
Capability assessment should evaluate fitness for purpose, not conformity to a universal implementation.
This also means that identical practices need not produce identical assessment results.
And different practices may legitimately support the same capability rating.
The question is whether the subject can achieve what is required of it in its particular context.
The Risk of Contextualizing Too Much
There is a complication.
If every organization, product or subsystem receives its own completely customized model, we may lose the consistency that made the reference model useful in the first place.
Assessments become difficult to compare. Common capability concepts disappear behind local terminology. Improvement approaches become unnecessarily fragmented.
But contextualizing too little creates the opposite problem.
We evaluate subjects against generic expectations that may be irrelevant, insufficient or inappropriate.
The challenge is therefore to distinguish between what should remain stable and what should vary.
The reference model provides the conceptual structure.
Context determines which capabilities matter, what adequate performance means and what evidence is needed to assess them.
Sometimes the context may even reveal capabilities or dimensions missing from the generic model.
That should be an opportunity to improve the reference model, rather than a reason to abandon it.
A Reference Model Is Not a Prescription
Perhaps the value of a capability reference model lies precisely in this distinction.
It allows us to reason consistently about different subjects without assuming that they should all operate identically.
It provides a shared language for understanding, assessing and developing capabilities.
But it does not eliminate the need for professional judgment.
We still need to understand the subject, its purpose, its environment and the outcomes it must achieve.
Only then can we determine what adequate capability looks like.
A reference model tells us what to reason about. Context tells us what good looks like.
Leave a Reply