Imagine assessing two organizational capabilities.
The first receives a maturity rating of Level 4. Interviews suggest that practices are well established, and the available documentation supports that impression. However, we have examined little evidence of actual behaviour or outcomes.
The second receives a maturity rating of Level 2. We have observed the work, examined several projects, investigated outcomes and found consistent evidence of significant weaknesses.
Which assessment is stronger?
The first capability has the higher maturity rating.
But the second assessment has considerably stronger evidence supporting its conclusion.
We are describing two different things.
The condition we believe a capability is in, and how confident we are in that belief.
Two Different Dimensions
A capability assessment attempts to establish something about an organization’s ability to achieve particular outcomes.
We examine evidence, interpret observations and arrive at a judgment.
Perhaps we express that judgment using a maturity scale.
Level 2. Level 3. Level 4.
But the rating alone doesn’t tell us how well the conclusion is supported.
A Level 4 assessment based primarily on interviews is not equivalent to a Level 4 assessment supported by extensive observations of actual behaviour and outcomes.
The assessed condition may be the same.
Our confidence in that condition may be very different.
This suggests that capability assessments should distinguish between two dimensions:
- Assessed condition: What does the available evidence suggest about the capability?
- Assessment confidence: How strongly does that evidence justify our conclusion?
Neither dimension can replace the other.
When Evidence Is Missing
Suppose we assess a team that appears to perform exceptionally well.
Stakeholders are satisfied. Deliveries are predictable. Defects are relatively uncommon.
But we have limited access to the team’s actual work.
We cannot establish how its practices operate, whether its results are sustainable or how it responds when conditions change.
What should we conclude?
We might suspect that the team has a mature capability.
But we cannot justify that conclusion with much confidence.
Does that mean the capability is weak?
Not necessarily.
Insufficient evidence of capability is not the same as evidence of insufficient capability.
This distinction matters because assessments can unintentionally penalize subjects that are difficult to observe.
The absence of evidence may reflect limitations in our investigation rather than limitations in the capability itself.
Of course, the ability to demonstrate capability may itself be important.
For some capabilities, maintaining records, providing traceability or demonstrating control is part of what good performance requires.
But then we should assess those expectations explicitly.
We shouldn’t confuse our inability to establish a condition with evidence that the condition is poor.
Confidence Changes the Next Decision
Consider two assessment results.
Capability A is assessed at Level 4, with low confidence.
Capability B is assessed at Level 2, with high confidence.
What should management do?
For Capability A, the immediate need may be further investigation.
We have reason to believe the capability is effective, but insufficient evidence to rely confidently on that conclusion.
For Capability B, additional assessment may contribute relatively little.
We already have strong evidence of weaknesses. The more useful next step may be to investigate their causes and consider improvement interventions.
The maturity ratings suggest different capability conditions.
The confidence levels suggest different degrees of uncertainty about those conditions.
Together, they inform what to do next.
Without this distinction, organizations may invest in improving capabilities that haven’t actually been demonstrated to be weak.
Or they may trust favourable assessments that were never adequately substantiated.
Confidence Is Not a Property of the Capability
There is another subtle distinction.
Confidence doesn’t belong to the capability itself.
It belongs to our assessment of it.
The capability exists and operates independently of whether we understand it correctly.
Our assessment is an interpretation based on selected evidence, observed under particular conditions, at a particular time.
That interpretation may be well supported.
Or it may be uncertain.
It may also be wrong.
This means that assessment confidence should reflect more than the quantity of evidence collected.
We need to consider its relevance, independence, reliability and representativeness.
Ten documents repeating the same claim may provide less justification than one direct observation of actual behaviour.
Consistent evidence from several ordinary projects may tell us little about how the capability performs under exceptional conditions.
And contradictory evidence may reduce our confidence even when most observations support the same conclusion.
Confidence is therefore not simply a measure of how much evidence we have.
It concerns how well that evidence supports the particular judgment we are making.
The Temptation of a Single Number
Organizations often prefer simple assessment results.
A maturity rating is easy to communicate.
It can be displayed in a dashboard, compared across organizational units and tracked over time.
Adding confidence complicates the picture.
But perhaps that complication reflects reality.
Consider an organization whose assessed maturity increases from Level 2 to Level 4.
Has the capability improved?
Possibly.
But perhaps the first assessment was based on extensive observation, while the second relied primarily on management interviews.
Or perhaps the capability hasn’t changed at all, but the second assessment used better evidence and corrected an earlier underestimate.
A change in the assessed rating does not necessarily demonstrate a change in the underlying capability.
To interpret the difference, we also need to understand how the assessments were conducted and how strongly their conclusions were supported.
Otherwise, apparent improvement may partly reflect changes in what we know rather than changes in what the organization can do.
Preserving Uncertainty
An assessment is intended to support decisions.
That creates pressure to produce clear conclusions.
Management wants to know whether a capability is good enough, where improvement is needed and what should happen next.
Uncertainty can seem inconvenient.
But concealing uncertainty doesn’t make a conclusion more reliable.
It merely makes its limitations less visible.
A defensible assessment should therefore communicate not only what we believe, but also how strongly we can justify that belief.
Sometimes the appropriate conclusion is that a capability appears mature, but the evidence is insufficient to establish this confidently.
Sometimes it is that a capability is demonstrably weak.
And sometimes the available evidence simply doesn’t justify a meaningful rating.
These are different situations.
They should not be made to look identical.
Assessing Our Understanding
Perhaps this distinction extends beyond capability assessment.
Whenever we make an evidence-based judgment, we are making two claims.
One concerns the subject.
The other concerns the quality of our understanding of that subject.
We might believe a product is reliable, an intervention is effective or an organizational capability is mature.
But how confidently can we make that claim?
And what evidence would cause us to reconsider it?
An assessment that acknowledges uncertainty isn’t necessarily weaker than one that produces a definitive answer.
It may simply be more accurate about what has actually been established.
The condition of a subject and our confidence in that condition are two different things.
A good assessment should help us understand both.
Leave a Reply