An organization wants its software development capability assessed.
Management selects three projects for examination.
All three are successful. They have experienced teams, established practices and satisfied stakeholders.
The assessors conduct interviews, examine documentation, observe activities and investigate outcomes.
The evidence is consistent.
The organization appears to have a mature software development capability.
But what about the projects that weren’t selected?
The struggling project with constantly changing requirements?
The team that depends on one exceptionally experienced developer?
The product that repeatedly fails to meet its release commitments?
Perhaps those projects would tell a different story.
And perhaps the assessment is perfectly accurate about the projects it examined.
The problem may not be the evidence. It may be what we believe the evidence represents.
What Have We Actually Assessed?
An assessment never examines an entire organization.
It examines selected aspects of the organization, through selected evidence, under particular conditions.
We interview certain people.
We examine particular projects.
We observe specific activities.
We investigate outcomes from a limited period.
From these observations, we develop conclusions about the subject being assessed.
But there is a difference between what we have observed and what we can reasonably conclude.
Suppose we observe a team consistently producing high-quality software.
We have evidence that this team can achieve good results under the conditions examined.
Does that demonstrate that the organization has a mature software development capability?
Not necessarily.
It may demonstrate that the organization has at least one team capable of producing good results.
That is useful information.
But it is a narrower conclusion.
Capability Under Favourable Conditions
Consider a team that delivers successfully because it has unusually experienced people.
Its requirements are clear. Its stakeholders are available. Its technical environment is stable.
The team consistently produces good results.
Now imagine changing the conditions.
A key developer leaves.
Several stakeholders disagree about priorities.
Requirements become uncertain.
The team must maintain an unfamiliar product while continuing development.
Would it still perform effectively?
Perhaps.
But the original evidence doesn’t necessarily establish that.
This matters because organizational capability concerns more than the ability to succeed under ideal circumstances.
It concerns the ability to achieve intended outcomes with sufficient reliability under the conditions the organization can reasonably expect to encounter.
A successful project demonstrates that success is possible.
It doesn’t automatically demonstrate that success is reproducible.
The Selection Problem
Sampling is unavoidable.
Assessors cannot investigate every team, project, product, decision and outcome.
They must select where to look.
But that selection influences what they are capable of discovering.
If we examine only successful projects, we may underestimate recurring weaknesses.
If we interview only management, we may miss the differences between intended and actual practices.
If we investigate only recent releases, we may overlook problems that emerge during long-term maintenance.
If we observe only normal operations, we may miss how the organization behaves under pressure.
The issue isn’t simply the number of observations.
A large sample can still be misleading if it systematically excludes relevant conditions.
Assessment quality depends partly on whether our inquiry exposes itself to situations in which its emerging conclusions might fail.
That doesn’t mean deliberately searching only for problems.
It means selecting evidence that allows us to understand the relevant variation in the subject being assessed.
The Organization Is Not Uniform
There is another complication.
Organizations are rarely homogeneous.
Different teams may operate under different constraints.
Products may have different quality expectations.
Some activities may be highly standardized, while others depend on local expertise.
A capability may be mature in one organizational unit and weak in another.
What does an organization-wide maturity rating mean under those circumstances?
Perhaps it describes the dominant pattern.
Perhaps it describes the weakest relevant part.
Perhaps it represents an aggregation of several distinct assessments.
Each approach may serve a purpose.
But they are not interchangeable.
A single rating can conceal important variation unless we understand what it represents.
And sometimes the variation is itself the most interesting finding.
Why can one team achieve results that others cannot?
What enables its success?
Can those conditions be reproduced?
Or is the apparent organizational capability actually concentrated in a few exceptional individuals?
Those questions may be more valuable than an average maturity score.
Accurate Evidence, Unjustified Conclusions
This creates an interesting possibility.
An assessment can be rigorous, evidence-based and accurate — yet still support a misleading organizational conclusion.
The interviews may have been conducted properly.
The observations may be reliable.
The documentation may be authentic.
The findings may accurately describe the selected projects.
But if we generalize those findings beyond the conditions they represent, we claim more than the evidence justifies.
The weakness isn’t necessarily in how we interpreted the observations.
It may be in how we defined the population to which our conclusions apply.
This is why assessment confidence cannot be separated from representativeness.
We might be highly confident that a particular team performs effectively.
And much less confident that the organization can reproduce that performance elsewhere.
Both judgments can be justified by the same evidence.
They simply concern different claims.
Making the Boundaries Visible
Perhaps every assessment should make its boundaries explicit.
Which parts of the organization were examined?
Which operating conditions were represented?
Which relevant variations were investigated?
What was deliberately excluded?
And how far can the conclusions reasonably be generalized?
These questions don’t make an assessment less useful.
They help establish what its findings actually mean.
Sometimes a limited assessment is entirely appropriate.
We may only need to understand one team, one product or one particular capability under specified conditions.
The problem arises when a narrow investigation is presented as evidence of something broader.
A focused assessment can support a focused conclusion.
It should not silently become an organization-wide judgment.
What We Didn’t See
There is a temptation to believe that assessment quality is primarily a matter of collecting and interpreting evidence correctly.
Those things are essential.
But they aren’t sufficient.
We must also consider whether the investigation gave us a reasonable opportunity to discover that our initial understanding was wrong.
Did we examine meaningful variation?
Did we investigate conditions under which the capability might struggle?
Did we look beyond the examples the organization was most comfortable showing us?
And did we preserve the distinction between what we observed and what we inferred?
An assessment doesn’t reveal the organization in its entirety.
It reveals aspects of the organization under the conditions we chose to examine.
The challenge is to understand what those observations justify — without assuming they represent everything we didn’t see.
An assessment is defined not only by the evidence it examines, but also by the reality it leaves unexamined.
Leave a Reply