Blog

An organization commissions an assessment of its software development capability.

The assessors conduct interviews, examine documentation, observe practices and evaluate the available evidence.

A few weeks later, management receives a report.

The organization’s software development capability has been assessed at maturity Level 3.

The report contains findings, supporting evidence and recommendations for improvement.

Management now has a basis for making decisions.

Or does it?

We know what the assessors concluded.

But how do we know whether their conclusions deserve our confidence?

Who assesses the assessment?

Evidence-Based Is a Claim

Assessment reports often describe their conclusions as evidence-based.

That sounds reassuring.

But what does it actually establish?

Perhaps the assessors examined fifty documents, interviewed twenty people and analyzed several years of performance data.

That’s a considerable amount of information.

Yet the quantity of evidence tells us relatively little about whether the assessment was sound.

Were the right questions asked?

Was the evidence relevant to those questions?

Were the observations representative of the subject being assessed?

Were contradictory findings investigated?

Did the assessors distinguish observations from interpretations?

And did their conclusions go beyond what the evidence could reasonably support?

An assessment can contain extensive evidence while drawing poorly justified conclusions.

“Evidence-based” is itself a claim that requires justification.

Assessing the Mechanism

An assessment is a mechanism for developing justified understanding of a subject.

It involves selecting a frame of reference, gathering evidence, interpreting observations and synthesizing conclusions.

Each step introduces choices.

The reference model influences what we consider relevant.

The sampling strategy determines which parts of reality we examine.

The evidence sources influence what we can observe.

The interpretation determines what meaning we assign to those observations.

And the synthesis determines which conclusions we ultimately communicate.

These choices affect the result.

So perhaps evaluating an assessment requires examining not only its conclusions, but also the mechanism through which those conclusions were produced.

Not to demand that every assessment follow an identical procedure.

But to establish whether its approach was appropriate for the subject, purpose and consequences of the assessment.

A Plausible Conclusion Can Still Be Wrong

Imagine an assessment concluding that an organization has weak requirements management.

The conclusion sounds plausible.

Projects experience rework. Stakeholders complain about unclear requirements. Development teams frequently request clarification.

But suppose the assessors never investigated how requirements are actually developed and managed.

They inferred a capability weakness from symptoms observed elsewhere.

Perhaps requirements management really is weak.

But perhaps the underlying problem lies in decision-making, product ownership or conflicting organizational priorities.

The assessment may have identified a genuine problem while attributing it to the wrong cause.

That distinction matters.

If management acts on the assessment, it may invest in improving requirements practices without addressing the mechanism responsible for the observed difficulties.

The recommendations may be sensible.

The evidence may be authentic.

And the assessment may still have failed to justify its central interpretation.

Can We Follow the Reasoning?

One way to examine an assessment is to ask whether its reasoning can be reconstructed.

What was observed?

Which evidence supports that observation?

How was the observation interpreted?

Which assumptions influenced the interpretation?

What alternative explanations were considered?

And how did these elements contribute to the final conclusion?

If we cannot follow that reasoning, we have limited ability to evaluate the assessment.

We may have to trust the assessor’s authority rather than the justification for the conclusion.

That doesn’t mean every judgment can be reduced to a mechanical calculation.

Professional assessment involves interpretation.

But interpretation should be open to examination.

A defensible assessment should make it possible to understand not only what the assessor concluded, but why.

The Problem of Independent Review

Perhaps the obvious answer is to have another assessor review the assessment.

That could help.

A second assessor might identify overlooked evidence, unsupported conclusions or alternative interpretations.

But independence alone doesn’t guarantee a better result.

Two assessors can share the same assumptions.

They can rely on the same reference model.

They can interpret the same evidence through the same professional conventions.

Agreement between them may increase confidence, but it doesn’t automatically establish correctness.

And this introduces a recursive problem.

If an assessment needs to be assessed, does the assessment of that assessment also need to be assessed?

And so on?

We could continue indefinitely.

But perhaps the objective isn’t to establish absolute certainty.

It is to make consequential judgments sufficiently transparent, justified and open to challenge.

An assessment should be capable of explaining and defending its conclusions without requiring an endless chain of additional assessments.

The Assessment as a Subject

There is another way to think about this.

We assess organizational capabilities because we want to understand whether they can reliably achieve their intended outcomes.

But assessment is itself an activity intended to achieve an outcome.

It should produce sufficiently justified understanding to support decisions.

That means assessment has characteristics we can examine.

Does it use an appropriate frame of reference?

Does it seek relevant and representative evidence?

Does it distinguish claims from observations?

Does it investigate significant contradictions?

Does it preserve uncertainty?

Does it avoid conclusions stronger than the available evidence?

These are not merely procedural requirements.

They concern the quality of the assessment’s ability to achieve its purpose.

In that sense, the assessment itself becomes a subject of assessment.

When AI Performs the Assessment

This becomes particularly interesting when AI enters the picture.

Imagine an AI system analyzing fifty thousand organizational documents.

It identifies recurring patterns, connects observations and produces an assessment of several organizational capabilities.

The result is detailed, coherent and supported by references to source material.

It may even be more comprehensive than a conventional assessment.

But what does that tell us about its reliability?

Did the AI distinguish independent evidence from repeated claims?

Did it identify contradictions?

Did it recognize that certain parts of the organization were poorly represented?

Did it confuse descriptions of intended practices with observations of actual behaviour?

Did it consider alternative explanations?

And can we reconstruct how it arrived at its conclusions?

The fact that AI examined fifty thousand documents doesn’t establish that its conclusions are justified.

Nor does the presence of source references.

The same questions apply whether an assessment is performed by a human, an AI system or a combination of both.

The mechanism through which evidence becomes judgment still matters.

Trust Through Challengeability

Perhaps the quality of an assessment cannot be established merely by checking whether it followed a prescribed method.

A method can be followed correctly while producing misleading conclusions.

And a sound assessment may sometimes require departing from the original plan when unexpected evidence emerges.

What matters is whether the assessment can justify its approach and conclusions in relation to its purpose.

Can we examine the evidence?

Can we follow the reasoning?

Can we identify the assumptions?

Can we understand the uncertainty?

Can we challenge the conclusions?

And can the assessment change when those challenges reveal weaknesses?

This doesn’t eliminate the need for professional judgment.

It makes that judgment accountable.

An assessment should not demand trust simply because it was performed by an expert, followed a methodology or processed a large amount of evidence.

It should earn confidence through the quality of its justification.

An assessment is not trustworthy merely because it is evidence-based.

It becomes trustworthy to the extent that its use of evidence can itself withstand examination.

An organization wants its software development capability assessed.

Management selects three projects for examination.

All three are successful. They have experienced teams, established practices and satisfied stakeholders.

The assessors conduct interviews, examine documentation, observe activities and investigate outcomes.

The evidence is consistent.

The organization appears to have a mature software development capability.

But what about the projects that weren’t selected?

The struggling project with constantly changing requirements?

The team that depends on one exceptionally experienced developer?

The product that repeatedly fails to meet its release commitments?

Perhaps those projects would tell a different story.

And perhaps the assessment is perfectly accurate about the projects it examined.

The problem may not be the evidence. It may be what we believe the evidence represents.

What Have We Actually Assessed?

An assessment never examines an entire organization.

It examines selected aspects of the organization, through selected evidence, under particular conditions.

We interview certain people.

We examine particular projects.

We observe specific activities.

We investigate outcomes from a limited period.

From these observations, we develop conclusions about the subject being assessed.

But there is a difference between what we have observed and what we can reasonably conclude.

Suppose we observe a team consistently producing high-quality software.

We have evidence that this team can achieve good results under the conditions examined.

Does that demonstrate that the organization has a mature software development capability?

Not necessarily.

It may demonstrate that the organization has at least one team capable of producing good results.

That is useful information.

But it is a narrower conclusion.

Capability Under Favourable Conditions

Consider a team that delivers successfully because it has unusually experienced people.

Its requirements are clear. Its stakeholders are available. Its technical environment is stable.

The team consistently produces good results.

Now imagine changing the conditions.

A key developer leaves.

Several stakeholders disagree about priorities.

Requirements become uncertain.

The team must maintain an unfamiliar product while continuing development.

Would it still perform effectively?

Perhaps.

But the original evidence doesn’t necessarily establish that.

This matters because organizational capability concerns more than the ability to succeed under ideal circumstances.

It concerns the ability to achieve intended outcomes with sufficient reliability under the conditions the organization can reasonably expect to encounter.

A successful project demonstrates that success is possible.

It doesn’t automatically demonstrate that success is reproducible.

The Selection Problem

Sampling is unavoidable.

Assessors cannot investigate every team, project, product, decision and outcome.

They must select where to look.

But that selection influences what they are capable of discovering.

If we examine only successful projects, we may underestimate recurring weaknesses.

If we interview only management, we may miss the differences between intended and actual practices.

If we investigate only recent releases, we may overlook problems that emerge during long-term maintenance.

If we observe only normal operations, we may miss how the organization behaves under pressure.

The issue isn’t simply the number of observations.

A large sample can still be misleading if it systematically excludes relevant conditions.

Assessment quality depends partly on whether our inquiry exposes itself to situations in which its emerging conclusions might fail.

That doesn’t mean deliberately searching only for problems.

It means selecting evidence that allows us to understand the relevant variation in the subject being assessed.

The Organization Is Not Uniform

There is another complication.

Organizations are rarely homogeneous.

Different teams may operate under different constraints.

Products may have different quality expectations.

Some activities may be highly standardized, while others depend on local expertise.

A capability may be mature in one organizational unit and weak in another.

What does an organization-wide maturity rating mean under those circumstances?

Perhaps it describes the dominant pattern.

Perhaps it describes the weakest relevant part.

Perhaps it represents an aggregation of several distinct assessments.

Each approach may serve a purpose.

But they are not interchangeable.

A single rating can conceal important variation unless we understand what it represents.

And sometimes the variation is itself the most interesting finding.

Why can one team achieve results that others cannot?

What enables its success?

Can those conditions be reproduced?

Or is the apparent organizational capability actually concentrated in a few exceptional individuals?

Those questions may be more valuable than an average maturity score.

Accurate Evidence, Unjustified Conclusions

This creates an interesting possibility.

An assessment can be rigorous, evidence-based and accurate — yet still support a misleading organizational conclusion.

The interviews may have been conducted properly.

The observations may be reliable.

The documentation may be authentic.

The findings may accurately describe the selected projects.

But if we generalize those findings beyond the conditions they represent, we claim more than the evidence justifies.

The weakness isn’t necessarily in how we interpreted the observations.

It may be in how we defined the population to which our conclusions apply.

This is why assessment confidence cannot be separated from representativeness.

We might be highly confident that a particular team performs effectively.

And much less confident that the organization can reproduce that performance elsewhere.

Both judgments can be justified by the same evidence.

They simply concern different claims.

Making the Boundaries Visible

Perhaps every assessment should make its boundaries explicit.

Which parts of the organization were examined?

Which operating conditions were represented?

Which relevant variations were investigated?

What was deliberately excluded?

And how far can the conclusions reasonably be generalized?

These questions don’t make an assessment less useful.

They help establish what its findings actually mean.

Sometimes a limited assessment is entirely appropriate.

We may only need to understand one team, one product or one particular capability under specified conditions.

The problem arises when a narrow investigation is presented as evidence of something broader.

A focused assessment can support a focused conclusion.

It should not silently become an organization-wide judgment.

What We Didn’t See

There is a temptation to believe that assessment quality is primarily a matter of collecting and interpreting evidence correctly.

Those things are essential.

But they aren’t sufficient.

We must also consider whether the investigation gave us a reasonable opportunity to discover that our initial understanding was wrong.

Did we examine meaningful variation?

Did we investigate conditions under which the capability might struggle?

Did we look beyond the examples the organization was most comfortable showing us?

And did we preserve the distinction between what we observed and what we inferred?

An assessment doesn’t reveal the organization in its entirety.

It reveals aspects of the organization under the conditions we chose to examine.

The challenge is to understand what those observations justify — without assuming they represent everything we didn’t see.

An assessment is defined not only by the evidence it examines, but also by the reality it leaves unexamined.

Imagine assessing two organizational capabilities.

The first receives a maturity rating of Level 4. Interviews suggest that practices are well established, and the available documentation supports that impression. However, we have examined little evidence of actual behaviour or outcomes.

The second receives a maturity rating of Level 2. We have observed the work, examined several projects, investigated outcomes and found consistent evidence of significant weaknesses.

Which assessment is stronger?

The first capability has the higher maturity rating.

But the second assessment has considerably stronger evidence supporting its conclusion.

We are describing two different things.

The condition we believe a capability is in, and how confident we are in that belief.

Two Different Dimensions

A capability assessment attempts to establish something about an organization’s ability to achieve particular outcomes.

We examine evidence, interpret observations and arrive at a judgment.

Perhaps we express that judgment using a maturity scale.

Level 2. Level 3. Level 4.

But the rating alone doesn’t tell us how well the conclusion is supported.

A Level 4 assessment based primarily on interviews is not equivalent to a Level 4 assessment supported by extensive observations of actual behaviour and outcomes.

The assessed condition may be the same.

Our confidence in that condition may be very different.

This suggests that capability assessments should distinguish between two dimensions:

• Assessed condition: What does the available evidence suggest about the capability?
• Assessment confidence: How strongly does that evidence justify our conclusion?

Neither dimension can replace the other.

When Evidence Is Missing

Suppose we assess a team that appears to perform exceptionally well.

Stakeholders are satisfied. Deliveries are predictable. Defects are relatively uncommon.

But we have limited access to the team’s actual work.

We cannot establish how its practices operate, whether its results are sustainable or how it responds when conditions change.

What should we conclude?

We might suspect that the team has a mature capability.

But we cannot justify that conclusion with much confidence.

Does that mean the capability is weak?

Not necessarily.

Insufficient evidence of capability is not the same as evidence of insufficient capability.

This distinction matters because assessments can unintentionally penalize subjects that are difficult to observe.

The absence of evidence may reflect limitations in our investigation rather than limitations in the capability itself.

Of course, the ability to demonstrate capability may itself be important.

For some capabilities, maintaining records, providing traceability or demonstrating control is part of what good performance requires.

But then we should assess those expectations explicitly.

We shouldn’t confuse our inability to establish a condition with evidence that the condition is poor.

Confidence Changes the Next Decision

Consider two assessment results.

Capability A is assessed at Level 4, with low confidence.

Capability B is assessed at Level 2, with high confidence.

What should management do?

For Capability A, the immediate need may be further investigation.

We have reason to believe the capability is effective, but insufficient evidence to rely confidently on that conclusion.

For Capability B, additional assessment may contribute relatively little.

We already have strong evidence of weaknesses. The more useful next step may be to investigate their causes and consider improvement interventions.

The maturity ratings suggest different capability conditions.

The confidence levels suggest different degrees of uncertainty about those conditions.

Together, they inform what to do next.

Without this distinction, organizations may invest in improving capabilities that haven’t actually been demonstrated to be weak.

Or they may trust favourable assessments that were never adequately substantiated.

Confidence Is Not a Property of the Capability

There is another subtle distinction.

Confidence doesn’t belong to the capability itself.

It belongs to our assessment of it.

The capability exists and operates independently of whether we understand it correctly.

Our assessment is an interpretation based on selected evidence, observed under particular conditions, at a particular time.

That interpretation may be well supported.

Or it may be uncertain.

It may also be wrong.

This means that assessment confidence should reflect more than the quantity of evidence collected.

We need to consider its relevance, independence, reliability and representativeness.

Ten documents repeating the same claim may provide less justification than one direct observation of actual behaviour.

Consistent evidence from several ordinary projects may tell us little about how the capability performs under exceptional conditions.

And contradictory evidence may reduce our confidence even when most observations support the same conclusion.

Confidence is therefore not simply a measure of how much evidence we have.

It concerns how well that evidence supports the particular judgment we are making.

The Temptation of a Single Number

Organizations often prefer simple assessment results.

A maturity rating is easy to communicate.

It can be displayed in a dashboard, compared across organizational units and tracked over time.

Adding confidence complicates the picture.

But perhaps that complication reflects reality.

Consider an organization whose assessed maturity increases from Level 2 to Level 4.

Has the capability improved?

Possibly.

But perhaps the first assessment was based on extensive observation, while the second relied primarily on management interviews.

Or perhaps the capability hasn’t changed at all, but the second assessment used better evidence and corrected an earlier underestimate.

A change in the assessed rating does not necessarily demonstrate a change in the underlying capability.

To interpret the difference, we also need to understand how the assessments were conducted and how strongly their conclusions were supported.

Otherwise, apparent improvement may partly reflect changes in what we know rather than changes in what the organization can do.

Preserving Uncertainty

An assessment is intended to support decisions.

That creates pressure to produce clear conclusions.

Management wants to know whether a capability is good enough, where improvement is needed and what should happen next.

Uncertainty can seem inconvenient.

But concealing uncertainty doesn’t make a conclusion more reliable.

It merely makes its limitations less visible.

A defensible assessment should therefore communicate not only what we believe, but also how strongly we can justify that belief.

Sometimes the appropriate conclusion is that a capability appears mature, but the evidence is insufficient to establish this confidently.

Sometimes it is that a capability is demonstrably weak.

And sometimes the available evidence simply doesn’t justify a meaningful rating.

These are different situations.

They should not be made to look identical.

Assessing Our Understanding

Perhaps this distinction extends beyond capability assessment.

Whenever we make an evidence-based judgment, we are making two claims.

One concerns the subject.

The other concerns the quality of our understanding of that subject.

We might believe a product is reliable, an intervention is effective or an organizational capability is mature.

But how confidently can we make that claim?

And what evidence would cause us to reconsider it?

An assessment that acknowledges uncertainty isn’t necessarily weaker than one that produces a definitive answer.

It may simply be more accurate about what has actually been established.

The condition of a subject and our confidence in that condition are two different things.

A good assessment should help us understand both.

Imagine assessing an organization’s testing capability.

Management explains that testing is integrated throughout development. Team members confirm that testing activities are part of every sprint. Process documentation describes how testing should be performed.

Everything appears consistent.

Then we examine the actual work.

Testing frequently happens at the end of the sprint. Defects are discovered late. Releases are delayed because important problems remain unresolved.

Now we have a contradiction.

What should we do with it?

We could conclude that the organization isn’t following its defined process.

But perhaps the more interesting question is:

Why does the organization believe testing is integrated when its actual behaviour suggests otherwise?

The contradiction may tell us more about the organization than the original evidence did.

When Evidence Doesn’t Agree

Assessments often involve collecting information from different sources.

We interview people, examine documentation, observe behaviour and investigate outcomes.

Ideally, these sources help us develop a coherent understanding of the subject being assessed.

But sometimes they disagree.

Management describes one reality. Employees describe another. Documentation describes a third. Actual outcomes suggest something else entirely.

It’s tempting to treat these inconsistencies as problems to resolve.

Perhaps someone misunderstood the question. Perhaps the documentation is outdated. Perhaps one observation was unrepresentative.

Any of these explanations might be correct.

But we shouldn’t assume that disagreement is merely noise.

A contradiction may reveal something that none of the individual observations could tell us.

The Process That Works Without Being Followed

Consider another example.

An organization has established a detailed development process.

An assessment discovers that one team routinely deviates from it.

At first glance, this appears to indicate weak process compliance.

But the team consistently delivers good results.

Now we have another contradiction.

The prescribed process isn’t being followed, yet the intended outcomes are being achieved.

Perhaps the team has developed a more effective way of working.

Perhaps the prescribed process contains unnecessary activities.

Perhaps the team succeeds because an exceptionally capable individual compensates for weaknesses elsewhere.

Or perhaps its apparent success depends on conditions that won’t last.

We don’t know yet.

But simply recording noncompliance would miss the opportunity to understand what is happening.

The contradiction gives us a reason to investigate.

The Danger of Making Evidence Agree

There is a subtle risk in the way we synthesize assessment findings.

We want to produce a coherent conclusion.

Conflicting observations complicate that task.

So we may be tempted to reconcile them into a single explanation.

Management’s description becomes the intended process. Employees’ accounts become variations in implementation. Unfavourable outcomes become exceptions.

The resulting assessment may sound perfectly reasonable.

But have we explained the contradictions?

Or merely organized them into a narrative that makes them less visible?

Coherence is desirable, but it isn’t proof of correctness.

An explanation can be internally consistent while failing to account for important observations.

Sometimes the most valuable assessment finding is precisely that our current explanation doesn’t adequately describe what we observe.

Contradiction as a Starting Point

This suggests a different approach.

When significant evidence contradicts our current understanding, we should become curious.

Why do these observations disagree?

Are they describing different conditions?

Do different parts of the organization operate differently?

Are people reporting intended behaviour rather than actual behaviour?

Has something changed that our existing understanding hasn’t captured?

Or is our original interpretation simply wrong?

The objective isn’t to pursue every inconsistency indefinitely.

Some contradictions are trivial. Others result from unreliable evidence or differences in terminology.

But significant contradictions deserve investigation, particularly when they challenge conclusions that would otherwise influence important decisions.

This is where assessment becomes more than checking conformity against a reference model.

It becomes an exercise in disciplined curiosity.

What Contradictions Reveal

A contradiction doesn’t automatically tell us which observation is correct.

Two conflicting accounts may both be partly correct.

A process may work under normal conditions but fail under pressure.

A capability may be effective in one team but absent elsewhere.

A product may satisfy its documented requirements while failing to achieve its organizational purpose.

In each case, the contradiction may indicate that we have overlooked an important distinction.

Perhaps our assessment is too general.

Perhaps we’re combining observations from different contexts.

Perhaps the reference model needs to be contextualized differently.

Or perhaps we have mistaken a shared organizational belief for demonstrated capability.

The contradiction doesn’t provide the answer.

It tells us that our current understanding may not be sufficient.

When the Model Is Wrong

There is another possibility worth considering.

What if the contradiction isn’t between different pieces of evidence?

What if it’s between the evidence and the model we’re using to interpret it?

A reference model describes concepts and relationships we believe are relevant to understanding a subject.

But the model is not reality.

Suppose an assessment repeatedly encounters effective practices that don’t fit the model’s expectations.

We could interpret those practices as deviations.

Or we could ask whether the model adequately represents the capability we’re trying to understand.

This doesn’t mean abandoning the model whenever evidence becomes inconvenient.

It means recognizing that models are themselves open to examination.

A useful reference model should help us interpret observations.

But observations should also be capable of challenging the model.

Otherwise, assessment risks becoming a mechanism for confirming what we already believe.

Learning From What Doesn’t Fit

Perhaps one of the most important qualities of an assessment is not how much evidence it collects, or how consistently it applies its criteria.

It is whether the assessment remains capable of discovering that its initial understanding was wrong.

Confirming evidence can strengthen an explanation.

Contradictory evidence can reveal where that explanation breaks down.

Both matter.

But they contribute differently to our understanding.

If we only seek agreement, we may produce assessments that are coherent, consistent and reassuring.

If we investigate meaningful contradictions, we may discover something we didn’t know to look for.

Contradiction is not necessarily a problem with the evidence.

Sometimes, it is the evidence that our understanding needs to change.