Marcel van Oosterwijk

Blog

  • Practice Note: An Inconvenient DoD

    Everybody wants quality. Until quality gets in the way.

    Imagine a development team approaching an important deadline. The functionality has been implemented, the documentation updated, and the tests executed. But some tests have failed. Others haven’t been executed. And there are still questions about whether the solution meets its quality expectations.

    The deadline is tomorrow. Is the work done? That depends on the Definition of Done. And that’s where things become interesting.

    The Convenience of Being Done

    A Definition of Done (DoD) establishes the conditions under which work can be considered complete. In practice, many DoDs concentrate on activities: code reviewed, tests executed, documentation updated.

    These are useful activities. But completing them doesn’t necessarily mean achieving their intended results. A review can be completed with significant findings unresolved. Tests can be executed without passing. Documentation can be updated without becoming accurate.

    An activity-based DoD allows us to demonstrate that we’ve performed the expected work without necessarily demonstrating that we’ve achieved the expected quality. And that’s rather convenient when a deadline approaches.

    Three Questions About Done

    A meaningful Definition of Done should address three questions.

    1. Criteria: What must be true for the work to be considered done?

    We need to establish the quality expectations that apply to the result.

    2. Evidence: How do we establish that those criteria have been satisfied?

    Quality expectations aren’t enough. We need evidence supporting the conclusion that they’ve been met.

    3. Consequences: What happens when they haven’t?

    Suppose the evidence is insufficient, or demonstrates that the criteria haven’t been satisfied. Can we still declare the work done? Who decides? And what happens to the deadline?

    The first question defines our expectations. The second makes them verifiable.

    The third determines whether they actually matter.

    Why Quality Is Inconvenient

    Teams are expected to deliver. Project managers are expected to meet deadlines.

    Quality work can interfere with those expectations. Testing reveals defects. Reviews uncover problems. Analysis demonstrates that a solution isn’t fit for its intended purpose. Of course, quality work can prevent expensive rework and improve delivery performance. But those benefits aren’t always immediate.

    The cost of investigating a problem is visible today. The cost of failing to investigate it may emerge months later, perhaps in another team’s budget. A missed deadline is immediately visible. Insufficient quality may not be. This creates an incentive to prioritize what can be demonstrated now over what might matter later.

    And teams adapt to the expectations placed upon them. When defining their DoD, they may gravitate towards conditions they can reliably satisfy without jeopardizing delivery commitments. Not because they don’t care about quality, but because they’re responding to the organization’s actual priorities.

    The DoD becomes compatible with the incentives of the organization.

    The Convenient Quality Gate

    A DoD that says tests have been executed is easier to satisfy than one requiring specified tests to pass and unresolved failures to be evaluated. A DoD that says documentation has been updated is easier to satisfy than one requiring evidence that it’s accurate and sufficient for its intended users. The difference is that the second formulation may prevent the team from declaring the work done.

    We can make a DoD appear rigorous by specifying activities while leaving the evidence of quality insufficiently defined. Whether consciously or unconsciously, this allows us to avoid the consequences of a genuine quality gate.

    Other industries offer useful perspectives. In safety-critical aviation development, verification evidence is essential to demonstrating compliance. In medicine, surgical safety procedures establish checks that must be addressed before proceeding. Neither approach guarantees perfect outcomes. Nor should ordinary software development adopt their practices wholesale.

    But they illustrate an important principle:

    Quality obligations can be conditions for proceeding, rather than considerations to be abandoned when delivery becomes difficult.

    What Happens When We Aren’t Done?

    Suppose we’ve defined meaningful criteria and the evidence needed to demonstrate that they’ve been satisfied. Now the deadline arrives. The evidence is insufficient. What happens?

    We might delay delivery, reduce scope, complete the necessary verification, or explicitly accept certain risks where that’s permissible. These can all be legitimate decisions. The important point is that someone must make one. We can no longer quietly transform incomplete assurance into completed work.

    And that decision may be uncomfortable. It may affect commitments, budgets or schedules. Someone may have to accept responsibility for a risk they would rather not acknowledge. But that’s precisely why the Definition of Done matters.

    A quality gate that cannot influence a delivery decision isn’t much of a gate.

    The Real Test

    We often discuss Definitions of Done as methodological instruments. We debate their contents and how teams should formulate them.

    But perhaps the more important question is organizational:

    Are we prepared to accept the consequences of defining Done properly?

    If management rewards meeting deadlines while treating quality as negotiable, teams have little incentive to create a DoD that might prevent them from declaring success.

    The vague DoD is where the problem becomes visible. The organizational incentives may be where it originates. And imposing stricter DoDs won’t necessarily help if those incentives remain unchanged.

    A Definition of Done is most valuable when it tells us something we don’t want to hear. Everybody wants quality. Until quality gets in the way.

    That’s when we discover whether our Definition of Done is a genuine quality gate—or merely a convenient way to declare that we’re finished.

    The next time your team fails to meet its Definition of Done, ask yourself: what happens next?

    If the answer is “We deliver anyway”, perhaps the Definition of Done isn’t where the improvement needs to start.

    Improving quality isn’t always about introducing better practices. Sometimes it starts with understanding why the practices we already have aren’t allowed to work.

  • Blog Retrospective #6

    I have now reached 120 posts.

    In the previous retrospective, I observed that the Body of Knowledge had begun behaving like a learning system. Practice Notes generated theory, theory generated approaches, and those approaches generated new observations that could flow back into the Practice Notes.

    During the latest twenty posts, that process continued.

    But something else happened.

    The inquiry increasingly turned towards the reliability of the understanding we produce.

    Not just what we know about products, capabilities and organizations, but how we know it, what our evidence actually supports, and whether our models might themselves be wrong.

    What Happened?

    The latest cycle explored several apparently different subjects.

    Some notes examined transformation, projects and interventions. Others explored product reference models, contextualization, organizational knowledge and AI. A few returned to everyday observations, including watches and model railway automation.

    The final group emerged directly from developing a standard approach to capability assessment.

    That work raised questions about evidence, contradictions, assessment confidence, sampling and the quality of assessments themselves.

    At first glance, these might seem like separate lines of inquiry.

    But I think they share an underlying concern.

    How do we establish that the mechanisms we use actually achieve their intended purpose?

    A project can deliver successfully without producing the intended change.

    An intervention can be completed without improving organizational capability.

    A product can be extensively documented without being properly understood.

    An assessment can contain considerable evidence without producing a justified conclusion.

    The presence of a mechanism doesn’t establish that the mechanism works.

    That observation has appeared throughout the blog. During this cycle, it became particularly difficult to ignore.

    From Reference Models to Context Layers

    One development I hadn’t anticipated was the connection between reference models and organizational knowledge.

    Earlier work on reference models focused primarily on assessment.

    A reference model provides a conceptual structure against which we can examine a subject. Contextualization helps us interpret that structure in relation to a particular organization, capability or product.

    But what if contextualized reference models could serve another purpose?

    Organizations already possess enormous Bodies of Knowledge. They contain documents, decisions, observations, assessments and accumulated experience.

    AI can make much of this information accessible.

    But accessibility isn’t necessarily understanding.

    Knowing that several documents mention requirements management doesn’t automatically establish how requirements management relates to organizational capabilities, product characteristics, decisions or outcomes.

    Perhaps contextualized reference models could provide part of the structure needed to interpret those relationships.

    That led to the idea of context layers: representations connecting generic concepts to particular organizational subjects and the evidence available about them.

    An interrogable organization might then be able to ask questions that require information from many different sources.

    Not merely retrieving documents, but investigating relationships.

    There is still an important open question.

    How much context must be explicitly modeled, and how much can AI reliably reconstruct when needed?

    I don’t know.

    But the possibility changes how I think about reference models.

    They might be useful not only as instruments for assessment, but also as part of the infrastructure through which an organization understands itself.

    When the Assessment Approach Started Asking Questions

    Perhaps the most interesting development came from working on the capability Assessment Approach.

    The approach was intended to operationalize ideas already developed in the Practice Notes and Continuity work.

    But making those ideas operational exposed questions that deserved attention in their own right.

    One concerned contradictions.

    What should an assessor do when interviews, documentation, observed behaviour and outcomes disagree?

    The obvious response is to reconcile the evidence.

    But perhaps the disagreement itself reveals something important.

    Why does management believe a process works when outcomes repeatedly suggest otherwise?

    Why does a team achieve excellent results despite apparently weak practices?

    Perhaps the contradiction exposes a weakness in our understanding rather than a weakness in the evidence.

    Another question concerned confidence.

    An assessment might conclude that a capability is mature, but how strongly does the evidence support that conclusion?

    A Level 4 capability assessment based on limited evidence is different from a Level 2 assessment supported by extensive observation.

    The assessed condition and our confidence in that condition are different dimensions.

    Then came sampling.

    An assessment can examine evidence rigorously and still produce a misleading conclusion if the evidence represents only a favourable part of the organization.

    We might establish that one team performs effectively without establishing that the organization can reproduce that performance elsewhere.

    Finally, the assessment itself became the subject of inquiry.

    An assessment report may claim to be evidence-based.

    But what justifies that claim?

    Were the right questions asked? Was the evidence representative? Were contradictions investigated? Were conclusions stronger than the evidence warranted?

    That produced Who Assesses the Assessment?

    By the time I finished that note, I was beginning to wonder whether I would ever dare conduct another assessment.

    When Disciplined Curiosity Becomes Indiscipline

    That reaction was useful.

    Each additional question about assessment quality seemed reasonable.

    But taken together, they threatened to make assessment impossibly demanding.

    Every conclusion could be challenged.

    Every sample could be questioned.

    Every interpretation could have alternatives.

    And every assessment of an assessment could presumably require another assessment.

    At some point, the pursuit of justified understanding can become an obstacle to using that understanding.

    This brought me back to Disciplined Curiosity.

    The purpose of assessment is not to eliminate uncertainty.

    It is to develop sufficiently justified understanding for the decisions we need to make.

    The necessary rigor depends on the purpose of the assessment, the consequences of getting it wrong and the resources available.

    Further investigation has a cost.

    So does acting on insufficient understanding.

    Professional judgment includes knowing when additional inquiry is worthwhile.

    Perhaps that is something the Assessment Approach still needs to express more clearly.

    Moving Upstream

    Another recurring theme became more explicit during this cycle.

    A professional can perform an assignment correctly without achieving the purpose that motivated it.

    A tester can thoroughly test the wrong requirements.

    A project manager can deliver a project that produces no meaningful change.

    A consultant can implement an improvement that doesn’t address the underlying problem.

    This led to the question of whether moving upstream is itself a professional quality.

    Perhaps professional competence involves more than executing an assignment.

    It also involves understanding why the assignment matters and recognizing when successful execution may not achieve its intended purpose.

    The AI-related notes added another perspective.

    When AI removes some of the friction involved in implementation, more attention can move towards intent, evidence and judgment.

    That doesn’t make implementation knowledge irrelevant.

    But it may change where human attention creates the most value.

    Looking back, moving upstream has been a recurring movement throughout the blog.

    Testing led to quality.

    Quality led to organizational capabilities.

    Capabilities led to organizational purpose.

    Projects led to interventions and their effects.

    Documentation led to understanding.

    And assessment results led to questions about the justification for those results.

    The subjects changed.

    The underlying curiosity remained remarkably consistent.

    What Surprised Me?

    The biggest surprise was how naturally the Assessment Approach generated new Practice Notes.

    I had expected the approach to apply existing ideas.

    Instead, operationalizing those ideas exposed distinctions that hadn’t previously received sufficient attention.

    That is encouraging.

    It suggests that the relationship between Practice Notes, theory and application can work in both directions.

    But I also noticed a potential danger.

    The ideas are becoming increasingly coherent.

    Almost every new observation can be connected to something already present in the Body of Knowledge.

    That makes synthesis easier.

    It also creates the risk that the framework becomes too good at explaining everything.

    If every observation can be interpreted as supporting the theory, how would we recognize evidence that challenges it?

    The recent note Contradiction Is Information applies just as much to this Body of Knowledge as it does to organizational assessments.

    Perhaps the next important development won’t be another concept that fits neatly into the existing framework.

    Perhaps it will be an observation that doesn’t fit.

    Looking Back at the Whole Blog

    At 120 posts, the scope of the inquiry has changed considerably.

    It began largely with questions about software quality, testing and assessment.

    Those questions led towards organizational capabilities and the conditions required to produce reliable outcomes.

    From there, the inquiry expanded towards products, portfolios, Subject Stewardship and Continuity.

    Then came organizational knowledge, the preservation of decisions and evidence, and the possibility of making accumulated knowledge interrogable.

    The latest cycle has increasingly examined the quality of the understanding produced by those mechanisms.

    There is an interesting symmetry here.

    The early notes questioned whether testing activities were sufficient to establish product quality.

    The recent notes question whether assessment activities are sufficient to establish justified understanding.

    Both challenge the assumption that performing an activity demonstrates that its purpose has been achieved.

    And both lead to questions about the capabilities and conditions needed to produce the intended outcome.

    Perhaps the blog hasn’t moved as far from its original subject as it sometimes appears.

    The subject of Quality Management has simply become broader.

    What Concerns Me?

    I see two concerns worth carrying into the next cycle.

    The first is that abstraction may be moving faster than practical application.

    Some of the newer ideas, particularly around context layers and interrogable organizational knowledge, remain propositions rather than demonstrated approaches.

    They need to encounter reality.

    The second is that the growing coherence of the Body of Knowledge may make it increasingly difficult to challenge its assumptions.

    I don’t want the Practice Notes to become a collection of arguments explaining why the existing theory is correct.

    They should remain a place where observations can challenge the theory, including observations that make earlier conclusions uncomfortable.

    That is also why I think the distinction between observations, interpretations and hypotheses deserves continued attention.

    Not everything that follows logically from an idea has been demonstrated in practice.

    What Next?

    I don’t think the next step necessarily requires another major theoretical synthesis.

    I would rather see what happens when some of these ideas are applied.

    What would a contextualized product reference model reveal about an actual product?

    Would context layers improve an AI assistant’s ability to investigate organizational questions?

    Would the Assessment Approach expose weaknesses or contradictions that a conventional assessment might overlook?

    And perhaps most importantly, what would happen when one of these approaches failed to work as expected?

    Those failures might generate more valuable Practice Notes than another successful conceptual synthesis.

    A Final Observation

    The first retrospectives were largely concerned with the writing process and the ideas emerging from it.

    Later retrospectives examined how the Practice Notes began connecting, generating theory and feeding applications.

    This retrospective has turned towards something different.

    The Body of Knowledge is increasingly examining the mechanisms through which it develops understanding.

    That feels like progress.

    But it also creates a responsibility.

    If I argue that organizations should challenge their assumptions, investigate contradictions and evaluate the evidence supporting their conclusions, the same expectations should apply to my own work.

    The objective cannot simply be to develop an increasingly coherent theory.

    It must remain possible for experience to change that theory.

    Perhaps the most important question for the next retrospective is not what new ideas have emerged.

    It is what I have learned that forced me to reconsider something I previously believed.

  • Practice Note: Who Assesses the Assessment?

    An organization commissions an assessment of its software development capability.

    The assessors conduct interviews, examine documentation, observe practices and evaluate the available evidence.

    A few weeks later, management receives a report.

    The organization’s software development capability has been assessed at maturity Level 3.

    The report contains findings, supporting evidence and recommendations for improvement.

    Management now has a basis for making decisions.

    Or does it?

    We know what the assessors concluded.

    But how do we know whether their conclusions deserve our confidence?

    Who assesses the assessment?

    Evidence-Based Is a Claim

    Assessment reports often describe their conclusions as evidence-based.

    That sounds reassuring.

    But what does it actually establish?

    Perhaps the assessors examined fifty documents, interviewed twenty people and analyzed several years of performance data.

    That’s a considerable amount of information.

    Yet the quantity of evidence tells us relatively little about whether the assessment was sound.

    Were the right questions asked?

    Was the evidence relevant to those questions?

    Were the observations representative of the subject being assessed?

    Were contradictory findings investigated?

    Did the assessors distinguish observations from interpretations?

    And did their conclusions go beyond what the evidence could reasonably support?

    An assessment can contain extensive evidence while drawing poorly justified conclusions.

    “Evidence-based” is itself a claim that requires justification.

    Assessing the Mechanism

    An assessment is a mechanism for developing justified understanding of a subject.

    It involves selecting a frame of reference, gathering evidence, interpreting observations and synthesizing conclusions.

    Each step introduces choices.

    The reference model influences what we consider relevant.

    The sampling strategy determines which parts of reality we examine.

    The evidence sources influence what we can observe.

    The interpretation determines what meaning we assign to those observations.

    And the synthesis determines which conclusions we ultimately communicate.

    These choices affect the result.

    So perhaps evaluating an assessment requires examining not only its conclusions, but also the mechanism through which those conclusions were produced.

    Not to demand that every assessment follow an identical procedure.

    But to establish whether its approach was appropriate for the subject, purpose and consequences of the assessment.

    A Plausible Conclusion Can Still Be Wrong

    Imagine an assessment concluding that an organization has weak requirements management.

    The conclusion sounds plausible.

    Projects experience rework. Stakeholders complain about unclear requirements. Development teams frequently request clarification.

    But suppose the assessors never investigated how requirements are actually developed and managed.

    They inferred a capability weakness from symptoms observed elsewhere.

    Perhaps requirements management really is weak.

    But perhaps the underlying problem lies in decision-making, product ownership or conflicting organizational priorities.

    The assessment may have identified a genuine problem while attributing it to the wrong cause.

    That distinction matters.

    If management acts on the assessment, it may invest in improving requirements practices without addressing the mechanism responsible for the observed difficulties.

    The recommendations may be sensible.

    The evidence may be authentic.

    And the assessment may still have failed to justify its central interpretation.

    Can We Follow the Reasoning?

    One way to examine an assessment is to ask whether its reasoning can be reconstructed.

    What was observed?

    Which evidence supports that observation?

    How was the observation interpreted?

    Which assumptions influenced the interpretation?

    What alternative explanations were considered?

    And how did these elements contribute to the final conclusion?

    If we cannot follow that reasoning, we have limited ability to evaluate the assessment.

    We may have to trust the assessor’s authority rather than the justification for the conclusion.

    That doesn’t mean every judgment can be reduced to a mechanical calculation.

    Professional assessment involves interpretation.

    But interpretation should be open to examination.

    A defensible assessment should make it possible to understand not only what the assessor concluded, but why.

    The Problem of Independent Review

    Perhaps the obvious answer is to have another assessor review the assessment.

    That could help.

    A second assessor might identify overlooked evidence, unsupported conclusions or alternative interpretations.

    But independence alone doesn’t guarantee a better result.

    Two assessors can share the same assumptions.

    They can rely on the same reference model.

    They can interpret the same evidence through the same professional conventions.

    Agreement between them may increase confidence, but it doesn’t automatically establish correctness.

    And this introduces a recursive problem.

    If an assessment needs to be assessed, does the assessment of that assessment also need to be assessed?

    And so on?

    We could continue indefinitely.

    But perhaps the objective isn’t to establish absolute certainty.

    It is to make consequential judgments sufficiently transparent, justified and open to challenge.

    An assessment should be capable of explaining and defending its conclusions without requiring an endless chain of additional assessments.

    The Assessment as a Subject

    There is another way to think about this.

    We assess organizational capabilities because we want to understand whether they can reliably achieve their intended outcomes.

    But assessment is itself an activity intended to achieve an outcome.

    It should produce sufficiently justified understanding to support decisions.

    That means assessment has characteristics we can examine.

    Does it use an appropriate frame of reference?

    Does it seek relevant and representative evidence?

    Does it distinguish claims from observations?

    Does it investigate significant contradictions?

    Does it preserve uncertainty?

    Does it avoid conclusions stronger than the available evidence?

    These are not merely procedural requirements.

    They concern the quality of the assessment’s ability to achieve its purpose.

    In that sense, the assessment itself becomes a subject of assessment.

    When AI Performs the Assessment

    This becomes particularly interesting when AI enters the picture.

    Imagine an AI system analyzing fifty thousand organizational documents.

    It identifies recurring patterns, connects observations and produces an assessment of several organizational capabilities.

    The result is detailed, coherent and supported by references to source material.

    It may even be more comprehensive than a conventional assessment.

    But what does that tell us about its reliability?

    Did the AI distinguish independent evidence from repeated claims?

    Did it identify contradictions?

    Did it recognize that certain parts of the organization were poorly represented?

    Did it confuse descriptions of intended practices with observations of actual behaviour?

    Did it consider alternative explanations?

    And can we reconstruct how it arrived at its conclusions?

    The fact that AI examined fifty thousand documents doesn’t establish that its conclusions are justified.

    Nor does the presence of source references.

    The same questions apply whether an assessment is performed by a human, an AI system or a combination of both.

    The mechanism through which evidence becomes judgment still matters.

    Trust Through Challengeability

    Perhaps the quality of an assessment cannot be established merely by checking whether it followed a prescribed method.

    A method can be followed correctly while producing misleading conclusions.

    And a sound assessment may sometimes require departing from the original plan when unexpected evidence emerges.

    What matters is whether the assessment can justify its approach and conclusions in relation to its purpose.

    Can we examine the evidence?

    Can we follow the reasoning?

    Can we identify the assumptions?

    Can we understand the uncertainty?

    Can we challenge the conclusions?

    And can the assessment change when those challenges reveal weaknesses?

    This doesn’t eliminate the need for professional judgment.

    It makes that judgment accountable.

    An assessment should not demand trust simply because it was performed by an expert, followed a methodology or processed a large amount of evidence.

    It should earn confidence through the quality of its justification.

    An assessment is not trustworthy merely because it is evidence-based.

    It becomes trustworthy to the extent that its use of evidence can itself withstand examination.

  • Practice Note: The Organization You Didn’t Assess

    An organization wants its software development capability assessed.

    Management selects three projects for examination.

    All three are successful. They have experienced teams, established practices and satisfied stakeholders.

    The assessors conduct interviews, examine documentation, observe activities and investigate outcomes.

    The evidence is consistent.

    The organization appears to have a mature software development capability.

    But what about the projects that weren’t selected?

    The struggling project with constantly changing requirements?

    The team that depends on one exceptionally experienced developer?

    The product that repeatedly fails to meet its release commitments?

    Perhaps those projects would tell a different story.

    And perhaps the assessment is perfectly accurate about the projects it examined.

    The problem may not be the evidence. It may be what we believe the evidence represents.

    What Have We Actually Assessed?

    An assessment never examines an entire organization.

    It examines selected aspects of the organization, through selected evidence, under particular conditions.

    We interview certain people.

    We examine particular projects.

    We observe specific activities.

    We investigate outcomes from a limited period.

    From these observations, we develop conclusions about the subject being assessed.

    But there is a difference between what we have observed and what we can reasonably conclude.

    Suppose we observe a team consistently producing high-quality software.

    We have evidence that this team can achieve good results under the conditions examined.

    Does that demonstrate that the organization has a mature software development capability?

    Not necessarily.

    It may demonstrate that the organization has at least one team capable of producing good results.

    That is useful information.

    But it is a narrower conclusion.

    Capability Under Favourable Conditions

    Consider a team that delivers successfully because it has unusually experienced people.

    Its requirements are clear. Its stakeholders are available. Its technical environment is stable.

    The team consistently produces good results.

    Now imagine changing the conditions.

    A key developer leaves.

    Several stakeholders disagree about priorities.

    Requirements become uncertain.

    The team must maintain an unfamiliar product while continuing development.

    Would it still perform effectively?

    Perhaps.

    But the original evidence doesn’t necessarily establish that.

    This matters because organizational capability concerns more than the ability to succeed under ideal circumstances.

    It concerns the ability to achieve intended outcomes with sufficient reliability under the conditions the organization can reasonably expect to encounter.

    A successful project demonstrates that success is possible.

    It doesn’t automatically demonstrate that success is reproducible.

    The Selection Problem

    Sampling is unavoidable.

    Assessors cannot investigate every team, project, product, decision and outcome.

    They must select where to look.

    But that selection influences what they are capable of discovering.

    If we examine only successful projects, we may underestimate recurring weaknesses.

    If we interview only management, we may miss the differences between intended and actual practices.

    If we investigate only recent releases, we may overlook problems that emerge during long-term maintenance.

    If we observe only normal operations, we may miss how the organization behaves under pressure.

    The issue isn’t simply the number of observations.

    A large sample can still be misleading if it systematically excludes relevant conditions.

    Assessment quality depends partly on whether our inquiry exposes itself to situations in which its emerging conclusions might fail.

    That doesn’t mean deliberately searching only for problems.

    It means selecting evidence that allows us to understand the relevant variation in the subject being assessed.

    The Organization Is Not Uniform

    There is another complication.

    Organizations are rarely homogeneous.

    Different teams may operate under different constraints.

    Products may have different quality expectations.

    Some activities may be highly standardized, while others depend on local expertise.

    A capability may be mature in one organizational unit and weak in another.

    What does an organization-wide maturity rating mean under those circumstances?

    Perhaps it describes the dominant pattern.

    Perhaps it describes the weakest relevant part.

    Perhaps it represents an aggregation of several distinct assessments.

    Each approach may serve a purpose.

    But they are not interchangeable.

    A single rating can conceal important variation unless we understand what it represents.

    And sometimes the variation is itself the most interesting finding.

    Why can one team achieve results that others cannot?

    What enables its success?

    Can those conditions be reproduced?

    Or is the apparent organizational capability actually concentrated in a few exceptional individuals?

    Those questions may be more valuable than an average maturity score.

    Accurate Evidence, Unjustified Conclusions

    This creates an interesting possibility.

    An assessment can be rigorous, evidence-based and accurate — yet still support a misleading organizational conclusion.

    The interviews may have been conducted properly.

    The observations may be reliable.

    The documentation may be authentic.

    The findings may accurately describe the selected projects.

    But if we generalize those findings beyond the conditions they represent, we claim more than the evidence justifies.

    The weakness isn’t necessarily in how we interpreted the observations.

    It may be in how we defined the population to which our conclusions apply.

    This is why assessment confidence cannot be separated from representativeness.

    We might be highly confident that a particular team performs effectively.

    And much less confident that the organization can reproduce that performance elsewhere.

    Both judgments can be justified by the same evidence.

    They simply concern different claims.

    Making the Boundaries Visible

    Perhaps every assessment should make its boundaries explicit.

    Which parts of the organization were examined?

    Which operating conditions were represented?

    Which relevant variations were investigated?

    What was deliberately excluded?

    And how far can the conclusions reasonably be generalized?

    These questions don’t make an assessment less useful.

    They help establish what its findings actually mean.

    Sometimes a limited assessment is entirely appropriate.

    We may only need to understand one team, one product or one particular capability under specified conditions.

    The problem arises when a narrow investigation is presented as evidence of something broader.

    A focused assessment can support a focused conclusion.

    It should not silently become an organization-wide judgment.

    What We Didn’t See

    There is a temptation to believe that assessment quality is primarily a matter of collecting and interpreting evidence correctly.

    Those things are essential.

    But they aren’t sufficient.

    We must also consider whether the investigation gave us a reasonable opportunity to discover that our initial understanding was wrong.

    Did we examine meaningful variation?

    Did we investigate conditions under which the capability might struggle?

    Did we look beyond the examples the organization was most comfortable showing us?

    And did we preserve the distinction between what we observed and what we inferred?

    An assessment doesn’t reveal the organization in its entirety.

    It reveals aspects of the organization under the conditions we chose to examine.

    The challenge is to understand what those observations justify — without assuming they represent everything we didn’t see.

    An assessment is defined not only by the evidence it examines, but also by the reality it leaves unexamined.

  • Practice Note: How Certain Are We About Our Assessment?

    Imagine assessing two organizational capabilities.

    The first receives a maturity rating of Level 4. Interviews suggest that practices are well established, and the available documentation supports that impression. However, we have examined little evidence of actual behaviour or outcomes.

    The second receives a maturity rating of Level 2. We have observed the work, examined several projects, investigated outcomes and found consistent evidence of significant weaknesses.

    Which assessment is stronger?

    The first capability has the higher maturity rating.

    But the second assessment has considerably stronger evidence supporting its conclusion.

    We are describing two different things.

    The condition we believe a capability is in, and how confident we are in that belief.

    Two Different Dimensions

    A capability assessment attempts to establish something about an organization’s ability to achieve particular outcomes.

    We examine evidence, interpret observations and arrive at a judgment.

    Perhaps we express that judgment using a maturity scale.

    Level 2. Level 3. Level 4.

    But the rating alone doesn’t tell us how well the conclusion is supported.

    A Level 4 assessment based primarily on interviews is not equivalent to a Level 4 assessment supported by extensive observations of actual behaviour and outcomes.

    The assessed condition may be the same.

    Our confidence in that condition may be very different.

    This suggests that capability assessments should distinguish between two dimensions:

    • Assessed condition: What does the available evidence suggest about the capability?
    • Assessment confidence: How strongly does that evidence justify our conclusion?

    Neither dimension can replace the other.

    When Evidence Is Missing

    Suppose we assess a team that appears to perform exceptionally well.

    Stakeholders are satisfied. Deliveries are predictable. Defects are relatively uncommon.

    But we have limited access to the team’s actual work.

    We cannot establish how its practices operate, whether its results are sustainable or how it responds when conditions change.

    What should we conclude?

    We might suspect that the team has a mature capability.

    But we cannot justify that conclusion with much confidence.

    Does that mean the capability is weak?

    Not necessarily.

    Insufficient evidence of capability is not the same as evidence of insufficient capability.

    This distinction matters because assessments can unintentionally penalize subjects that are difficult to observe.

    The absence of evidence may reflect limitations in our investigation rather than limitations in the capability itself.

    Of course, the ability to demonstrate capability may itself be important.

    For some capabilities, maintaining records, providing traceability or demonstrating control is part of what good performance requires.

    But then we should assess those expectations explicitly.

    We shouldn’t confuse our inability to establish a condition with evidence that the condition is poor.

    Confidence Changes the Next Decision

    Consider two assessment results.

    Capability A is assessed at Level 4, with low confidence.

    Capability B is assessed at Level 2, with high confidence.

    What should management do?

    For Capability A, the immediate need may be further investigation.

    We have reason to believe the capability is effective, but insufficient evidence to rely confidently on that conclusion.

    For Capability B, additional assessment may contribute relatively little.

    We already have strong evidence of weaknesses. The more useful next step may be to investigate their causes and consider improvement interventions.

    The maturity ratings suggest different capability conditions.

    The confidence levels suggest different degrees of uncertainty about those conditions.

    Together, they inform what to do next.

    Without this distinction, organizations may invest in improving capabilities that haven’t actually been demonstrated to be weak.

    Or they may trust favourable assessments that were never adequately substantiated.

    Confidence Is Not a Property of the Capability

    There is another subtle distinction.

    Confidence doesn’t belong to the capability itself.

    It belongs to our assessment of it.

    The capability exists and operates independently of whether we understand it correctly.

    Our assessment is an interpretation based on selected evidence, observed under particular conditions, at a particular time.

    That interpretation may be well supported.

    Or it may be uncertain.

    It may also be wrong.

    This means that assessment confidence should reflect more than the quantity of evidence collected.

    We need to consider its relevance, independence, reliability and representativeness.

    Ten documents repeating the same claim may provide less justification than one direct observation of actual behaviour.

    Consistent evidence from several ordinary projects may tell us little about how the capability performs under exceptional conditions.

    And contradictory evidence may reduce our confidence even when most observations support the same conclusion.

    Confidence is therefore not simply a measure of how much evidence we have.

    It concerns how well that evidence supports the particular judgment we are making.

    The Temptation of a Single Number

    Organizations often prefer simple assessment results.

    A maturity rating is easy to communicate.

    It can be displayed in a dashboard, compared across organizational units and tracked over time.

    Adding confidence complicates the picture.

    But perhaps that complication reflects reality.

    Consider an organization whose assessed maturity increases from Level 2 to Level 4.

    Has the capability improved?

    Possibly.

    But perhaps the first assessment was based on extensive observation, while the second relied primarily on management interviews.

    Or perhaps the capability hasn’t changed at all, but the second assessment used better evidence and corrected an earlier underestimate.

    A change in the assessed rating does not necessarily demonstrate a change in the underlying capability.

    To interpret the difference, we also need to understand how the assessments were conducted and how strongly their conclusions were supported.

    Otherwise, apparent improvement may partly reflect changes in what we know rather than changes in what the organization can do.

    Preserving Uncertainty

    An assessment is intended to support decisions.

    That creates pressure to produce clear conclusions.

    Management wants to know whether a capability is good enough, where improvement is needed and what should happen next.

    Uncertainty can seem inconvenient.

    But concealing uncertainty doesn’t make a conclusion more reliable.

    It merely makes its limitations less visible.

    A defensible assessment should therefore communicate not only what we believe, but also how strongly we can justify that belief.

    Sometimes the appropriate conclusion is that a capability appears mature, but the evidence is insufficient to establish this confidently.

    Sometimes it is that a capability is demonstrably weak.

    And sometimes the available evidence simply doesn’t justify a meaningful rating.

    These are different situations.

    They should not be made to look identical.

    Assessing Our Understanding

    Perhaps this distinction extends beyond capability assessment.

    Whenever we make an evidence-based judgment, we are making two claims.

    One concerns the subject.

    The other concerns the quality of our understanding of that subject.

    We might believe a product is reliable, an intervention is effective or an organizational capability is mature.

    But how confidently can we make that claim?

    And what evidence would cause us to reconsider it?

    An assessment that acknowledges uncertainty isn’t necessarily weaker than one that produces a definitive answer.

    It may simply be more accurate about what has actually been established.

    The condition of a subject and our confidence in that condition are two different things.

    A good assessment should help us understand both.