Marcel van Oosterwijk

Category: Assessment

Quality, testing, assessment, evidence, judgment, reference models, and the development and evaluation of quality and capability.

  • Practice Note: Knowing When to Stop

    Curiosity has probably driven more of this blog than anything else. I like asking why. Why does this process exist? Why was this decision made? Why does this product behave this way? Why did an organization choose one path rather than another?

    Quite often, asking Why is rewarding. One answer reveals another question, and following the chain can expose assumptions, explain strange decisions, reveal inconsistencies or lead somewhere entirely unexpected. Much of what I find interesting begins this way.

    For a long time, I think I also carried an implicit assumption: if I kept asking long enough and searched hard enough, eventually I would find the answer.

    I am less certain of that now.

    When the Evidence Runs Out

    Some questions do not resist answering because we have not investigated them thoroughly enough. Sometimes the available evidence simply runs out. The people who knew may be gone. The decision may never have been recorded. The surviving sources may contradict one another. Several explanations may remain plausible without any way to establish which one was actually true.

    There is always the temptation to continue. Perhaps one more source will resolve it. Perhaps another interpretation will make everything fit. Perhaps if we reason carefully enough, the missing answer can still be reconstructed. Sometimes it can.

    But there comes a point where further reasoning stops producing knowledge and starts producing speculation. Recognizing that boundary is difficult precisely because speculation can be convincing. A plausible explanation can become a likely explanation, and a likely explanation can gradually acquire the status of fact. The more effort we have invested in finding an answer, the harder it may become to accept that we have not found one.

    Yet there is an important difference between saying that there is no answer and saying that I cannot justify an answer. The first makes a claim about reality. The second acknowledges the limits of what I know. Often, the second is all the evidence allows.

    The Discipline of Curiosity

    Curiosity is valuable because it refuses to accept unexplained things too easily. It keeps asking when the first answer is superficial. It challenges assumptions that have become invisible through familiarity. It encourages us to look behind the artifact, process or decision and understand what it rests upon.

    But perhaps rigorous curiosity also requires knowing when to stop. If every question must eventually produce an answer, then uncertainty becomes something that has to be eliminated. Once the evidence can no longer do that for us, we become tempted to eliminate it ourselves.

    That is where curiosity can undermine the very understanding it was supposed to create. The objective was never merely to have an answer. It was to have an answer we have sufficient reason to believe. Sometimes those are not the same thing.

    Becoming Comfortable With Unknown

    I have therefore become increasingly comfortable with I don’t know as a legitimate conclusion. That does not mean the investigation failed. Establishing that something is genuinely unknown can itself be useful. It tells us where justified understanding ends. It prevents an inference from quietly becoming historical fact. It leaves uncertainty visible instead of hiding it behind a convincing explanation.

    Nor does I don’t know mean nobody will ever know. New evidence may appear. Someone may remember something. Another source may surface. A different line of investigation may eventually resolve the question.

    Stopping simply means recognizing that, with the evidence currently available, I have reached the limit of what I can honestly justify. The question can remain open without requiring me to fill the space where its answer should be.

    Reflection

    I still ask why. Probably too often. I still enjoy following questions further than is strictly necessary, particularly when one Why unexpectedly reveals another. I do not think that will change.

    What has changed is my expectation of where that journey must end. I no longer believe that every Why owes me an answer. Sometimes the evidence leads to a confident conclusion. Sometimes it supports only a hypothesis. And sometimes, after all the questioning and searching, the most accurate conclusion available is simply:

    I don’t know.

    Knowing when to ask Why matters.

    Knowing when to stop may matter just as much.

  • Practice Note: From Improvement to Management?

    Organizations constantly invest in improving capabilities. Testing becomes inconsistent, so an improvement initiative is launched. Requirements quality declines, so new templates, training and governance are introduced. Architecture loses direction, so a maturity program begins. Security, DevOps, business analysis and many other disciplines follow the same pattern.

    The sequence is familiar. A problem is identified, an assessment is performed, interventions are designed, people are trained, practices are changed, tools are introduced and new ways of working are established.

    Eventually the initiative ends. The consultants leave. The project team dissolves. The transformation roadmap is closed. Everyone returns to normal work.

    A few years later, the same capability is often under discussion again. Testing has become inconsistent. Requirements quality has deteriorated. Architectural direction has weakened. Another maturity initiative is needed.

    This raises a simple question. Perhaps capability improvement is not the same thing as capability management.

    Improvement Has an End Date

    Improvement initiatives are temporary by design. They exist because the organization has identified a gap between its current ability and the ability it believes it needs. Their purpose is to create change.

    That makes them valuable, but it also gives them a natural ending. Once the new practices have been introduced, the training delivered and the immediate objectives achieved, the initiative can reasonably be considered complete.

    The capability, however, is not complete. The environment continues to change. New technologies appear. People leave. New people join. Products evolve. Business priorities shift. Practices that once worked well become less appropriate. Dependencies on neighboring capabilities change. Lessons accumulate. Organizational ability therefore continues to evolve long after the improvement project has ended.

    Improvement projects change organizations.

    Capability Management sustains organizational ability.

    Those are different responsibilities.

    What Happens After Improvement?

    Organizations often recognize this problem informally. When testing becomes a recurring concern, someone eventually hears:

    “You are now responsible for testing.”

    The intention is sensible. Somebody should make sure the organization does not lose what it has just improved. But the statement hides a difficult question. What exactly has that person become responsible for? The testing process? Testing tools? Competence development? Communities of practice? Governance? Quality assurance? Hiring? Methodology?

    Trying to answer the question by listing artifacts and activities quickly becomes problematic, because none of them is the capability itself. A capability is the organizational ability that those arrangements collectively enable.

    The more useful responsibility may therefore be neither ownership of a process nor control of every specialist activity. It may be stewardship of the organizational ability.

    Managing the Capability, Not Its Realization

    A capability can be realized in many different ways. Practices can change. Tools can change. Organizational structures can change. Governance can become more or less formal. Responsibilities can move between teams. New technologies may make previously important activities unnecessary. If a Capability Management becomes attached too strongly to one of those realizations, it risks preserving the current solution rather than the underlying ability.

    The responsibility should instead remain connected to the capability itself. What organizational ability do we need? Why do we need it? Under which conditions must it work? What evidence tells us that the ability actually exists? Where is it becoming weaker? What changes in the environment require it to evolve? What have recent experiences taught us?

    Those questions operate on a longer timescale than an improvement project. They concern continuity.

    A Parallel with Product Management

    This begins to resemble Product Management. Product Management does not need to own every specialist discipline contributing to a product. Architecture remains architecture. Engineering remains engineering. Testing remains testing. Operations remains operations.

    What Product Management provides is continuity around the product itself. It maintains the connection between why the product exists, what it needs to become, what is currently being built, what evidence returns from use and what should change next. Specialist knowledge remains distributed, but the subject does not disappear between projects.

    Organizational capabilities may require something similar. Capability Management would not replace the specialists who realize the capability. It would maintain continuity around the capability while those specialists continue to contribute their expertise.

    That is a different proposition from appointing someone to “own testing.” It suggests a management discipline rather than merely a role.

    When Nobody Manages the Capability

    The absence of such stewardship may help explain another recurring pattern. Capabilities often expand into neighboring areas. Testing begins improving requirements because poor requirements make testing difficult. Architecture becomes involved in operational concerns because architectural decisions repeatedly create operational problems. Security begins governing development practices because insecure development creates security risk.

    These interventions may be entirely reasonable. Capabilities are interdependent, and specialists should influence the conditions that affect their ability to succeed.

    But there is an important difference between influence and accidental ownership. Testing should be able to challenge weak requirements. That does not necessarily mean Testing should become responsible for Requirements Engineering. Security should influence development practices. That does not automatically mean Security should become the organization’s steward of software engineering.

    When nobody is responsible for maintaining a capability over time, the consequences of its weakness do not remain contained. They appear elsewhere, and another capability begins compensating for the gap. The vacuum does not stay empty.

    From Influence to Accidental Ownership

    This distinction becomes useful because capability boundaries are rarely clean. Testing depends on requirements. Architecture depends on product direction. Security depends on engineering practices. Operations depends on architecture. Product Management depends on business analysis.

    Capability Management should therefore not attempt to isolate capabilities from one another. It should make the dependencies visible and preserve appropriate responsibility across them.

    One capability may legitimately influence another. It may identify weaknesses, provide evidence, recommend changes or participate in improvement. But influence should not silently become permanent stewardship simply because nobody else is holding the responsibility. Otherwise organizational capabilities can gradually absorb one another’s problems until their boundaries become difficult to understand.

    Improvement Becomes Part of a Longer Lifecycle

    Seen this way, capability improvement becomes one activity within a longer lifecycle. An assessment may reveal that the current capability is insufficient. Capability Management helps interpret what that means for the future. A transformation or improvement initiative may then introduce deliberate changes. Reassessment provides evidence about whether those changes actually improved the organizational ability.

    But once the intervention ends, Capability Management continues. It preserves the intent, observes the realized capability, incorporates learning and determines when further development is needed.

    The lifecycle therefore does not end with transformation. It continues through stewardship.

    Capability Ownership as a Realization

    This also changes how the idea of a Capability Owner should be understood. If Capability Management is the discipline, a Capability Owner is one possible organizational realization of it. In some organizations, one person may hold that responsibility explicitly. In others, it may be distributed across a leadership group, community or existing management role. Different capabilities may require different arrangements. The organizational design is secondary. What matters is that somebody, or some mechanism, maintains continuity of the capability.

    The responsibility is not necessarily to own every process, document, specialist or decision associated with the capability. It is to ensure that the organization continues to possess and deliberately develop the required ability. That is a much more durable responsibility.

    A Portfolio of Capabilities

    Once individual capabilities are viewed this way, another level becomes visible. Organizations possess many capabilities. Testing, Requirements Engineering, Architecture, Security, Business Analysis, Product Management, Data Management and countless others coexist and depend on one another.

    Each may be strong or weak. Each may evolve at a different pace. Each may compensate for gaps elsewhere. Each consumes investment and management attention. This begins to look less like a collection of isolated improvement disciplines and more like a portfolio of organizational capabilities.

    Capability Management concerns the continuity of an individual capability while Capability Portfolio Management would ask a different question: whether the organization possesses the right capabilities, in the right relationships, for what it is trying to accomplish.

    That is a broader question and deserves treatment of its own. But it reinforces the same point: capabilities are not merely things organizations improve when they become problematic. They are organizational assets that require ongoing stewardship.

    Reflection

    Organizations are often quite good at launching capability improvement initiatives. They are less consistent at preserving responsibility once the initiative has done its work. That may be why the same capability problems return in cycles. The intervention succeeds, but continuity remains weak.

    If that is true, then the missing discipline is not another improvement method. It is the management of organizational ability between improvements.

    Assessment tells us where the capability is. Transformation helps change it. Capability Management ensures that somebody continues asking where it needs to go next.

    Improvement projects change organizations.

    Capability Management sustains organizational ability.

  • Practice note: When Local Heroes Hide Missing Capability

    Organizations often depend on people who make things work despite the organization around them. They build professional communities. They connect people who would otherwise remain disconnected. They maintain methods and knowledge. They support Sales. They create training. They solve recurring problems. They preserve relationships between organizational units. They know whom to call when something needs to happen.

    These people are often extremely valuable. They can also make organizational weakness remarkably difficult to see.

    Success Does Not Necessarily Demonstrate Capability

    Capability concerns organizational ability. If an organization repeatedly produces a valued effect, it is tempting to conclude that the organization possesses the capability required to produce it.

    That conclusion may be premature. Sometimes the effect exists because one or a few individuals continuously compensate for weaknesses in the surrounding organization.

    The organization may lack clear ownership, coherent processes, sufficient knowledge, effective collaboration, appropriate decision rights, or sustainable mechanisms for learning. Yet an experienced and motivated individual bridges those gaps personally.

    The desired effect still appears. From the outside, the capability seems to work. But the organization may actually be observing exceptional individual performance rather than robust organizational capability.

    The Local Hero Paradox

    This creates a paradox. The better the local hero becomes at compensating for organizational weakness, the less visible that weakness becomes.

    Management sees that:

    • problems are being solved;
    • people are being connected;
    • knowledge is being maintained;
    • customers are being supported;
    • initiatives continue to move;
    • important outcomes are still being produced.

    The immediate organizational need therefore appears to have been satisfied.

    Why invest further?

    The answer only becomes obvious when the local hero becomes unavailable, overloaded, changes role, or leaves. Suddenly relationships disappear. Knowledge becomes difficult to find. Activities that seemed routine stop happening. Decisions take longer. People discover that nobody knows why particular arrangements exist. The organization then learns that an apparently functioning capability depended upon a person rather than upon a sustainable organizational ability.

    The Hero Can Preserve the Problem

    There is a more uncomfortable consequence. Local heroes do not merely compensate for missing capability. Their success can help preserve the conditions that made their heroics necessary. Investment is easiest to justify when organizational weakness produces visible consequences. The local hero prevents those consequences. The stronger their performance, the weaker the apparent case for investing in the underlying capability.

    A self-reinforcing pattern can emerge:

    Weak organizational capability

    → individual compensates

    → desired effects continue

    → weakness becomes less visible

    → investment appears unnecessary

    → dependency on the individual continues

    The organization receives the benefit without developing a sustainable ability to produce it. Success therefore becomes misleading evidence.

    Look Beyond the Effect

    Capability Assessment should therefore not ask only whether the intended effect currently occurs. It should also investigate how reliably the organization can continue producing that effect under relevant conditions.

    Important questions include:

    • What happens when key individuals are unavailable?
    • Is important understanding distributed or concentrated?
    • Can responsibilities be transferred without substantial loss?
    • Are important relationships institutional or primarily personal?
    • Can another competent person reconstruct why current arrangements exist?
    • Does the organization have mechanisms for maintaining and evolving the capability?
    • Would the capability survive reorganization, growth, turnover, or changing priorities?

    A capability that works only while particular individuals remain personally engaged may still produce considerable value.

    But its resilience is limited. That limitation should be visible in the assessment.

    Local Heroes Are Not the Problem

    The lesson should not be to suppress exceptional individual initiative. Organizations need people who see gaps, connect knowledge, challenge existing arrangements, and make things happen.

    The problem arises when the organization treats their extraordinary contribution as evidence that no further organizational development is required.

    A local hero may actually provide some of the strongest evidence about where capability needs strengthening.

    Their work reveals which relationships matter, which knowledge must be preserved, which decisions require coordination, and which organizational mechanisms are currently missing.

    Instead of asking:

    How do we make this person less important?

    a better question is:

    What organizational ability is this person currently making possible, and how can we make that ability sustainable?

    The objective is not to eliminate individual excellence.

    It is to ensure that excellence strengthens organizational capability rather than permanently substitutes for it.

    From Heroism to Capability

    Capability Development can use successful local practice as evidence. The organization can examine what the individual actually does:

    • which functions they perform;
    • which relationships they maintain;
    • which knowledge they preserve;
    • which decisions they enable;
    • which gaps they bridge;
    • which effects their contribution produces.

    The objective is not necessarily to formalize every activity. That would risk replacing useful human judgment with bureaucracy.

    Instead, the organization should understand the functions behind the successful behavior and determine which of those functions need to become sustainable organizational abilities. Some may remain dependent on professional expertise.

    Some may be distributed across several people. Some may require explicit stewardship. Some may benefit from better information, authority, processes, tools, communities, or governance. The realization can vary. What matters is that the organizational ability no longer disappears when one person does.

    A Continuity Problem

    Local-hero dependency is therefore also a Continuity problem. Understanding may exist, but remain concentrated. Relationships may work, but remain personal. Decisions may be effective, but their rationale may remain undocumented.Learning may occur, but remain inside one person’s experience. The organization possesses knowledge without necessarily possessing continuity of that knowledge.

    A useful test is therefore:

    Can the next generation continue the reasoning rather than merely inherit the artifacts?

    If not, the apparent capability contains significant Understanding Debt.

    The Management Trap

    There is a final irony. Organizations sometimes respond to highly capable individuals by giving them even more responsibility. Because the person repeatedly solves difficult problems, additional problems are routed toward them. The local hero becomes increasingly valuable and increasingly overloaded at the same time.

    This further concentrates knowledge and relationships. Eventually the organization has created a critical dependency around precisely the person who appeared to demonstrate that the capability was strong.

    The appropriate response to successful local heroism may therefore be investment rather than additional dependency.

    The question for management is not merely:

    How much more can this person deliver?

    It is:

    What does this person’s success teach us about the capability we should build around them?

    Closing Thought

    Local heroes can create extraordinary organizational value. But the effects they produce should not automatically be mistaken for evidence of sustainable organizational capability.

    Sometimes the opposite is true. Their success reveals how much organizational ability currently depends upon individual effort. The strongest organizations should therefore learn from their local heroes rather than simply depend upon them.

    Because a local hero can compensate for missing capability. And, paradoxically, the better they become at doing so, the longer that missing capability may remain hidden.

  • Practice Note: Building on Established Reference Models

    The term reference model is already well established within systems engineering, enterprise architecture and other engineering disciplines. It generally refers to an abstract representation of a domain that defines concepts, their relationships and a common frame of reference while remaining independent of a particular implementation.

    For readers unfamiliar with the concept, the Wikipedia article provides a concise overview of the commonly accepted definition:

    https://en.wikipedia.org/wiki/Reference_model

    This Practice Note does not attempt to redefine what a reference model is. Instead, it establishes the perspective adopted throughout this body of knowledge.

    Many reference models successfully describe what exists. They identify concepts, components and relationships. They provide a shared vocabulary and a structural understanding of a domain.

    In practice, however, organizations face a different challenge. Over time, the structure often survives while the understanding behind it gradually disappears. Documentation remains available, yet the reasoning that shaped it becomes difficult to reconstruct. New team members can read the model without fully understanding why it exists, what assumptions influenced it or under which circumstances it should evolve.

    From the perspective of these Practice Notes, a reference model is therefore valuable not only because it represents a domain, but because it enables the continuity of understanding of that domain.

    Three questions become particularly important.

    • Does the model preserve the intent behind its concepts and relationships?
    • Can its meaning be reconstructed when context has been lost?
    • Can the model evolve without losing the reasoning that originally made it useful?

    These questions extend beyond documentation. They concern the preservation of organizational knowledge and the ability to make informed decisions as organizations, products and capabilities evolve.

    The Practice Notes therefore build upon the established concept of a reference model rather than replacing it. Their contribution is to explore how reference models can support continuity of understanding, deliberate development and defensible assessment throughout the lifecycle of organizations, capabilities and products.

    Future reference documents in this body of knowledge adopt this perspective as their foundation.

  • Practice Note: The Danger of Reference Objects

    Is, let’s say, a Seiko Presage Sharp Edged suitable as a dress watch? One way to answer that question is to find an undisputed dress watch—perhaps a Rolex—and compare the Seiko against it. The Rolex may be thinner. More restrained. Its dial may be simpler, its case less assertive, its overall appearance more traditional. Conclusion: the Seiko is not a dress watch.

    But something has gone wrong. We did not assess whether the Seiko was suitable as a dress watch. We assessed whether it resembled the Rolex.

    The Rolex became a reference object.

    That matters because a Rolex is not the definition of a dress watch. It is one implementation of a dress watch. Its proportions, materials, dial layout and styling are particular design choices through which it realizes certain characteristics. By comparing the Seiko directly with the Rolex, we quietly turn those choices into assessment criteria.

    We are no longer asking:

    Does the Seiko possess the characteristics of a dress watch?

    We are asking:

    Does the Seiko implement those characteristics in the same way as the Rolex?

    A better approach is to define the typical characteristics of a dress watch.

    Perhaps:

    • restrained proportions;
    • an elegant case;
    • a leather strap;
    • limited complications;
    • an appearance that complements rather than dominates formal clothing;
    • and the ability to fit comfortably beneath a shirt cuff.

    Now we have something closer to a reference model.

    Against those characteristics, the answer becomes more interesting. The Seiko may not be the purest or most traditional dress watch. Its case and dial may be more expressive than the classical ideal. But it satisfies enough of the relevant characteristics that it can reasonably function as one. So yes: perhaps it is at least kind of a dress watch.

    The assessment changed because we stopped comparing to a reference object and started measuring against a reference model.

    Now turn to software. What are the requirements for the new system?

    “We don’t really have proper requirements. We don’t have the time, money or capability to define them. But the new system should look and work like the old system.”

    So the old system becomes the reference object. The same old system that must be replaced.

    Now, with considerably more money at stake, we make exactly the same mistake again. The old system is one implementation of the required business behavior. But instead of identifying that behavior, we copy the implementation. Its screens become requirements. Its workflows become requirements. Its terminology and data structures become requirements. Even its limitations, workarounds and historical accidents risk becoming requirements.

    Apparently, the old system is not good enough to keep—but its implementation is good enough to copy. The irony is difficult to miss.

    A transformation project begins because the current system is no longer suitable, then uses that same system as the model for the future. We call it transformation, but make resemblance to the past the acceptance criterion.

    Of course, the old system can still be useful. It may contain valuable examples, business terminology, accumulated knowledge and behaviors that must not be lost. But it should help us discover the reference model, not replace it.

    The real questions remain:

    • What must the new system enable?
    • Which outcomes must it support?
    • Which behaviors must remain possible?
    • Which characteristics must it possess?
    • Which constraints still matter?
    • Which parts of the old system were merely consequences of one particular implementation?

    A reference model describes the required behavior and characteristics.

    A reference object shows one possible implementation of them.

    When the reference object replaces the reference model, one implementation quietly becomes the requirement. And QA verifies the implementation rather than the required behavior.