Marcel van Oosterwijk

Category: Assessment

Quality, testing, assessment, evidence, judgment, reference models, and the development and evaluation of quality and capability.

  • Practice Note: A Reference Model Is Not a Prescription

    Two teams use exactly the same testing practices.

    One receives a positive capability assessment. The other does not.

    Is the assessment inconsistent?

    Not necessarily.

    One team develops a relatively simple internal application. The other develops a safety-critical subsystem.

    The practices may be identical, but the expectations should not be.

    This raises an interesting question.

    How can a generic capability reference model remain useful when adequate capability depends on context?

    From Generic to Specific

    A capability reference model describes what an organization should be capable of doing, without necessarily prescribing how that capability must be realized.

    Consider Quality Assurance.

    A generic reference model might describe capabilities related to quality strategy, verification, validation, test design, defect management and quality evaluation.

    These concepts can apply across many organizations.

    But their realization depends on context.

    An organization developing medical devices faces different risks, obligations and quality expectations from one developing internal business applications.

    Even within the same organization, different products and subsystems may require different approaches.

    A safety-critical component might demand extensive traceability and formal verification. A user interface might place greater emphasis on usability and exploratory testing. A data integration component might require particular attention to data integrity and failure handling.

    The underlying capability concepts remain recognizable.

    What constitutes adequate capability changes.

    Contextualizing the Reference Model

    This suggests a progression:

    Generic capability → Organizational context → Product or subsystem context

    At the generic level, the reference model provides a common vocabulary and a basis for reasoning about capability.

    At the organizational level, we consider the organization’s objectives, responsibilities, operating environment, risks and constraints.

    At the product or subsystem level, we consider the specific characteristics and quality expectations of the subject being assessed.

    Each level adds context.

    But that doesn’t necessarily mean creating an entirely new capability model at every level.

    The reference model can remain conceptually stable while the expectations, evidence and appropriate realization become increasingly specific.

    For example, risk-based testing may be a relevant capability in both a business application and a safety-critical subsystem.

    But what constitutes sufficient risk analysis, test coverage and supporting evidence may differ substantially.

    The capability is recognizable in both contexts.

    Its adequate realization is not necessarily identical.

    Assessing Capability or Checking Compliance?

    This distinction matters because reference models can easily become checklists.

    Does the organization have a test strategy?

    Does it use automated testing?

    Does it perform reviews?

    Does it maintain traceability?

    These questions may be useful, but the existence of a practice doesn’t necessarily establish that the capability is adequate.

    An organization can perform all the expected activities while remaining unable to achieve the intended outcomes.

    Conversely, an organization may realize a capability effectively through practices that differ from those anticipated when the reference model was created.

    If we assess compliance with prescribed practices, we risk confusing the means with the capability.

    Capability assessment should evaluate fitness for purpose, not conformity to a universal implementation.

    This also means that identical practices need not produce identical assessment results.

    And different practices may legitimately support the same capability rating.

    The question is whether the subject can achieve what is required of it in its particular context.

    The Risk of Contextualizing Too Much

    There is a complication.

    If every organization, product or subsystem receives its own completely customized model, we may lose the consistency that made the reference model useful in the first place.

    Assessments become difficult to compare. Common capability concepts disappear behind local terminology. Improvement approaches become unnecessarily fragmented.

    But contextualizing too little creates the opposite problem.

    We evaluate subjects against generic expectations that may be irrelevant, insufficient or inappropriate.

    The challenge is therefore to distinguish between what should remain stable and what should vary.

    The reference model provides the conceptual structure.

    Context determines which capabilities matter, what adequate performance means and what evidence is needed to assess them.

    Sometimes the context may even reveal capabilities or dimensions missing from the generic model.

    That should be an opportunity to improve the reference model, rather than a reason to abandon it.

    A Reference Model Is Not a Prescription

    Perhaps the value of a capability reference model lies precisely in this distinction.

    It allows us to reason consistently about different subjects without assuming that they should all operate identically.

    It provides a shared language for understanding, assessing and developing capabilities.

    But it does not eliminate the need for professional judgment.

    We still need to understand the subject, its purpose, its environment and the outcomes it must achieve.

    Only then can we determine what adequate capability looks like.

    A reference model tells us what to reason about. Context tells us what good looks like.

  • Practice Note: Congratulations. You’re Now a Quality Manager.

    In a previous Practice Note, I suggested that AI agents might turn everyone into a team lead.

    You no longer need people reporting to you to have a team. You might have twenty AI agents performing work on your behalf.

    But there’s a problem.

    Having a team doesn’t mean you can trust its work.

    And when your team consists of AI agents, you may discover that you’ve acquired another responsibility.

    Congratulations. You’re now a Quality Manager.

    Delegating Work Is Easy. Trusting the Results Is Harder.

    Imagine you’re working with several AI agents.

    One investigates a problem. Another develops a solution. A third implements it. A fourth evaluates the result.

    Everything happens remarkably quickly.

    But how do you know whether the work is any good?

    The agents may produce convincing results. They may explain their reasoning, provide evidence and confidently report that their tasks are complete.

    They may also misunderstand the objective, overlook important constraints or make incorrect assumptions.

    You could review everything yourself.

    But if you have to repeat all the work to verify it, much of the benefit of delegation disappears.

    You need another approach.

    Welcome to Quality Management

    Before delegating work, you need to establish what you expect.

    What should be achieved? What constraints apply? What would constitute an acceptable result?

    You also need to decide how the work should be performed.

    Which activities can be delegated independently? Which require coordination? Where are the risks? What should be checked, and by whom?

    And once the work is complete, you need evidence that the results are acceptable.

    You might review deliverables, introduce automated checks, have agents evaluate each other’s work or require human approval for particularly important decisions.

    None of these mechanisms guarantees quality.

    But together, they can provide a more systematic basis for confidence.

    You’ve established expectations, organized work, managed risks and introduced mechanisms for evaluating results.

    Whether you intended to or not, you’re practicing Quality Management.

    Assessing Products and Capabilities

    There’s another distinction worth making.

    Suppose an agent produces an excellent result.

    Does that mean you can trust it to produce excellent results next time?

    Not necessarily.

    Evaluating an individual result tells you something about that result.

    It doesn’t necessarily tell you whether the agent can reliably produce similar results under different circumstances.

    For that, you need to understand the capability behind the result.

    Does the agent have access to the necessary information? Are its instructions adequate? Does it recognize situations it cannot handle? Are appropriate controls in place?

    You may discover that an agent produces good results only when given very specific instructions, or that it performs reliably for routine tasks but struggles with exceptions.

    This information helps you improve how work is delegated.

    And now you’re doing something beyond checking outputs.

    You’re assessing and developing the capability of your little AI organization.

    Quality Becomes Everyone’s Responsibility

    For decades, quality professionals have argued that quality cannot simply be delegated to a testing or quality assurance department.

    Quality depends on decisions and activities throughout the organization.

    Yet in practice, many professionals have been able to treat quality management as someone else’s responsibility.

    AI agents may change that.

    When you personally perform an activity, your expertise and experience provide some confidence in the result.

    When you delegate the activity to an agent, you need other ways to establish that confidence.

    You must become more explicit about expectations, evidence, risks and acceptance criteria.

    And as you delegate more work, you may need increasingly systematic mechanisms for managing its quality.

    Not because someone has introduced a new corporate quality initiative.

    But because you need to know whether you can trust the work being performed on your behalf.

    The Unexpected Quality Revolution

    We often discuss AI as a productivity revolution.

    People will produce more code, documents, analyses and decisions with less effort.

    Perhaps.

    But the more interesting development may be what happens to our responsibilities.

    As execution becomes easier to delegate, defining objectives, evaluating evidence and exercising judgment become increasingly important.

    And those are fundamental concerns of Quality Management.

    Perhaps AI will accomplish something that decades of quality professionals have struggled to achieve.

    It may make Quality Management everyone’s business.

    Not because we’ve finally convinced everyone of its importance.

    But because everyone suddenly needs it.

  • Practice Note: How Testing Became Quality Management Without Leaving Testing Behind

    Testing has changed considerably over the years.

    What once focused primarily on evaluating software products has expanded into test automation, quality assurance, quality engineering, process improvement, and increasingly organizational capability development.

    We have moved upstream, becoming involved in requirements, architecture, development practices, and organizational ways of working.

    Yet we still tend to describe much of this work as testing.

    Perhaps we have expanded the discipline without reconsidering what the discipline has become.

    Following Quality Upstream

    Testing provides information about products.

    When something doesn’t work as expected, testing helps us discover it, investigate it, and understand its consequences.

    But many problems discovered through testing originate elsewhere.

    Ambiguous requirements, architectural decisions, inadequate development practices, or insufficient organizational capabilities may all become visible through testing.

    Improving testing alone does not necessarily address these problems.

    So we started moving upstream.

    We became involved in requirements reviews, development practices, risk management, process improvement, and eventually organizational assessments.

    The underlying motivation remained the same: improving our ability to achieve quality.

    But the subject of our attention gradually changed.

    We were no longer concerned only with evaluating products. We were increasingly concerned with understanding and improving the organizational mechanisms that produce them.

    From Testing to Quality Management

    This development introduced a collection of overlapping concepts.

    Quality Assurance, Quality Control, Quality Engineering, Test Management, and Quality Management are frequently used in ways that make their responsibilities difficult to distinguish.

    Perhaps the problem is not primarily terminology.

    Perhaps we have been trying to accommodate an increasingly broad set of responsibilities within a discipline originally organized around testing.

    A useful distinction might be:

    • Testing investigates products and provides evidence about their behaviour and characteristics.
    • Quality Control evaluates whether products satisfy relevant quality expectations.
    • Quality Assurance provides confidence that the ways of working used to achieve quality are adequate.
    • Quality Engineering develops and applies engineering practices that help achieve quality.
    • Quality Management provides the organizational mechanisms for directing, evaluating, and improving the ability to achieve quality.

    These are not necessarily competing disciplines.

    They address different aspects of the same challenge.

    And testing remains an essential part of that challenge.

    Assessing More Than Products

    Organizations routinely evaluate their products.

    They test software, inspect deliverables, monitor performance, and investigate defects.

    But how systematically do they evaluate the organizational capabilities responsible for producing those products?

    When testing repeatedly reveals poor requirements, do we assess the organization’s requirements engineering capability?

    When defects originate in architectural decisions, do we examine the architecture capability?

    When problems recur across projects, do we investigate the organizational mechanisms that should enable learning and improvement?

    Often, we address the immediate problem without systematically evaluating the capability behind it.

    This suggests an interesting extension of Quality Management.

    Just as product assessment provides evidence about products, capability assessment could provide evidence about the organization’s ability to achieve intended outcomes.

    And just as product development responds to findings about products, capability development could respond to findings about organizational capabilities.

    Quality Management would then concern itself not only with whether quality is achieved, but also with whether the organization has the capabilities required to achieve it repeatedly.

    Without Leaving Testing Behind

    None of this makes testing less important.

    Testing remains one of our most valuable sources of evidence about whether products behave as intended.

    It also provides an important perspective on the consequences of decisions made elsewhere.

    But testing cannot compensate indefinitely for weaknesses throughout the organization.

    Nor should it be expected to.

    Moving upstream does not mean abandoning testing. It means recognizing that the information testing provides can lead us toward questions that testing alone cannot answer.

    And perhaps that is how testing gradually led us toward Quality Management.

    Not because testing became something else, but because understanding quality required us to look beyond testing.

    Quality is created throughout the organization. Quality Management makes that creation intentional, coordinated, and improvable.

  • Practice Note: Ready, Done, and the Risk Between Them

    It is Friday afternoon. The end of the Sprint is approaching. Several stories are not finished and the pressure rises. A few tests could perhaps be skipped. A defect may not be serious enough to hold the story back. Regression testing could be completed later.

    Nobody says, “Let’s accept more product risk so that we can avoid Sprint spillover.” Yet that may be exactly what is happening.

    This made me look differently at the Definition of Done. Why are its criteria there in the first place? If testing, review, security checks or other quality controls are part of the DoD, they should not merely be activities the team has agreed to perform. They should be there because they control risks that matter.

    Seen that way, there are two quite different problems with a Definition of Done.

    The first is that it may not have been defined particularly well. It can become a generic checklist of things teams conventionally do rather than a set of controls related to the actual risks of the product.

    The second is that even a meaningful DoD may become negotiable when delivery pressure rises.

    “Can we skip this test so we can finish the Sprint?” sounds like a delivery question. But if the test exists to control a risk, the real question is closer to:

    What risk was this test controlling, and are we prepared to accept the residual risk without it?

    That may be a perfectly legitimate decision. The point is that it should be recognized as a risk decision. This suggests that risk should be made more explicit in the work itself.

    A user story normally contains intent and acceptance criteria. It could also identify the significant quality risks associated with the story: what might prevent the intended outcome, how likely that is, and how serious the consequences would be.

    Acceptance criteria and risk then serve related but different purposes. Acceptance criteria help us judge whether we implemented what we intended. Risk-based evidence helps us judge whether the important undesirable outcomes have been sufficiently controlled.

    That leads to an interesting reinterpretation of Ready and Done.

    The Definition of Ready becomes a shared checkpoint asking whether we understand the work — including its significant risks — sufficiently to start responsibly.

    The Definition of Done becomes the corresponding checkpoint at the other end: have those significant risks been sufficiently mitigated, and do we have appropriate evidence to support that judgment?

    In simple terms:

    Ready: do we understand the important risks sufficiently to proceed?

    Done: do we have sufficient evidence that those risks have been addressed to proceed further?

    But then another problem appears: stories do not exist independently.

    Story A can be Ready. Story B can be Ready. Story C can be Ready. Yet combining them in the same increment may create risks that none of the stories contains individually. They may change the same component. One may invalidate assumptions made by another. Performance, security, resilience or operational complexity may emerge only when the changes interact. So readiness has an increment aspect too.

    It is not enough to ask whether the individual stories are Ready. We also need to ask whether we understand the significant risks created by combining them.

    The same applies to Done. Three stories can individually satisfy their DoD while the integrated increment still behaves unacceptably.

    A set of Done stories does not automatically make a Done increment.

    As work aggregates, risk can emerge at the new level. Assurance therefore has to aggregate with it.

    The same logic can continue toward release. A completed increment can still be unsuitable for deployment because release and operational risks only become meaningful at that level.

    What started as a Scrum mechanism therefore begins to look more like a broader assurance chain:

    Intent → Risk → Control → Evidence → Judgment

    At each level, we are asking what we are trying to achieve, what could prevent it, what controls those risks, what evidence we have, and whether the remaining risk is acceptable.

    And the chain should not end when the release reaches production. Production gives us new evidence. Some risks materialize despite our controls. Some turn out to have been underestimated. Others were never identified at all.

    Eventually, the production defect may come back to the development team. But the more important question is:

    Did the defect come back, or did the lesson come back?

    If an incident reveals that a risk was missed, misunderstood or inadequately controlled, that knowledge should change future stories, risk assessments, controls, and perhaps even what Ready and Done mean.

    The loop becomes something like:

    Ready → Develop → Done → Integrate → Release → Operate → Learn → Ready…

    The product continues, and so should our understanding of its quality.

    The principle that seems to emerge is simple:

    As work aggregates, assurance must aggregate with it.

    That turns Ready and Done from static checklists into checkpoints in a continuous quality-learning system.

  • Practice Note: Reference Model as the Antidote to Understanding Debt?

    This morning I woke up with a rather practical question.

    Since my laptop is getting older but is still considered rather capable, I wondered how suitable it is to run a language model locally on this machine. And I wondered what a local LLM could be used for. A specific use case came up quickly:

    Organizations accumulate enormous amounts of material over time: architecture descriptions, requirements, decision records, source repositories, project documentation, incident histories, assessments, reports, presentations, and wikis. Could a local LLM collect, analyze, and synthesize such material into a body of knowledge that could then be interrogated?

    We had already explored the idea that this might reduce the need to keep producing new documents simply to carry existing insights into another context. Instead of anticipating every future question and writing a document to answer it, perhaps we could maintain the knowledge and construct the answer when somebody actually asks.

    There is an attractive side effect to this. Documents start aging almost as soon as they are published. If the underlying body of knowledge changes instead, the same question asked six months later can produce an answer based on what is known six months later. The emphasis shifts from periodically republishing knowledge to maintaining something that can continuously be interrogated.

    Reconstructing a Product Reference Model

    I had already been thinking about this in relation to Product Reference Models. An existing product tends to leave a substantial trail of evidence behind it: architecture, requirements, code, decisions, incidents, operational information, and many other artifacts. Rather than manually recreating an understanding of the product from scratch, perhaps AI could work across those existing sources and help reconstruct a Product Reference Model.

    The interesting part would not simply be collecting links. A useful model would also need the relationships between those sources and, particularly, the rationale for their inclusion. Why does this architecture repository matter? What does this incident history tell us about the product? Why is an old decision record still relevant? Where do different sources contradict each other, and where are there white spots for which no useful evidence appears to exist?

    A list of artifacts gives us information. The relationships and rationale between them begin to give us understanding.

    Such a Product Reference Model would not necessarily need to become another physical document. It could remain connected to its underlying sources and be synthesized when required. If a production incident exposed a previously unknown dependency, or a new business need resulted in changes to architecture and code, the evidence available to the model would change with it. The next interrogation could therefore reflect a different understanding of the product without somebody first having to rewrite a reference document.

    From products to capabilities

    That thought made me look differently at another Reference Model I had been working with.

    In parallel, I had been developing a concept Quality Assurance Capability Reference Model. Unlike the rather distributed Product Reference Model I was imagining, this one is very tangible: a spreadsheet with multiple views of the capability, including dimensions and maturity characteristics, stakeholder perspectives, an evidence landscape, indicators, anti-patterns, and example practices. It is intended to support both capability assessment and capability development.

    If a Product Reference Model could become interrogable, why couldn’t this one?

    An assessor could ask what stronger capability looks like in a particular dimension, which stakeholder perspectives might be relevant, or what evidence could challenge an emerging interpretation. Someone working with capability development could interrogate the same model from a different perspective. The spreadsheet might remain useful for maintaining the model, but it would no longer have to be the primary way in which people consume it.

    There is an additional twist here. Assessment and development do not only use the Reference Model; they also generate evidence about the subject it describes. An intervention may succeed, fail, or work for entirely different reasons than expected. Repeated assessments may reveal a pattern that the existing model does not explain particularly well.

    So a Reference Model is not simply a stable body of knowledge from which answers are retrieved. It participates in a loop. The model helps us interpret evidence, while evidence can support, extend, or challenge the model. The Reference Model therefore becomes part of a learning loop.

    And that started to connect the local-LLM thought experiment with another idea we had already been approaching: stewardship.

    From management to stewardship

    Management disciplines such as product management and portfolio management necessarily spend considerable attention on the things associated with their subject: decisions, processes, plans, repositories, ownership structures, and other artifacts. But even when those things are managed well, the organization can still gradually lose its accumulated understanding of the subject itself.

    Stewardship suggests a different concern:

    Can the organization continue to understand a subject as people leave, evidence accumulates, and reality changes?

    That question exposes something slightly awkward. Where does persistent organizational understanding actually exist?

    Some of it exists in people’s heads, some in documents and systems, some in evidence, and a great deal of it exists in the relationships between those things. We had already suspected that Reference Models might play a foundational role in stewardship, but I think the reason for that is now becoming clearer.

    Reference Model = Understanding?

    We normally describe a Reference Model as a representation of our understanding.

    Perhaps that distinction is unnecessary.

    If a Reference Model maintains what matters about a subject, how things relate, why they matter, what evidence supports them, where uncertainty remains, and how all of that changes as we learn, then what separate thing are we calling “the understanding”?

    Perhaps the Subject Reference Model is the maintained organizational understanding of the subject.

    That does not mean every fact or artifact needs to be contained inside it. The Product Reference Model demonstrates why. Much of its detailed knowledge can remain in the sources to which it refers. Knowing that a source matters, understanding why it matters, and maintaining its relationships to other parts of the subject can itself be part of the model.

    A Capability Reference Model may encode much more of its domain knowledge explicitly. The physical forms are different because the subjects are different, but the fundamental function may be the same.

    A Product Reference Model is maintained product understanding. A Capability Reference Model is maintained capability understanding.

    More generally:

    Subject Reference Model = Subject Understanding

    And that connects to a problem we have encountered from an entirely different direction.

    Understanding debt

    We have been using the term understanding debt for something that documentation alone does not seem to solve. Organizations are extraordinarily good at accumulating information while gradually losing the context, relationships, and rationale that make the information understandable.

    The architecture document remains, but nobody remembers why the architecture was chosen. The old system remains, but nobody is entirely sure what depends on it. A practice continues long after its original purpose has disappeared from organizational memory. A decision can still be found, but the alternatives that were considered and the reasons they were rejected are gone.

    Nothing necessarily went undocumented. There may be more documentation than ever.

    What disappeared was the understanding that connected it.

    If the Subject Reference Model is maintained understanding, then understanding debt becomes easier to describe. It is the gap between what the organization needs to understand about a subject and what its maintained Subject Reference Model actually allows it to understand.

    Seen that way, Reference Models become rather more important than useful assessment frameworks, maturity models, or knowledge structures. They become a mechanism for allowing organizational understanding to persist beyond the people who happen to hold it at a particular moment.

    AI may make Reference Models more important, not less

    It would be easy to assume that increasingly capable AI makes Reference Models less important. If an AI can read everything, why bother maintaining a model?

    I suspect the opposite may be true.

    Giving AI access to everything gives it information: twenty years of documents, code, incidents, presentations, abandoned decisions, obsolete descriptions, and contradictory claims. Without some maintained conception of what matters and how it fits together, the AI has to reconstruct an understanding from that history every time we ask a question.

    A stewarded Reference Model gives that information meaning and continuity. AI, in turn, may make it practical to reconstruct such models from existing evidence, interrogate them rather than continually publishing new documents, and help keep them aligned with new evidence as the organization learns.

    The interesting role of AI may therefore not be producing more organizational knowledge at all. It may be helping us maintain the understanding underneath it.

    If Subject Reference Model = Subject Understanding, then stewardship of Reference Models is stewardship of organizational understanding.

    And that may make the Reference Model our best antidote yet to understanding debt.