Marcel van Oosterwijk

Category: Practice Notes

  • Practice Note: A Product Reference Model Is More Than Product Documentation

    An organization has a software product.

    It has requirements, architecture documentation, source code, test results, incident reports and records of important decisions.

    Everything appears to be documented.

    But ask a simple question:

    Why does this product exist?

    The answer may be surprisingly difficult to reconstruct.

    Ask which organizational capabilities depend on it, why its architecture looks the way it does, or what evidence demonstrates that it remains fit for purpose, and the problem becomes even more apparent.

    The information may exist.

    But that doesn’t necessarily mean the organization can use it to understand the product.

    Documentation Is Not Understanding

    Product documentation usually describes particular aspects of a product.

    Requirements describe expected behaviour. Architecture models describe structure. Test results provide evidence about certain characteristics. Decision records explain selected choices.

    Each artifact serves a purpose.

    But understanding a product requires more than understanding its individual artifacts.

    It requires understanding their relationships.

    Why was a particular architectural decision made? Which requirement or quality expectation motivated it? What alternatives were considered? What evidence supported the decision?

    And perhaps more importantly, does the original reasoning still hold?

    A collection of individually useful documents does not automatically provide a coherent understanding of the product.

    From Reference Model to Product Model

    This is where a product reference model becomes interesting.

    A product reference model describes the concepts and relationships we consider relevant when understanding products.

    For example, a product has a purpose. It supports organizational capabilities. It has expected behaviours and characteristics. It has an architecture and a realization. Decisions shape its development, and evidence supports claims about its qualities.

    These concepts are generic.

    They do not describe any particular product.

    A product model applies them to a specific product, connecting its actual purpose, characteristics, architecture, decisions and evidence.

    The distinction matters.

    The reference model provides a common structure for reasoning.

    The product model represents our understanding of a particular product.

    The documentation supplies much of the information and evidence on which that understanding depends.

    A reference model is therefore not another documentation template.

    It helps us understand how different pieces of information relate.

    Connecting the Product to Its Purpose

    Consider an organization replacing a legacy system.

    The replacement project may produce extensive documentation. The new system may satisfy its requirements and pass its acceptance tests.

    But why was the system replaced?

    Perhaps the old system prevented the organization from responding quickly to changing customer needs.

    In that case, the replacement is not merely a technical exercise.

    It is an intervention intended to improve an organizational capability.

    Understanding the product therefore requires connecting its technical characteristics to the organizational purpose it serves.

    Which capabilities depend on it?

    Which limitations of the previous system motivated the replacement?

    Which characteristics of the new product are expected to address those limitations?

    And what evidence would demonstrate that the intended improvement has actually occurred?

    These relationships may be more important than any individual document.

    Without them, we may know a great deal about how the product works while knowing surprisingly little about whether it achieves its purpose.

    Understanding What We Don’t Know

    There is another benefit to making these relationships explicit.

    We can begin to identify gaps in our understanding.

    Perhaps an architectural decision exists without a recorded rationale.

    Perhaps a quality requirement has no supporting evidence.

    Perhaps a product supports an important organizational capability, but nobody has established whether the product remains adequate for that purpose.

    Or perhaps a decision was justified by assumptions that are no longer valid.

    A product model doesn’t automatically resolve these uncertainties.

    But it can make them visible.

    And that changes how we think about completeness.

    The objective is not necessarily to produce more documentation.

    It is to establish whether we have sufficient understanding and evidence to manage the product responsibly.

    The Product Outlives the Project

    Products often continue long after the projects that created them have ended.

    The original architects leave. Development teams change. Technologies evolve. New requirements emerge.

    Yet the product must still be understood, maintained, assessed and developed.

    Traditional documentation can preserve valuable information, but it may not preserve the relationships needed to interpret that information.

    A product model could provide continuity.

    Not by anticipating every question future people might ask, but by maintaining a coherent representation of the product and its relationship to organizational intent, decisions and evidence.

    It could also make that understanding easier to interrogate.

    Why was this component introduced? Which capabilities would be affected if we removed it? What evidence supports the assumption that this design remains appropriate?

    The answers may require information from several sources.

    The reference model helps establish how those sources relate.

    More Than Documentation

    Perhaps the important distinction is not between having documentation and having a model.

    Both can be useful.

    The distinction is between preserving information about a product and maintaining an understanding of the product as a subject that must continue to be managed.

    A product reference model provides the conceptual structure.

    A product model applies that structure to a particular product.

    Documentation and other evidence support the resulting understanding.

    Documenting a product helps us preserve what we know about it.

    Modeling a product helps us understand how that knowledge relates — and what we still don’t know.

  • Practice Note: A Reference Model Is Not a Prescription

    Two teams use exactly the same testing practices.

    One receives a positive capability assessment. The other does not.

    Is the assessment inconsistent?

    Not necessarily.

    One team develops a relatively simple internal application. The other develops a safety-critical subsystem.

    The practices may be identical, but the expectations should not be.

    This raises an interesting question.

    How can a generic capability reference model remain useful when adequate capability depends on context?

    From Generic to Specific

    A capability reference model describes what an organization should be capable of doing, without necessarily prescribing how that capability must be realized.

    Consider Quality Assurance.

    A generic reference model might describe capabilities related to quality strategy, verification, validation, test design, defect management and quality evaluation.

    These concepts can apply across many organizations.

    But their realization depends on context.

    An organization developing medical devices faces different risks, obligations and quality expectations from one developing internal business applications.

    Even within the same organization, different products and subsystems may require different approaches.

    A safety-critical component might demand extensive traceability and formal verification. A user interface might place greater emphasis on usability and exploratory testing. A data integration component might require particular attention to data integrity and failure handling.

    The underlying capability concepts remain recognizable.

    What constitutes adequate capability changes.

    Contextualizing the Reference Model

    This suggests a progression:

    Generic capability → Organizational context → Product or subsystem context

    At the generic level, the reference model provides a common vocabulary and a basis for reasoning about capability.

    At the organizational level, we consider the organization’s objectives, responsibilities, operating environment, risks and constraints.

    At the product or subsystem level, we consider the specific characteristics and quality expectations of the subject being assessed.

    Each level adds context.

    But that doesn’t necessarily mean creating an entirely new capability model at every level.

    The reference model can remain conceptually stable while the expectations, evidence and appropriate realization become increasingly specific.

    For example, risk-based testing may be a relevant capability in both a business application and a safety-critical subsystem.

    But what constitutes sufficient risk analysis, test coverage and supporting evidence may differ substantially.

    The capability is recognizable in both contexts.

    Its adequate realization is not necessarily identical.

    Assessing Capability or Checking Compliance?

    This distinction matters because reference models can easily become checklists.

    Does the organization have a test strategy?

    Does it use automated testing?

    Does it perform reviews?

    Does it maintain traceability?

    These questions may be useful, but the existence of a practice doesn’t necessarily establish that the capability is adequate.

    An organization can perform all the expected activities while remaining unable to achieve the intended outcomes.

    Conversely, an organization may realize a capability effectively through practices that differ from those anticipated when the reference model was created.

    If we assess compliance with prescribed practices, we risk confusing the means with the capability.

    Capability assessment should evaluate fitness for purpose, not conformity to a universal implementation.

    This also means that identical practices need not produce identical assessment results.

    And different practices may legitimately support the same capability rating.

    The question is whether the subject can achieve what is required of it in its particular context.

    The Risk of Contextualizing Too Much

    There is a complication.

    If every organization, product or subsystem receives its own completely customized model, we may lose the consistency that made the reference model useful in the first place.

    Assessments become difficult to compare. Common capability concepts disappear behind local terminology. Improvement approaches become unnecessarily fragmented.

    But contextualizing too little creates the opposite problem.

    We evaluate subjects against generic expectations that may be irrelevant, insufficient or inappropriate.

    The challenge is therefore to distinguish between what should remain stable and what should vary.

    The reference model provides the conceptual structure.

    Context determines which capabilities matter, what adequate performance means and what evidence is needed to assess them.

    Sometimes the context may even reveal capabilities or dimensions missing from the generic model.

    That should be an opportunity to improve the reference model, rather than a reason to abandon it.

    A Reference Model Is Not a Prescription

    Perhaps the value of a capability reference model lies precisely in this distinction.

    It allows us to reason consistently about different subjects without assuming that they should all operate identically.

    It provides a shared language for understanding, assessing and developing capabilities.

    But it does not eliminate the need for professional judgment.

    We still need to understand the subject, its purpose, its environment and the outcomes it must achieve.

    Only then can we determine what adequate capability looks like.

    A reference model tells us what to reason about. Context tells us what good looks like.

  • Exploration: Is Moving Upstream a Professional Quality?

    A tester can execute tests perfectly against the wrong requirements.

    A developer can implement an elegant solution to the wrong problem.

    A project manager can deliver a project successfully without achieving the change it was intended to create.

    A consultant can implement an improvement initiative without improving anything.

    In each case, the professional may have performed the assigned work competently.

    But was it good professional work?

    Looking Beyond the Assignment

    Consider a tester who discovers that a requirement is ambiguous.

    The immediate response might be to request clarification so that testing can continue.

    But perhaps the ambiguity suggests a more fundamental problem with requirements management.

    And perhaps that problem originates in how stakeholders communicate, how decisions are made or how the organization understands customer needs.

    The tester can keep moving upstream.

    From testing to requirements. From requirements to product development. From product development to organizational decision-making.

    At each step, the question changes.

    Not simply How can I do my work better?

    But Why does this work need to be done, and what determines whether it creates value?

    This pattern isn’t unique to testing.

    A developer might question the architecture rather than simply implement another workaround.

    A project manager might question whether the planned deliverables will produce the intended outcomes.

    A consultant might question whether the requested intervention addresses the capability that actually needs improvement.

    In each case, moving upstream means looking beyond the immediate assignment toward the conditions and purposes that give it meaning.

    Is This Professional Development?

    We often associate professional development with increasing expertise.

    A tester becomes better at testing. A developer becomes better at programming. A consultant becomes more proficient in a methodology.

    But perhaps professional development also has another dimension.

    The ability to understand how one’s work contributes to something larger.

    That doesn’t necessarily mean moving into management or changing professions.

    A tester doesn’t need to become a business analyst to recognize weaknesses in requirements.

    A developer doesn’t need to become a product manager to question whether a feature solves the intended problem.

    And a consultant doesn’t need to become a strategist to recognize that implementing a framework is not the same as developing an organizational capability.

    Perhaps moving upstream isn’t about changing what we do.

    It’s about changing how we understand what we do.

    How Far Upstream?

    There is a problem with this reasoning.

    We can always ask another question.

    Why are the requirements poor? Why does the organization work this way? Why were these priorities chosen? Why does the business pursue these objectives?

    Eventually, questioning the purpose of the work can consume more attention than performing it.

    And not every upstream question is useful.

    Sometimes the problem is understood, the assignment is appropriate and the most valuable contribution is excellent execution.

    Specialization exists for good reasons. Deep expertise matters.

    A professional who continually challenges the assignment but cannot perform it well isn’t necessarily demonstrating superior professional quality.

    So perhaps moving upstream is not inherently better.

    The important quality might instead be knowing when the boundaries of the assignment prevent us from addressing the problem.

    And having the judgment to recognize when those boundaries should be questioned.

    When Implementation Becomes Easier

    AI adds another dimension to this question.

    As AI becomes increasingly capable of producing code, tests, documents, analyses and other artifacts, some implementation activities become easier.

    That doesn’t eliminate the need for expertise.

    But it may change where professional judgment contributes the most value.

    If producing a solution becomes easier, choosing the right problem becomes more important.

    AI can help implement what we ask for.

    It cannot make our intentions appropriate merely by implementing them successfully.

    Perhaps this makes the ability to move upstream more valuable.

    Not because everyone should become a strategist, but because the consequences of poorly understood intentions may become easier to realize.

    We may become increasingly effective at doing things without becoming equally effective at determining whether those things are worth doing.

    A Different Dimension of Professional Quality?

    Perhaps we should distinguish three things.

    Competence: Can I perform the assigned work well?

    Understanding: Do I know why the work matters?

    Judgment: Can I recognize when performing the assigned work well will not achieve its intended purpose?

    The first is essential.

    The second provides context.

    The third allows a professional to challenge the relationship between the assignment and its purpose.

    But even this distinction raises questions.

    How much upstream understanding should we expect from a professional?

    When does questioning an assignment demonstrate responsibility, and when does it become interference?

    Can organizations genuinely encourage professionals to challenge assumptions while continuing to evaluate them primarily on how efficiently they complete assigned tasks?

    And what happens when a professional recognizes an upstream problem but has no authority to influence it?

    Perhaps moving upstream is not simply an individual quality. It may also depend on whether the organization is capable of using the understanding that professionals develop.

    Perhaps the Question Is Different

    Maybe professional quality isn’t about moving upstream as far as possible.

    Nor is it about constantly questioning everything.

    It may be about retaining the ability to connect our work to its purpose, and recognizing when that connection is missing.

    A good professional knows how to perform the work.

    A better professional may also recognize when performing that work well is insufficient.

    But that leaves an uncomfortable question.

    If we recognize that the assignment will not achieve its intended purpose, is it still professional to complete it exactly as requested?

  • Practice Note: A Portfolio as a Vocabulary for Expression

    A portfolio is often justified through coverage. Different constituents provide different capabilities, allowing the portfolio as a whole to address more situations than any individual constituent could.

    But a portfolio can provide another kind of value: expression.

    Consider a small collection of watches.

    An industrial Victorinox I.N.O.X., an elegant ice-blue Presage, a green Alpinist and a solar panda chronograph all perform essentially the same basic function. Their practical use cases also overlap considerably.

    Yet choosing between them can still be meaningful.

    The I.N.O.X. might express resilience and confidence. The Presage refinement without complete conformity. The Alpinist curiosity and adventure. The panda technical practicality and precision.

    The question in the morning is therefore not necessarily:

    Which watch do I need today?

    It might be:

    Which aspect of myself do I want to express today?

    A Portfolio as a Vocabulary

    Words are not valuable because one ranks above another. They are valuable because different words allow different things to be expressed.

    Even words with similar meanings can carry different nuances.

    Replacing one word with a supposedly superior word makes little sense if the two communicate different things.

    The same can be true of portfolio constituents.

    Two constituents may have different specifications while communicating essentially the same thing. Their contribution to the expressive range of the portfolio may therefore be small.

    Conversely, two constituents may overlap substantially in practical capability while remaining valuable together because they enable different expressions.

    This gives us another question for evaluating portfolio composition:

    What can this constituent express that the existing portfolio cannot express adequately?

    Difference alone is not enough. A collection containing many minor variations may contain many objects while providing little additional expressive range.

    The differences must be meaningful in relation to the intent of the portfolio.

    Beyond Functional Coverage

    Functional coverage remains important, but it need not be the complete explanation for portfolio composition.

    Some portfolios also provide a vocabulary of meaningful differences.

    Their constituents are valuable not because they rank above one another, but because they allow different things to be expressed.

    The portfolio provides the vocabulary.

    The choice provides the expression.

  • Practice Note: The Project Is Not the Subject of the Change

    Projects have a peculiar ability to become the center of attention.

    We establish plans, milestones, budgets, governance structures and reporting mechanisms. We monitor progress, manage risks and eventually determine whether the project succeeded.

    Yet the organization rarely wanted a project in the first place.

    It wanted something to change.

    A testing improvement project exists to improve testing capability. A system replacement project exists to change a system or product. A transformation programme exists to develop organizational capabilities.

    The project exists in service of that change.

    But this raises an interesting distinction.

    The Project Is a Subject Too

    A project can itself be understood as a subject.

    It has boundaries, responsibilities, capabilities and characteristics that can be assessed and improved.

    There is nothing wrong with managing a project as a subject.

    The problem arises when we confuse it with the subject it exists to change.

    Consider a system replacement project.

    The project is one subject: a temporary organization coordinating change.

    The system being replaced is another. The organizational capability depending on that system may be a third.

    Each can be understood, developed and assessed.

    But evidence about one does not necessarily tell us much about the others.

    Two Different Claims of Success

    Suppose a project finishes on schedule, within budget and with all agreed deliverables accepted.

    That is meaningful evidence about the project.

    But has the intended change occurred?

    Does the new system deliver the expected outcomes? Has the organizational capability improved? Have the risks that motivated the project been reduced?

    Those are different questions requiring different evidence.

    The difficulty is that project performance is often easier to observe than the effects of the change.

    Milestones can be counted. Deliverables can be accepted. Activities can be marked complete.

    Capability development and organizational outcomes are harder to assess.

    So the visible subject can gradually substitute for the important one.

    A training programme is completed, so competence is assumed to have improved.

    A process is implemented, so behaviour is assumed to have changed.

    A system goes live, so the transformation is considered successful.

    But completing an intervention is not evidence that it achieved its intended effect.

    The Project as an Intervention

    Seen from the perspective of the subject being changed, the project is a temporary mechanism for coordinating interventions.

    It may introduce technologies, practices, roles, structures or governance arrangements intended to move another subject toward a desired state.

    The project therefore contains an implicit hypothesis:

    If we perform these interventions, we expect the subject to change in these ways.

    The project plan describes what we currently believe is necessary to produce that change.

    But what happens when an intervention is completed and the expected effect does not appear?

    Simply completing the remaining activities more efficiently may not help.

    The evidence may be telling us that our understanding of the subject, or of how to change it, was incomplete.

    The ability to adjust the interventions then becomes more important than adherence to the original plan.

    The Temporary and the Continuing Subject

    There is another complication.

    Projects are deliberately temporary.

    We create them with the intention of eventually dismantling them.

    Project teams dissolve. Governance structures disappear. Budgets close. Responsibilities are transferred.

    But the subject being changed may need to continue for decades.

    During the project, considerable understanding accumulates within the temporary organization.

    People learn why decisions were made, which alternatives were rejected, what assumptions proved incorrect and where uncertainty remains.

    What happens to that understanding when the project disappears?

    Traditional handover often concentrates on transferring deliverables, documentation and operational responsibilities.

    But continuity requires more than possession of artifacts.

    Can those who inherit the subject understand why it is the way it is?

    Do they know what outcomes were intended, which assumptions shaped the solution and what remains uncertain?

    Can they continue assessing and developing the subject without depending on the people who participated in the project?

    A project may successfully deliver a new realization while leaving the organization poorly equipped to understand or evolve it.

    The project is supposed to disappear. The understanding required to continue the subject should not disappear with it.

    What Does Project Completion Mean?

    Perhaps project completion deserves a broader interpretation.

    We need evidence that the project has been managed effectively.

    We need different evidence that the intended change has occurred.

    And we need confidence that the organization can continue managing and developing the changed subject after the project has ended.

    The project doesn’t need to preserve itself.

    Its temporary structures can disappear.

    But the understanding, responsibility and ability to learn must continue somewhere.

    Because the project is not the subject of the change.

    It is the temporary subject responsible for helping another subject become something different — and remain capable of evolving afterward.