Marcel van Oosterwijk

Author: Marcel

  • Practice Note: Bodies of Knowledge and Context Layers

    Imagine giving AI access to everything an organization knows.

    Every document, presentation, architectural decision, assessment report, project deliverable and incident record.

    Would the AI understand the organization?

    Not necessarily.

    It might retrieve relevant information, summarize documents and identify recurring themes.

    But would it understand what the information refers to, how different subjects relate, and which evidence supports which conclusions?

    Perhaps making organizational knowledge interrogable requires more than access to information.

    Perhaps it also requires context.

    A Body of Knowledge Is Not a Model

    Organizations accumulate enormous Bodies of Knowledge.

    They contain observations, decisions, experiences, explanations and evidence gathered over time.

    But much of this information was created for particular purposes, by different people, using different terminology.

    A project report describes a problem. An assessment identifies a capability weakness. An incident record describes a failure. An architectural decision explains a technical choice.

    These artifacts may concern related aspects of the same organizational reality.

    Yet their relationships are rarely explicit.

    An AI assistant might retrieve all four documents without necessarily understanding how they relate.

    The information exists.

    The structure needed to interpret it may not.

    Could Reference Models Provide the Context?

    A reference model provides a conceptual structure for understanding a particular kind of subject.

    A capability reference model describes capabilities and their relationships.

    A product reference model describes concepts such as purpose, characteristics, architecture, decisions and evidence.

    These models are generic. They do not describe any particular organization or product.

    But they can be contextualized.

    A generic capability model can be related to an organization’s actual capabilities, responsibilities and operating environment.

    A product reference model can be applied to a particular product, connecting its purpose, realization, characteristics and supporting evidence.

    This suggests an interesting possibility.

    Could contextualized reference models serve as context layers between an organization’s Body of Knowledge and the questions people want to ask?

    The Body of Knowledge would provide the information and evidence.

    The context layers would help establish what that information concerns and how it relates.

    AI could use both to investigate questions that were never anticipated when the information was created.

    Asking Better Questions

    Consider a consultant asking:

    Why are we experiencing recurring defects in Product X?

    Without an explicit context model, AI might retrieve incident reports, defect statistics and retrospective findings.

    That could already be useful.

    But suppose it also had access to contextualized models describing Product X, its expected characteristics, the organizational capabilities involved in developing it and the relationships between them.

    The investigation could become more structured.

    Which product characteristics are affected?

    Which capabilities contribute to producing those characteristics?

    What evidence exists about those capabilities?

    Have previous interventions attempted to address the problem?

    What happened afterward?

    The context layer would not provide the answer.

    Nor would it establish that a particular capability caused the defects.

    But it could help identify relevant relationships, locate evidence and expose gaps in our understanding.

    The model would help us determine what to investigate. The evidence would help us determine what we can reasonably conclude.

    But Who Says the Model Is Correct?

    There is a complication.

    A contextualized reference model is itself a representation of organizational reality.

    And representations can be incomplete, outdated or wrong.

    Perhaps the model assigns responsibility to a team that no longer performs the work.

    Perhaps it describes a dependency that has disappeared.

    Perhaps it assumes that a capability operates in a particular way, while the available evidence suggests something different.

    If AI treats the context layer as unquestionable truth, it may produce convincing answers based on incorrect assumptions.

    That would make the organization easier to interrogate without necessarily making its answers more reliable.

    So perhaps the relationship must work in both directions.

    The model helps interpret evidence.

    But evidence must also be capable of challenging the model.

    When observations contradict the contextualized understanding, that contradiction should become visible.

    And when the evidence warrants it, the model should change.

    An Interrogable Organization

    This raises another possibility.

    Perhaps an interrogable organization should not merely allow people to ask questions about its accumulated knowledge.

    It should also allow them to question its understanding of itself.

    Why do we believe this capability depends on that one?

    What evidence supports this assessment?

    When was this relationship last validated?

    Which assumptions are no longer supported?

    What has changed since this model was established?

    AI might even help identify discrepancies between the models and the evidence accumulated through everyday work.

    But identifying a discrepancy is not the same as resolving it.

    Someone still needs to evaluate the evidence, interpret its significance and decide whether the model should change.

    The context layer would therefore need to evolve alongside the organization.

    Not simply as documentation maintained periodically, but as a representation that can be challenged and refined through continued observation.

    How Much Context Must We Model?

    There is still an open question.

    How much organizational context needs to be explicitly represented?

    Modern AI can already infer relationships from documents and other information.

    Perhaps some contextual understanding can be reconstructed when a question is asked.

    But relying entirely on inference introduces uncertainty.

    Different questions might produce different interpretations of the same organization. Important relationships may be overlooked. Assumptions may remain invisible.

    Explicit models could provide greater consistency and make assumptions easier to examine.

    Yet maintaining comprehensive models of an organization could become an enormous undertaking.

    Perhaps the answer lies somewhere between the two.

    Reference models provide relatively stable conceptual structures.

    Contextualized models represent selected aspects of organizational reality.

    AI helps connect those models to available evidence, infer missing relationships and identify inconsistencies.

    Humans evaluate what should be accepted, challenged or changed.

    The models need not contain everything.

    But they may provide enough structure to make the organization’s knowledge more useful.

    From Information to Understanding

    Perhaps the real opportunity is not simply to create better search facilities.

    It is to connect three things that organizations often manage separately:

    Their accumulated knowledge.

    Their models of how the organization works.

    And the evidence through which those models can be evaluated.

    An interrogable Body of Knowledge could make information accessible.

    Context layers could make relationships understandable.

    And continued interrogation could help expose where the organization’s understanding no longer agrees with reality.

    That suggests a potentially different role for reference models.

    Not merely as instruments for assessment or standardization, but as part of the infrastructure through which an organization understands itself.

    Perhaps an interrogable organization needs more than access to what it knows.

    It also needs a way to question what it believes it knows.

  • Practice Note: A Product Reference Model Is More Than Product Documentation

    An organization has a software product.

    It has requirements, architecture documentation, source code, test results, incident reports and records of important decisions.

    Everything appears to be documented.

    But ask a simple question:

    Why does this product exist?

    The answer may be surprisingly difficult to reconstruct.

    Ask which organizational capabilities depend on it, why its architecture looks the way it does, or what evidence demonstrates that it remains fit for purpose, and the problem becomes even more apparent.

    The information may exist.

    But that doesn’t necessarily mean the organization can use it to understand the product.

    Documentation Is Not Understanding

    Product documentation usually describes particular aspects of a product.

    Requirements describe expected behaviour. Architecture models describe structure. Test results provide evidence about certain characteristics. Decision records explain selected choices.

    Each artifact serves a purpose.

    But understanding a product requires more than understanding its individual artifacts.

    It requires understanding their relationships.

    Why was a particular architectural decision made? Which requirement or quality expectation motivated it? What alternatives were considered? What evidence supported the decision?

    And perhaps more importantly, does the original reasoning still hold?

    A collection of individually useful documents does not automatically provide a coherent understanding of the product.

    From Reference Model to Product Model

    This is where a product reference model becomes interesting.

    A product reference model describes the concepts and relationships we consider relevant when understanding products.

    For example, a product has a purpose. It supports organizational capabilities. It has expected behaviours and characteristics. It has an architecture and a realization. Decisions shape its development, and evidence supports claims about its qualities.

    These concepts are generic.

    They do not describe any particular product.

    A product model applies them to a specific product, connecting its actual purpose, characteristics, architecture, decisions and evidence.

    The distinction matters.

    The reference model provides a common structure for reasoning.

    The product model represents our understanding of a particular product.

    The documentation supplies much of the information and evidence on which that understanding depends.

    A reference model is therefore not another documentation template.

    It helps us understand how different pieces of information relate.

    Connecting the Product to Its Purpose

    Consider an organization replacing a legacy system.

    The replacement project may produce extensive documentation. The new system may satisfy its requirements and pass its acceptance tests.

    But why was the system replaced?

    Perhaps the old system prevented the organization from responding quickly to changing customer needs.

    In that case, the replacement is not merely a technical exercise.

    It is an intervention intended to improve an organizational capability.

    Understanding the product therefore requires connecting its technical characteristics to the organizational purpose it serves.

    Which capabilities depend on it?

    Which limitations of the previous system motivated the replacement?

    Which characteristics of the new product are expected to address those limitations?

    And what evidence would demonstrate that the intended improvement has actually occurred?

    These relationships may be more important than any individual document.

    Without them, we may know a great deal about how the product works while knowing surprisingly little about whether it achieves its purpose.

    Understanding What We Don’t Know

    There is another benefit to making these relationships explicit.

    We can begin to identify gaps in our understanding.

    Perhaps an architectural decision exists without a recorded rationale.

    Perhaps a quality requirement has no supporting evidence.

    Perhaps a product supports an important organizational capability, but nobody has established whether the product remains adequate for that purpose.

    Or perhaps a decision was justified by assumptions that are no longer valid.

    A product model doesn’t automatically resolve these uncertainties.

    But it can make them visible.

    And that changes how we think about completeness.

    The objective is not necessarily to produce more documentation.

    It is to establish whether we have sufficient understanding and evidence to manage the product responsibly.

    The Product Outlives the Project

    Products often continue long after the projects that created them have ended.

    The original architects leave. Development teams change. Technologies evolve. New requirements emerge.

    Yet the product must still be understood, maintained, assessed and developed.

    Traditional documentation can preserve valuable information, but it may not preserve the relationships needed to interpret that information.

    A product model could provide continuity.

    Not by anticipating every question future people might ask, but by maintaining a coherent representation of the product and its relationship to organizational intent, decisions and evidence.

    It could also make that understanding easier to interrogate.

    Why was this component introduced? Which capabilities would be affected if we removed it? What evidence supports the assumption that this design remains appropriate?

    The answers may require information from several sources.

    The reference model helps establish how those sources relate.

    More Than Documentation

    Perhaps the important distinction is not between having documentation and having a model.

    Both can be useful.

    The distinction is between preserving information about a product and maintaining an understanding of the product as a subject that must continue to be managed.

    A product reference model provides the conceptual structure.

    A product model applies that structure to a particular product.

    Documentation and other evidence support the resulting understanding.

    Documenting a product helps us preserve what we know about it.

    Modeling a product helps us understand how that knowledge relates — and what we still don’t know.

  • Practice Note: A Reference Model Is Not a Prescription

    Two teams use exactly the same testing practices.

    One receives a positive capability assessment. The other does not.

    Is the assessment inconsistent?

    Not necessarily.

    One team develops a relatively simple internal application. The other develops a safety-critical subsystem.

    The practices may be identical, but the expectations should not be.

    This raises an interesting question.

    How can a generic capability reference model remain useful when adequate capability depends on context?

    From Generic to Specific

    A capability reference model describes what an organization should be capable of doing, without necessarily prescribing how that capability must be realized.

    Consider Quality Assurance.

    A generic reference model might describe capabilities related to quality strategy, verification, validation, test design, defect management and quality evaluation.

    These concepts can apply across many organizations.

    But their realization depends on context.

    An organization developing medical devices faces different risks, obligations and quality expectations from one developing internal business applications.

    Even within the same organization, different products and subsystems may require different approaches.

    A safety-critical component might demand extensive traceability and formal verification. A user interface might place greater emphasis on usability and exploratory testing. A data integration component might require particular attention to data integrity and failure handling.

    The underlying capability concepts remain recognizable.

    What constitutes adequate capability changes.

    Contextualizing the Reference Model

    This suggests a progression:

    Generic capability → Organizational context → Product or subsystem context

    At the generic level, the reference model provides a common vocabulary and a basis for reasoning about capability.

    At the organizational level, we consider the organization’s objectives, responsibilities, operating environment, risks and constraints.

    At the product or subsystem level, we consider the specific characteristics and quality expectations of the subject being assessed.

    Each level adds context.

    But that doesn’t necessarily mean creating an entirely new capability model at every level.

    The reference model can remain conceptually stable while the expectations, evidence and appropriate realization become increasingly specific.

    For example, risk-based testing may be a relevant capability in both a business application and a safety-critical subsystem.

    But what constitutes sufficient risk analysis, test coverage and supporting evidence may differ substantially.

    The capability is recognizable in both contexts.

    Its adequate realization is not necessarily identical.

    Assessing Capability or Checking Compliance?

    This distinction matters because reference models can easily become checklists.

    Does the organization have a test strategy?

    Does it use automated testing?

    Does it perform reviews?

    Does it maintain traceability?

    These questions may be useful, but the existence of a practice doesn’t necessarily establish that the capability is adequate.

    An organization can perform all the expected activities while remaining unable to achieve the intended outcomes.

    Conversely, an organization may realize a capability effectively through practices that differ from those anticipated when the reference model was created.

    If we assess compliance with prescribed practices, we risk confusing the means with the capability.

    Capability assessment should evaluate fitness for purpose, not conformity to a universal implementation.

    This also means that identical practices need not produce identical assessment results.

    And different practices may legitimately support the same capability rating.

    The question is whether the subject can achieve what is required of it in its particular context.

    The Risk of Contextualizing Too Much

    There is a complication.

    If every organization, product or subsystem receives its own completely customized model, we may lose the consistency that made the reference model useful in the first place.

    Assessments become difficult to compare. Common capability concepts disappear behind local terminology. Improvement approaches become unnecessarily fragmented.

    But contextualizing too little creates the opposite problem.

    We evaluate subjects against generic expectations that may be irrelevant, insufficient or inappropriate.

    The challenge is therefore to distinguish between what should remain stable and what should vary.

    The reference model provides the conceptual structure.

    Context determines which capabilities matter, what adequate performance means and what evidence is needed to assess them.

    Sometimes the context may even reveal capabilities or dimensions missing from the generic model.

    That should be an opportunity to improve the reference model, rather than a reason to abandon it.

    A Reference Model Is Not a Prescription

    Perhaps the value of a capability reference model lies precisely in this distinction.

    It allows us to reason consistently about different subjects without assuming that they should all operate identically.

    It provides a shared language for understanding, assessing and developing capabilities.

    But it does not eliminate the need for professional judgment.

    We still need to understand the subject, its purpose, its environment and the outcomes it must achieve.

    Only then can we determine what adequate capability looks like.

    A reference model tells us what to reason about. Context tells us what good looks like.

  • Exploration: Is Moving Upstream a Professional Quality?

    A tester can execute tests perfectly against the wrong requirements.

    A developer can implement an elegant solution to the wrong problem.

    A project manager can deliver a project successfully without achieving the change it was intended to create.

    A consultant can implement an improvement initiative without improving anything.

    In each case, the professional may have performed the assigned work competently.

    But was it good professional work?

    Looking Beyond the Assignment

    Consider a tester who discovers that a requirement is ambiguous.

    The immediate response might be to request clarification so that testing can continue.

    But perhaps the ambiguity suggests a more fundamental problem with requirements management.

    And perhaps that problem originates in how stakeholders communicate, how decisions are made or how the organization understands customer needs.

    The tester can keep moving upstream.

    From testing to requirements. From requirements to product development. From product development to organizational decision-making.

    At each step, the question changes.

    Not simply How can I do my work better?

    But Why does this work need to be done, and what determines whether it creates value?

    This pattern isn’t unique to testing.

    A developer might question the architecture rather than simply implement another workaround.

    A project manager might question whether the planned deliverables will produce the intended outcomes.

    A consultant might question whether the requested intervention addresses the capability that actually needs improvement.

    In each case, moving upstream means looking beyond the immediate assignment toward the conditions and purposes that give it meaning.

    Is This Professional Development?

    We often associate professional development with increasing expertise.

    A tester becomes better at testing. A developer becomes better at programming. A consultant becomes more proficient in a methodology.

    But perhaps professional development also has another dimension.

    The ability to understand how one’s work contributes to something larger.

    That doesn’t necessarily mean moving into management or changing professions.

    A tester doesn’t need to become a business analyst to recognize weaknesses in requirements.

    A developer doesn’t need to become a product manager to question whether a feature solves the intended problem.

    And a consultant doesn’t need to become a strategist to recognize that implementing a framework is not the same as developing an organizational capability.

    Perhaps moving upstream isn’t about changing what we do.

    It’s about changing how we understand what we do.

    How Far Upstream?

    There is a problem with this reasoning.

    We can always ask another question.

    Why are the requirements poor? Why does the organization work this way? Why were these priorities chosen? Why does the business pursue these objectives?

    Eventually, questioning the purpose of the work can consume more attention than performing it.

    And not every upstream question is useful.

    Sometimes the problem is understood, the assignment is appropriate and the most valuable contribution is excellent execution.

    Specialization exists for good reasons. Deep expertise matters.

    A professional who continually challenges the assignment but cannot perform it well isn’t necessarily demonstrating superior professional quality.

    So perhaps moving upstream is not inherently better.

    The important quality might instead be knowing when the boundaries of the assignment prevent us from addressing the problem.

    And having the judgment to recognize when those boundaries should be questioned.

    When Implementation Becomes Easier

    AI adds another dimension to this question.

    As AI becomes increasingly capable of producing code, tests, documents, analyses and other artifacts, some implementation activities become easier.

    That doesn’t eliminate the need for expertise.

    But it may change where professional judgment contributes the most value.

    If producing a solution becomes easier, choosing the right problem becomes more important.

    AI can help implement what we ask for.

    It cannot make our intentions appropriate merely by implementing them successfully.

    Perhaps this makes the ability to move upstream more valuable.

    Not because everyone should become a strategist, but because the consequences of poorly understood intentions may become easier to realize.

    We may become increasingly effective at doing things without becoming equally effective at determining whether those things are worth doing.

    A Different Dimension of Professional Quality?

    Perhaps we should distinguish three things.

    Competence: Can I perform the assigned work well?

    Understanding: Do I know why the work matters?

    Judgment: Can I recognize when performing the assigned work well will not achieve its intended purpose?

    The first is essential.

    The second provides context.

    The third allows a professional to challenge the relationship between the assignment and its purpose.

    But even this distinction raises questions.

    How much upstream understanding should we expect from a professional?

    When does questioning an assignment demonstrate responsibility, and when does it become interference?

    Can organizations genuinely encourage professionals to challenge assumptions while continuing to evaluate them primarily on how efficiently they complete assigned tasks?

    And what happens when a professional recognizes an upstream problem but has no authority to influence it?

    Perhaps moving upstream is not simply an individual quality. It may also depend on whether the organization is capable of using the understanding that professionals develop.

    Perhaps the Question Is Different

    Maybe professional quality isn’t about moving upstream as far as possible.

    Nor is it about constantly questioning everything.

    It may be about retaining the ability to connect our work to its purpose, and recognizing when that connection is missing.

    A good professional knows how to perform the work.

    A better professional may also recognize when performing that work well is insufficient.

    But that leaves an uncomfortable question.

    If we recognize that the assignment will not achieve its intended purpose, is it still professional to complete it exactly as requested?

  • Practice Note: A Portfolio as a Vocabulary for Expression

    A portfolio is often justified through coverage. Different constituents provide different capabilities, allowing the portfolio as a whole to address more situations than any individual constituent could.

    But a portfolio can provide another kind of value: expression.

    Consider a small collection of watches.

    An industrial Victorinox I.N.O.X., an elegant ice-blue Presage, a green Alpinist and a solar panda chronograph all perform essentially the same basic function. Their practical use cases also overlap considerably.

    Yet choosing between them can still be meaningful.

    The I.N.O.X. might express resilience and confidence. The Presage refinement without complete conformity. The Alpinist curiosity and adventure. The panda technical practicality and precision.

    The question in the morning is therefore not necessarily:

    Which watch do I need today?

    It might be:

    Which aspect of myself do I want to express today?

    A Portfolio as a Vocabulary

    Words are not valuable because one ranks above another. They are valuable because different words allow different things to be expressed.

    Even words with similar meanings can carry different nuances.

    Replacing one word with a supposedly superior word makes little sense if the two communicate different things.

    The same can be true of portfolio constituents.

    Two constituents may have different specifications while communicating essentially the same thing. Their contribution to the expressive range of the portfolio may therefore be small.

    Conversely, two constituents may overlap substantially in practical capability while remaining valuable together because they enable different expressions.

    This gives us another question for evaluating portfolio composition:

    What can this constituent express that the existing portfolio cannot express adequately?

    Difference alone is not enough. A collection containing many minor variations may contain many objects while providing little additional expressive range.

    The differences must be meaningful in relation to the intent of the portfolio.

    Beyond Functional Coverage

    Functional coverage remains important, but it need not be the complete explanation for portfolio composition.

    Some portfolios also provide a vocabulary of meaningful differences.

    Their constituents are valuable not because they rank above one another, but because they allow different things to be expressed.

    The portfolio provides the vocabulary.

    The choice provides the expression.