Category: Assessment

Quality, testing, assessment, evidence, judgment, reference models, and the development and evaluation of quality and capability.

  • Practice Note: The Roadmap Should Not Be Part of an Assessment

    Why “What’s next?” should begin transformation, not complete assessment

    Assessment reports traditionally end with recommendations and a roadmap. The pattern is familiar:

    current state → gaps → recommendations → roadmap → implementation

    It appears complete and helpful. The assessor identifies what is wrong and then explains how the organization should improve it. The client receives both a diagnosis and a treatment plan. I have worked with that assumption myself.

    But the more clearly I distinguish assessment from transformation, the more I think this familiar sequence hides an important change in professional authority and ownership. An assessment can establish what the evidence supports about the current situation. It cannot, by itself, determine which future the organization should choose. The roadmap is therefore not the final assessment deliverable. It is instead one of the first transformation artifacts.

    Why the traditional sequence feels so natural

    Assessment findings often seem to point directly toward action. If accountability is unclear, clarify accountability. If feedback arrives too late, shorten the feedback loop. If evidence is unreliable, strengthen the evidence process. If a capability dimension is weak, improve it toward the next maturity level.

    This appears to create an almost mechanical progression from gap to recommendation and from recommendation to roadmap. This is especially driven by maturity models, where improvement automatically means a higher maturity level.

    But a gap does not determine a unique response. An organization may choose to:

    • close the gap;
    • reduce its consequences;
    • accept the risk;
    • postpone action;
    • gather more evidence;
    • change its ambition;
    • narrow the scope;
    • or stop doing the work that makes the capability necessary.

    Even when improvement is chosen, several interventions may be legitimate. A capability weakness might be addressed through leadership, roles, skills, practices, tools, governance, information, incentives, capacity, or organizational structure.

    The assessment can reveal the gap and explain its significance. It cannot determine automatically which response is right.

    What the assessor can own

    The assessor, supported by a domain specialist, can own the professional integrity of the current-state judgment. That includes:

    • defining the subject and scope;
    • identifying the decision context;
    • selecting and applying an appropriate reference model;
    • gathering and validating evidence;
    • interpreting that evidence;
    • identifying strengths, gaps, risks, and uncertainties;
    • judging evidence quality and completeness;
    • stating the confidence that can be placed in the conclusions;
    • determining whether the subject is good enough for its purpose and context;
    • explaining which decisions the judgment can support.

    This is substantial professional work.

    The assessor should be able to explain what was observed, how it was interpreted, which reference was used, and why the resulting judgment is justified. Together, the assessor and domain specialist can responsibly own the statement:

    This is what the available evidence supports about the current situation.

    They cannot independently own the statement:

    This is the future the organization should pursue, and this is how it should get there.

    The assessor can explain implications, but not choose the future

    The boundary does not mean that an assessment must end with a neutral description of facts. The assessor can explain the implications of the current state. The assessment can state:

    • which risks are created by the current capability;
    • which ambitions the organization may be unable to support;
    • which dependencies or weaknesses constrain future decisions;
    • what may happen if nothing changes;
    • where evidence is insufficient;
    • which matters require leadership attention.

    These are consequences of the current-state judgment. They remain within assessment because they explain what the evidence means.

    Improvement goals are different. An improvement goal expresses what the organization intends to make possible in the future. It requires choices about ambition, timing, investment, risk tolerance, affected groups, competing priorities, and acceptable disruption. Those choices cannot be derived from assessment evidence alone.

    Improvement goals require business ownership

    Suppose an assessment concludes that an application delivery capability cannot reliably support the organization’s intended growth.

    The assessment may establish the capability gap with high confidence. But the appropriate improvement goal depends on questions such as:

    • How important is the growth ambition?
    • Which products, customers, or markets matter most?
    • How quickly must the organization respond?
    • What investment and capacity are available?
    • Which other initiatives compete for the same people?
    • How much disruption is acceptable?
    • Which risks is leadership prepared to carry?
    • Which existing strengths must be protected?
    • Is the original ambition still realistic?

    These are business questions.

    The assessor and domain specialist can contribute evidence, challenge assumptions, and explain capability implications. But they do not possess all the knowledge or authority needed to choose the future state alone.

    The client must own the improvement goals because the client owns the ambition, consequences, resources, and trade-offs.

    Improvement suggestions are already transformation choices

    It may be tempting to say that the roadmap belongs to transformation while recommendations or improvement suggestions can remain assessment outputs. But even a suggestion introduces assumptions about the future.

    “Introduce clearer governance” assumes that additional governance will improve decisions rather than add reporting and delay.

    “Create a dedicated role” assumes that the problem concerns responsibility rather than authority, incentives, or capacity.

    “Standardize the process” assumes that consistency is more valuable than local flexibility.

    “Implement a new tool” assumes that tooling is an important constraint and that the organization can adopt the tool effectively.

    “Provide training” assumes that insufficient knowledge is the cause of weak behavior.

    These may all be reasonable intervention hypotheses.

    They are not conclusions contained directly in the assessment evidence.

    Improvement suggestions should therefore be developed after the organization has decided to intervene and clarified what it wants to achieve.

    “What’s next?” is a transition signal

    When an assessment has made the current situation visible, the most natural client response is often:

    What should we do next?

    That question does not reveal a missing assessment deliverable. It shows that the assessment has succeeded in creating enough clarity for the organization to consider change.

    Answering “What’s next?” requires a different kind of work:

    • clarifying business ambition;
    • defining the required future state;
    • setting improvement goals;
    • involving the people who will own and experience the change;
    • developing alternative interventions;
    • evaluating feasibility and dependencies;
    • making priorities and trade-offs explicit;
    • assigning ownership and capacity;
    • deciding how progress and learning will be evaluated;
    • creating an initial roadmap.

    This is not administrative completion of the assessment. It is the substantive beginning of transformation.

    The assessment should not end with the assessor’s roadmap. It should end with sufficient shared clarity for the client to decide whether to create one.

    Client ownership does not mean client-only work

    Moving goals, suggestions, and roadmaps into transformation does not mean that the assessor leaves after presenting the judgment. The assessor and domain specialist may be well positioned to facilitate or lead the next phase. They understand the evidence, reference model, risks, uncertainties, and reasoning behind the current-state judgment.

    They can help the organization:

    • translate business ambition into capability outcomes;
    • define a realistic future state;
    • develop and compare intervention options;
    • preserve traceability to the assessment evidence;
    • identify dependencies and possible unintended consequences;
    • distinguish outcomes from activities;
    • treat interventions as hypotheses;
    • define evidence of adoption and improved capability;
    • establish review and reassessment points.

    Their role has changed, however.

    As assessors, they determine what the evidence supports about the present. As transformation advisors, they help the organization make choices about the future. The distinction is between leading the process and owning the decisions.

    The advisor may structure the work, make the reasoning visible, challenge weak assumptions, and contribute expertise. The client must still choose the goals, accept the trade-offs, commit the capacity, and own the roadmap.

    A consultant can facilitate the creation of a roadmap. Only the organization can turn it into an organizational commitment.

    The roadmap is more than a list of actions

    A roadmap is sometimes treated as a prioritized list of assessment recommendations. But a real roadmap does much more.

    It chooses which outcomes matter. It commits scarce capacity. It establishes dependencies and ownership. It decides which risks will be addressed and which will remain. It affects teams, budgets, roles, and existing commitments. It creates consequences for people who may not have participated in the assessment. A roadmap therefore contains organizational decisions.

    A consultant-created list of actions may be useful input, but without client choice and ownership it remains a proposal. It is not yet a roadmap in the full sense.

    From assessment to transformation

    Assessment creates clarity about the present. It establishes the current state, the relevant reference state, the evidence, the risks, the uncertainty, and the resulting judgment. The organization then decides whether the current state should be accepted, investigated further, or changed.

    When change is authorized, transformation begins. Its initial work includes defining the intended future state, setting improvement goals, developing and comparing intervention options, assigning ownership and capacity, and creating an initial roadmap.

    Later transformation work focuses on implementing, adapting, scaling, embedding, and stabilizing the changes that appear to improve the capability. Reassessment then determines whether the subject itself has become more capable, reliable, or fit for purpose.

    A stronger service transition

    There is also a practical and commercial consequence. When the roadmap is included as an appendix to the assessment, the engagement can end awkwardly:

    “Here are our findings, our suggestions, and the roadmap. Good luck implementing it. Would you now like our help with Accelerate?”

    That gives away much of the most important transformation reasoning before the necessary business conversations, ownership decisions, and trade-offs have occurred.

    A stronger transition is:

    “We now have a shared, evidence-based understanding of the current situation. The next question is what future capability you need, which goals matter, which interventions are realistic, and how the organization wants to own and sequence the change.”

    The next phase is not created artificially by withholding useful advice. It exists because defining the future and creating an owned roadmap require knowledge, participation, and authority that assessment alone cannot provide. The client’s question—“What’s next?”—naturally opens that phase.

    A cleaner professional boundary

    Assessment, transformation, and reassessment can remain part of one continuous service. But they are not the same professional act.

    Assessment asks:

    What does the evidence support about the current state?

    Transformation asks:

    What does the organization intend to change, why should it change, and how will that change be pursued?

    Reassessment asks:

    Did the subject actually become more capable, reliable, or fit for purpose?

    Separating these responsibilities does not weaken continuity. It makes the continuity more trustworthy. The assessor owns the integrity of the current-state judgment. The client owns the future it chooses to pursue.

    The assessor and domain specialist may facilitate or lead the journey between them—but the roadmap becomes real only when the organization makes it its own.

  • Practice Note: A Product Reference Model Through the Eyes of an Enterprise Architect

    When I first began thinking about a Product Reference Model, I imagined it as something new. Over time, I have become less convinced that it is.

    Perhaps I have simply been rediscovering ideas that Enterprise Architecture has explored for decades—but applying them at a different scale and with a different objective. Enterprise Architects rarely begin with source code. They think about relationships. Business capabilities support strategic objectives. Applications realize capabilities. Technology supports applications. Information flows between systems. The architecture is not merely a collection of artifacts. It is the set of relationships that makes those artifacts meaningful.

    That sounds surprisingly familiar. My Product Reference Model has gradually evolved into something that also emphasizes relationships. Business objectives relate to capabilities. Capabilities relate to features. Features relate to requirements. Requirements relate to BDD scenarios. BDD scenarios relate to implementation. Implementation relates to architecture. Architecture relates to operational procedures. Assessments relate back to business outcomes. Rather than replacing individual artifacts, the Product Reference Model attempts to make those relationships explicit.

    Perhaps that is exactly what Enterprise Architecture has always tried to achieve—only at enterprise scale. If that is true, then perhaps the Product Reference Model should not be viewed as a competing framework. Instead, it may simply be Enterprise Architecture viewed through the lens of a single product.

    The emphasis is different. Enterprise Architecture asks:

    How does this product fit within the enterprise?

    The Product Reference Model asks:

    How does every important aspect of this product fit together?

    That distinction may appear subtle, but it changes the audience. Enterprise Architects often think across portfolios, organizations and technologies. Product Owners, developers, testers and operations engineers think about one product. The underlying architectural principles remain remarkably similar.

    What also changes is the purpose. Traditional Enterprise Architecture often focuses on describing structure. The Product Reference Model attempts to preserve understanding.

    Every relationship should ultimately help answer one of the questions future teams inevitably ask: Why does this capability exist? Why was this architectural decision made? Why is this business rule still here? Where is the evidence that this requirement is implemented? Which assessment concluded that this capability was fit for purpose? How do all these artifacts relate to one another?

    Perhaps the Product Reference Model is therefore less concerned with documenting a product than with making the product understandable.

    It becomes a navigational model rather than merely a descriptive one. That observation also changes my own perspective. Instead of inventing another framework, perhaps I should learn more from Enterprise Architecture. Architects have already spent decades thinking about repositories, viewpoints, traceability, relationships, stakeholders and governance.

    The Product Reference Model may simply extend those principles into the daily life of a product team. Its contribution would not be another modeling language. Its contribution would be placing product understanding at the center.

    Perhaps the Product Reference Model can therefore be described quite simply:

    The Product Reference Model applies Enterprise Architecture principles to a single product, with particular emphasis on preserving product understanding throughout the product’s lifecycle.

    If Enterprise Architecture asks how an enterprise remains coherent…

    …perhaps the Product Reference Model asks how a product remains understandable.

  • Practice Note: A Product Reference Model Through the Eyes of a Product Owner

    When I first began thinking about a Product Reference Model, I assumed it would become another documentation framework.

    As a Product Owner, that was not particularly attractive. Most Product Owners already spend their days balancing stakeholder expectations, refining the backlog, making priority decisions and answering questions from delivery teams. Another repository sounded like another responsibility.

    But perhaps that is the wrong way to think about it. As Product Owners, we rarely inherit a greenfield product. We inherit decisions. Business rules. Architecture. Operational procedures. Quality concerns. Technical debt. And, perhaps most importantly, we inherit understanding—or the absence of it.

    One of the most common questions a Product Owner asks is remarkably simple:

    Why?

    Why does this capability exist? Why do these customers receive different treatment? Why was this integration introduced? Why is this feature impossible to remove? Why was this architectural decision made? Why is this quality requirement so important?

    Those answers rarely exist in one place. Some are hidden in Jira. Some in architecture documentation. Some in source code. Some in BDD scenarios. Some in assessment reports. Many remain only in the memories of people who happened to be there.

    The Product Reference Model begins to make sense when it is no longer viewed as documentation. Instead, it becomes a map of product understanding. The Product Owner continues owning the backlog. Architects continue owning architecture. Developers continue owning source code. QA continues owning assessments and quality strategy. Operations continues owning operational knowledge.

    Nothing changes. Except that all these pieces become visible to one another. The Product Owner is no longer expected to know everything. The Product Owner is expected to know where the understanding lives.

    That changes the role in an important way. The backlog is no longer the product. It is merely one viewpoint of the product. The Product Reference Model allows the Product Owner to navigate the complete product landscape without becoming responsible for maintaining every artifact personally. Perhaps that is the real value.

    The Product Owner becomes the steward of product understanding rather than merely the owner of the backlog.

    Stewardship is different from ownership. Ownership often implies creating and maintaining artifacts. Stewardship means ensuring that future Product Owners inherit enough understanding to continue making good decisions.

    The Product Reference Model therefore does not make the Product Owner responsible for more documentation. It makes the Product Owner responsible for ensuring that the important answers remain connected. Perhaps continuity is not ultimately about preserving software. Perhaps it is about ensuring that every new Product Owner can understand the product they inherit before deciding how it should evolve.

  • Practice Note: The Anatomy of a Product Reference Model

    When I first imagined a Product Reference Model, I thought of it as another repository for product knowledge. I no longer think that is the right mental model.

    A product already contains many repositories:

    • The Product Owner maintains the backlog.
    • Enterprise Architects maintain architecture models and Architecture Decision Records.
    • Developers maintain source code.
    • QA maintains assessments, test strategies and quality evidence.
    • Operations maintains runbooks and operational procedures.
    • Security teams maintain security documentation.

    Every discipline already has tools and repositories that support its own work. The Product Reference Model should not replace them. Instead, it should index them. For every significant artifact, it should make clear:

    • What is this artifact?
    • Where can it be found?
    • Who owns it?
    • Why does it exist?
    • How does it relate to the rest of the product?
    • Who depends on it?
    • Can everyone who needs it access it?

    The detailed knowledge remains within the artifact itself. The Product Reference Model provides the shared navigation structure through which that knowledge can be discovered and understood.

    This distinction is important. The Product Reference Model is not another documentation repository.

    It is a lightweight knowledge architecture.

    Perhaps Enterprise Architecture provides the right inspiration. Enterprise Architects have spent decades thinking about repositories, viewpoints, relationships, governance and traceability across an enterprise.

    The Product Reference Model applies many of those same principles at the scale of a single product. Rather than describing how an enterprise fits together, it describes how one product remains understandable throughout its lifecycle.

    Preserve knowledge where it belongs

    The Product Reference Model should duplicate as little information as possible. Every artifact should preserve its own knowledge.

    Requirements should explain the need behind them. Architecture Decision Records should preserve the trade-offs behind important decisions. BDD scenarios should preserve expected behavior. Operational procedures should explain the situations they were created for.

    Whenever an artifact already contains sufficient rationale, the Product Reference Model simply links to it. Whenever the rationale is missing, the Product Reference Model becomes the place where that missing understanding is preserved.

    The objective is therefore not more documentation. The objective is preserving understanding.

    A federated knowledge model

    The Product Reference Model also changes how I think about collaboration. Initially I believed that every stakeholder simply contributes information and everyone benefits.

    That is true, but incomplete: every stakeholder already possesses unique understanding. The Product Owner understands business intent. Architects understand structural decisions. Developers understand implementation. QA understands quality and evidence. Operations understands behavior in production.

    No single discipline possesses the complete understanding of the product. The Product Reference Model therefore creates a federated knowledge model.

    Each stakeholder maintains ownership of their own artifacts while making that understanding discoverable to everyone else. Ownership remains distributed. Understanding becomes shared.The Product Owner does not become an architect. The architect does not become a tester. QA does not become an operations engineer. Every discipline remains responsible for its own expertise.

    The Product Reference Model simply allows everyone to benefit from the expertise maintained by every other discipline.

    Knowledge amplification

    This also reveals the true value of the Product Reference Model. It does not create new knowledge. It amplifies the value of knowledge that already exists. Without the Product Reference Model, every discipline works largely within its own information space. With the Product Reference Model, each discipline gains visibility into the understanding maintained by all the others. The Product Owner better understands architectural constraints. The architect better understands business intent. QA better understands operational realities. Operations better understands the reasons behind design decisions. Developers better understand business priorities.

    Collective understanding increases without requiring everyone to become an expert in every discipline. Its value is therefore not additive. It is multiplicative. Every additional viewpoint increases the value of all the others.

    A practical implementation

    The Product Reference Model could simply exist as a structured wiki or similar workspace. Rather than storing all product knowledge itself, it provides a consistent structure through which stakeholders expose the artifacts they already own. Each section contains references to relevant artifacts together with any rationale that is missing from those artifacts themselves. The Product Reference Model therefore becomes the product’s navigation layer rather than its documentation repository.

    A simple definition

    Perhaps the Product Reference Model can be summarized as follows:

    The Product Reference Model is a federated knowledge architecture that indexes the significant artifacts of a product, preserves their relationships and rationale, and amplifies collective understanding by allowing every stakeholder to contribute the understanding they already possess while gaining access to the understanding maintained by every other stakeholder.

    Design principles

    • The Product Reference Model should contain as little duplicated information as possible.
    • Knowledge remains with the artifact that owns it.
    • Understanding is created by connecting those artifacts.
    • Every significant artifact should preserve not only what it contains, but also why it exists.
    • Every stakeholder contributes local understanding.
    • Every stakeholder gains systemic understanding.
  • Practice Note: The Boundary Defines the Object

    We often think that an object’s identity is determined solely by the object itself. It isn’t. The boundary around an object profoundly influences how we perceive the object within it.

    A photograph does not change when you replace its frame. Yet a black frame can make the colors appear richer, while a wide white mat can make the same photograph feel more spacious or more like a piece of art. A television does not become larger when Philips Ambilight projects light onto the wall. Yet the image appears to extend beyond the physical screen because the boundary of the display has changed.

    The same principle appears in industrial design. While experimenting with different straps on a Victorinox I.N.O.X. wristwatch, I noticed something unexpected. The watch itself—the steel case, dial and movement—remained exactly the same. Only the strap changed.

    On the steel bracelet, the watch is perceived as one continuous piece of engineering—a substantial object made of steel. The steel bracelet and the case compete for attention, together forming a single architectural object. Replacing the bracelet with a black leather strap completely changes the visual hierarchy. The leather quietly recedes into the background, allowing the eye to focus almost entirely on the distinctive hexagonal steel case. The watch has not changed. Yet it feels like a different object. The boundary determines what our attention considers to be the object.

    Music demonstrates the same principle in time rather than space. A skilled musician knows that silence is not the absence of music—it is part of the composition. Bass players, for example, will sometimes stop playing briefly to create tension. The temporary void changes how the audience experiences what follows. When the bass returns, listeners instinctively experience a sense of release. The notes have not become better. The silence has changed how they are perceived. In music, silence is the boundary that defines the notes.

    Good design often understands this instinctively. It is not only the product that is designed; it is also the context in which the product is experienced. Architecture uses pedestals, lighting and empty space to make sculptures more powerful. Graphic design uses margins and whitespace to make information easier to understand.

    Perhaps this is a more general design principle:

    The boundary does not merely surround the object. It defines the object.

    When trying to improve something, we often redesign the object itself. Sometimes the better question is:

    Can I change the boundary instead?

    If this principle applies equally to photography, architecture, music, industrial design and product design…

    …what might it mean for software?