Blog

  • Field Note: Product Quality Can Only Be Fully Understood in Operation

    Product quality must be assessed before release. Testing, reviews and demonstrations help us decide whether the product is ready.

    But that judgement is always provisional. Before release, we assess the product under selected conditions. After release, reality adds evidence we could not fully create in advance.

    Pre-release evidence is necessary, but incomplete

    Testing can show whether expected behaviour is present, whether important risks have been explored and whether quality characteristics such as performance, security or resilience meet defined expectations.

    But testing always uses models. We select scenarios, create representative data, simulate load and define expected outcomes.

    All of this is essential. But it cannot reproduce the full complexity of real use. Actual users behave differently from test users. Real data contains unexpected patterns. Dependencies fail in unfamiliar combinations. Support demand reveals confusion that no acceptance test exposed. A product may pass every planned assessment and still teach us something important after release.

    Operation produces a different kind of evidence

    Once the product is in use, new evidence appears:

    • incidents and escaped defects;
    • helpdesk calls and customer complaints;
    • performance under real load;
    • usage and abandonment patterns;
    • manual workarounds;
    • operational effort;
    • customer and business outcomes.

    This evidence does not merely tell us whether engineering made mistakes. It tells us how the product behaves in its actual context. A repeated support question may reveal weak usability. A production incident may expose an architectural assumption that was never explicit. Low adoption may show that a correctly implemented feature does not solve the intended problem. High operational effort may reveal maintainability or supportability problems. Reality makes the product more understandable.

    The team’s responsibility continues after release

    In many agile and DevOps environments, the development team does not hand the product over and disappear. The same people who design, build and test the system also help operate, support, maintain and improve it. That creates a direct quality-learning loop. Incidents reveal weaknesses in design, testing or operational preparation. Difficult fixes expose maintainability problems. Repeated manual interventions show where operability or automation is insufficient. Production behaviour tests assumptions made during engineering.

    This evidence should matter deeply to the team. It is not merely reporting about a product they once delivered. It is feedback on the quality of their own engineering decisions. The team that creates the product should also learn from how that product behaves in reality. Done may end the implementation cycle. It should not end the team’s interest in quality.

    Monitoring is not yet assessment

    Production systems generate dashboards, logs, alerts and incident records. But observations do not interpret themselves. A green dashboard does not prove that the product is good. A metric may remain within its threshold while users still struggle.

    Operational evidence must be interpreted against:

    • intended outcomes;
    • stakeholder needs;
    • quality expectations;
    • acceptable risks;
    • operating context.

    Monitoring produces evidence. Assessment turns that evidence into a quality judgement.

    The reference model must survive release

    A product-quality reference model cannot be used only before delivery. It must remain relevant in operation. The expectations defined earlier should help us interpret what happens in reality.

    At the same time, operational evidence should be able to change the reference model. An incident may reveal a missing quality characteristic. Customer behavior may challenge an assumption. A support pattern may expose a need that was never understood. A change in scale may make yesterday’s acceptable quality level inadequate today.

    The reference model must therefore be living. It provides the basis for assessment, but it also evolves through assessment.

    Quality learning follows the lifecycle

    The Definition of Done marks an important engineering boundary. Release marks another. Neither marks the end of quality. Before release, we ask whether the available evidence gives us enough confidence to proceed. After release, we ask what reality reveals about the product in actual use. These are different stages of the same quality-learning process.

    Product quality must be assessed before release, but it can only be fully understood in operation. A mature team prepares the product carefully before release—and remains willing to learn what reality reveals afterwards.

  • Field Note: From Shared Responsibility to Shared Quality Judgment

    Agile approaches have argued for years that quality is a shared responsibility. This was an important correction to the idea that quality belongs to testers at the end of delivery. Product Owners, analysts, architects, developers, testers and specialists all contribute to the quality of the product.

    But shared responsibility does not automatically create shared understanding. Different roles may all take quality seriously while meaning different things by it:

    • The Product Owner may focus on value and customer outcomes.
    • The architect may focus on resilience, security and maintainability.
    • Developers may focus on technical sustainability.
    • Testers may focus on risks, behaviour and evidence.
    • Operations may focus on stability and recoverability.
    • Users may focus on whether the product actually helps them accomplish their work.

    None of these perspectives is wrong. Product quality is multidimensional.

    The problem is that the perspectives often remain fragmented across separate artefacts, conversations and professional practices. They may only be brought together when a release decision must be made—or when something fails in production. Everyone may be responsible for quality while nobody has a coherent view of the whole.

    Shared responsibility is not shared judgment

    Responsibility describes who contributes to the work. Judgment asks a different question:

    Does the combined evidence support the conclusion that the product is good enough for its intended purpose?

    That conclusion cannot be reached by adding together completed quality activities. Code review, automated testing, security analysis, usability research and operational monitoring may all be valuable. But their meaning depends on the quality needs, risks and context of the product.

    The organisation therefore needs more than shared participation. It needs a shared basis for reasoning about quality.

    A shared object of reasoning

    A product-quality reference model could provide that basis.

    It would not assign all quality responsibility to one role. Instead, it would describe the quality space within which different roles contribute. It might make explicit:

    • which quality characteristics matter;
    • why they matter in this product context;
    • which stakeholders and scenarios are affected;
    • what risks arise when they are weak;
    • what evidence is relevant;
    • how adequacy should be judged.

    The reference model would allow different professional perspectives to remain distinct while still contributing to one product-quality judgment:

    • Architecture decisions could be connected to resilience or scalability needs.
    • Engineering practices could be connected to the evidence they produce.
    • Operational incidents could be connected to assumptions about reliability or usability.
    • Customer feedback could be connected to intended outcomes.

    The different roles would no longer contribute only to their own documents and controls. They would contribute to a shared quality profile of the product.

    This may be one of the most important functions of a reference model:

    It makes quality a shared object of reasoning.

    Shared does not mean undifferentiated

    There is a danger in saying that quality belongs to everyone. When everyone is responsible in the same vague way, important accountabilities may disappear.

    Shared quality work still requires different contributions:

    • Product Ownership maintains the connection between product intent, priorities and quality trade-offs.
    • Architects explain how design decisions support—or endanger—important quality characteristics.
    • Developers create the product and much of the technical evidence.
    • Specialists contribute depth in areas such as security, usability, data and operations.
    • Quality professionals contribute assessment expertise: challenging the completeness of the model, identifying relevant evidence, exposing uncertainty and supporting a defensible judgment.
    • Business decision-makers remain accountable for accepting residual risks associated with release, rollout or continued investment.

    These responsibilities are related, but they are not interchangeable. Shared responsibility should not erase differentiated accountability.

    From coordinated activity to coordinated reasoning

    Scrum helps make quality part of the work of the whole team. Scaled approaches add mechanisms for coordinating quality across teams, architectures and integrated solutions.

    But coordinated delivery does not necessarily produce a coherent product-quality judgment. Teams may satisfy their own acceptance criteria and Definitions of Done while system-level concerns remain unresolved.

    A product may be locally complete but still lack sufficient resilience, accessibility, operability or end-to-end coherence.

    A reference model could help distinguish:

    • what can be assessed within one team or backlog item;
    • what requires integrated product evidence;
    • what can only be judged in operation;
    • who must contribute to the final decision.

    The result would be more than coordinated quality activity. It would be coordinated quality reasoning.

    A further step for agile quality

    Agile methods helped move quality away from a separate downstream phase. They made quality part of creating the product.

    A product-quality reference model may enable the next step: making explicit how different perspectives, evidence and accountabilities combine into a credible judgment of fitness for purpose.

    The proposition is therefore not only:

    Everyone is responsible for quality.

    It is:

    Shared responsibility for quality requires a shared model of what quality means, differentiated accountability for contributing to it, and a shared basis for judging whether it has been achieved.

    That is the shift from shared responsibility to shared quality judgment.

  • Field Note: Product Ownership Has a Business Side. Where Are Its Tools?

    Product Ownership carries broad business accountability.

    The Product Owner is expected to understand customers, engage stakeholders, maximize value, establish product goals, make trade-offs, order the backlog and inspect outcomes. These responsibilities sound substantial.

    But when we look for practical tools to perform them, the picture becomes less clear. “Understand the customer” is not a customer-analysis technique. “Manage stakeholders” is not a method for reconciling conflicting interests. “Maximize value” does not explain how value should be defined, compared or assessed. “Inspect and adapt” does not tell us which evidence matters or how it should influence product direction.

    The Product Owner is told what to think about. But where are the tools for doing the work?

    Engineering has a visible operating model

    Scrum gives engineering delivery a clear structure:

    Product Backlog → Sprint → Increment → inspection and adaptation

    The roles, events, artifacts and commitments make the work visible. A team can discuss whether backlog items are ready, whether the Sprint Goal is clear, whether the Increment is Done and what should change next. The Product Owner appears at the beginning of this flow through the backlog.

    But the backlog is not the beginning of Product Ownership. Before something becomes a story, a much broader translation should already have taken place. Someone must connect:

    • business and product strategy;
    • customer and stakeholder needs;
    • intended outcomes;
    • assumptions and risks;
    • quality expectations;
    • business and operational constraints;
    • dependencies and wider product context;

    to something engineering can understand and act upon. The story is the visible engineering-facing output. The reasoning that should produce it remains largely hidden.

    Scrum hides the business perspective behind the Product Owner

    Scrum gives Product Ownership broad accountability but only a thin formal operating structure outside backlog and delivery. Advanced Product Owner training adds useful stances, exercises and techniques, but these do not necessarily form an explicit lifecycle model for translating business intent into engineering work and operational evidence back into business learning.

    SAFe provides more—but spreads it out

    SAFe makes more of the business perspective visible. It introduces concepts such as Product Management, customer centricity, design thinking, roadmaps, epics, features, non-functional requirements, Lean Business Cases and outcome hypotheses. That is more operational support than Scrum provides on its own.

    But the broader Product Ownership capability is also distributed across many roles, levels and artifacts. Strategy may sit with portfolio leadership. Product Management may own vision and features. Product Owners manage team backlogs. Business Owners represent broader outcomes. Architects address technical direction. Operations and support hold important evidence after release. The pieces exist.

    What is harder to see is the model that keeps them coherent. Who maintains the complete connection between:

    • why the product exists;
    • who it serves;
    • what quality means;
    • which assumptions and risks matter;
    • what engineering is asked to build;
    • what evidence the organisation receives;
    • what production teaches us;
    • what should change next?

    Scrum hides much of the business perspective behind one Product Owner. SAFe exposes more of it, but risks fragmenting it across the organisation.

    Why Product Owners are pulled downstream

    The most practical tools and recurring activities available to Product Owners are usually engineering-facing:

    • backlog refinement;
    • story clarification;
    • prioritisation;
    • Sprint Planning;
    • answering team questions;
    • accepting completed work.

    These activities are concrete, visible and urgent. They are also supported by the delivery rhythm.

    The upstream work is less structured. So is the work after Done and release. How should a Product Owner translate strategy into a coherent product perspective? How should production incidents, support calls, customer behavior and outcomes be interpreted? How should that learning change product priorities and future engineering work?

    Without equally practical tools for these responsibilities, the role is naturally pulled towards the part of the system where the work is clearest. Product Owners may not choose to become backlog administrators. The operating model pulls them there.

    The backlog is not enough

    The backlog is an essential engineering-facing artifact. It organizes and prioritizes work. It supports refinement and delivery. But it cannot carry the complete business perspective of the product. It is not designed to preserve all relevant knowledge about purpose, stakeholders, outcomes, quality, risks, context, evidence and learning. Yet in many organisations, it is the primary practical instrument provided to the Product Owner.

    Product Ownership will remain centred on backlog administration as long as the backlog is the only operational tool we provide.

    In search of a broader model

    Perhaps Product Ownership needs a more explicit way to maintain the product perspective across the lifecycle.

    Something that helps connect:

    business intent → product definition → engineering work → delivery evidence → production experience → outcomes → learning

    It may need to preserve intended outcomes, stakeholders, quality expectations, assumptions, risks, constraints, operating context and required evidence. Perhaps this could eventually take the form of a product reference model.

    We do not yet know exactly what that model should contain or how it should work. But the gap is becoming visible. Product Ownership has a business side. It carries broad accountability for that business perspective.

    What it still lacks are the practical tools to keep that perspective coherent before, through and after engineering.

  • Field Note: Product Ownership: Closing the Loop

    In an earlier field note, I argued for the return of the Product Owner.

    Product Ownership can easily become reduced to backlog administration: writing stories, prioritizing work, answering questions and accepting completed items. But that places the Product Owner almost entirely inside the engineering process.

    Product Ownership should connect engineering work to customer needs, business intent, desired outcomes and relevant quality expectations. I now think the Product Owner must return twice. First, before engineering begins. Then, after engineering declares the work done.

    But even that may describe the responsibility too narrowly. Product Ownership is not two separate handovers. It is the function that closes the complete loop between business intent, engineering delivery and product learning.

    Before the backlog

    We often picture the Product Owner at the start of the engineering sequence:

    Product Owner → story → implementation → testing → Definition of Done

    But Product Ownership should not begin with a story. Before work enters the backlog, someone must translate:

    • strategic intent;
    • customer and stakeholder needs;
    • desired outcomes;
    • risks and trade-offs;
    • business and operational constraints;
    • expectations of product quality;

    into product decisions that engineering can understand and act upon.

    The backlog is therefore not the beginning of product thinking. It is the point where the broader business and product perspective is translated into the engineering perspective.

    A story should not merely describe something that somebody requested. It should represent a sufficiently coherent product decision. This is the first return of the Product Owner: moving upstream from backlog administration towards responsibility for product intent.

    Engineering produces more than functionality

    Once work enters the backlog, the perspective changes. Stories, acceptance criteria, designs, implementation, testing and the Definition of Done help engineering turn product intentions into working increments.

    This sequence asks whether the intended behaviour is sufficiently clear, whether the solution has been implemented correctly and whether the agreed engineering work has been completed. It also produces evidence.

    By the time an increment is done, we know more about what was feasible, how the product behaves, which assumptions were correct, which risks remain and how the change affects the wider system.

    But Definition of Done is an engineering boundary. It is not the end of Product Ownership.

    After Done, reality provides new evidence

    Testing produces evidence under selected conditions. After release, the product enters a different evidence environment.

    Now we learn through:

    • incidents and defects in production;
    • helpdesk calls and customer complaints;
    • customer behaviour and usage patterns;
    • workarounds and abandonment;
    • operational effort and cost;
    • performance under actual load;
    • security and reliability events;
    • business and customer outcomes.

    These signals are not merely operational noise. They tell us something about the product’s quality in reality. A feature can pass every test and still confuse users. A system can satisfy its acceptance criteria and still generate excessive support demand. A correctly implemented capability can fail to create the intended value.

    Implementation produces a product change. Ownership continues until the organisation understands what that change means.

    The second return

    The evidence produced by engineering, operations, support and real use must be translated back into the product and business perspective:

    • Did the change improve the product as intended?
    • Does it contribute to a coherent experience?
    • Is it suitable for actual customers and operating conditions?
    • Did the expected outcome occur?
    • What unintended consequences appeared?
    • What should change in the product, its priorities or our understanding of quality?

    This is the second return of the Product Owner.

    But it is not simply the acceptance of a completed story. It may continue long after release, as evidence accumulates and outcomes become visible.

    A completed story is an engineering result. It becomes product learning only when it is interpreted in its wider context and influences what happens next.

    Closing the loop

    The complete model is not:

    Business request → backlog → engineering → done

    It is closer to:

    Business intent → product interpretation → engineering → evidence → release → operation → outcomes → learning → revised intent

    Upstream, Product Ownership translates business intent into coherent and assessable product expectations. Downstream, it connects engineering evidence, production experience and actual outcomes to product judgement and future decisions.

    Without the upstream connection, engineering receives requests rather than coherent product intentions. Without the downstream connection, delivered functionality becomes output without structured learning. The backlog may keep moving, but the product loop remains open.

    A quality-governance function

    Product Ownership does not own every quality activity. Engineering, architecture, operations, support, security, leadership and other stakeholders all contribute evidence and expertise.

    But Product Ownership has a distinctive responsibility for maintaining coherence between:

    • why the product exists;
    • what engineering is asked to build;
    • what evidence the organisation receives;
    • what the product achieves in reality;
    • what should be decided next.

    That makes Product Ownership part of quality governance.

    Quality begins before engineering, is shaped and assessed during engineering, and continues to be revealed after release. Product Ownership keeps those phases connected.

    It begins before the backlog and continues after implementation. Its purpose is not merely to keep engineering supplied with work.

    It is to close the loop between intent, delivery, evidence, outcomes and learning.

  • Field Note: The Delivery Process Is Not a Quality Model

    In several recent field notes, I have explored how quality begins before testing. Requirements may form part of the reference model against which a product is assessed. Acceptance criteria can make expectations explicit. Test cases often reveal details that were missing from the original product definition. Definition of Ready and Definition of Done can connect intention, implementation and assessment. Product Ownership has a much more important quality role than simply maintaining a backlog.

    Together, these ideas suggest a stronger quality sequence:

    1. The Product Owner defines the intention.
    2. The story makes it concrete.
    3. The team implements it.
    4. Testing provides evidence.
    5. The Definition of Done confirms completion.

    That would already be a significant improvement over treating quality as something testers add near the end. But I am beginning to think it is still not enough.

    A well-delivered story is not necessarily a good product

    A story can be clearly formulated. Its acceptance criteria can be precise. The implementation can be technically sound. The tests can pass. Every condition in the Definition of Done can be satisfied. And the product can still be poor.

    The story may solve the wrong problem. It may work in isolation but create inconsistency elsewhere. It may support an obvious scenario while failing across a complete user journey. It may introduce complexity that weakens maintainability. It may function at current scale but not at the scale required by future customers.

    This does not necessarily mean that the story, testing or Definition of Done failed. It may mean that we asked them to carry more of the quality model than they can support.

    Stories are delivery instruments

    Stories help us divide work into manageable pieces. They support conversation, prioritization, implementation and feedback. But users do not experience a collection of stories. They experience a complete product.

    Usability emerges across journeys. Performance appears under realistic load. Resilience becomes visible when dependencies fail. Maintainability is affected by accumulated design decisions. Security and data integrity cut across features and organisational boundaries. These qualities cannot always be assessed meaningfully within a single story.

    This creates an important distinction:

    Story-level acceptance is not the same as product-level assessment.

    The first asks whether an increment meets its expectations. The second asks whether the product as a whole is good enough for its purpose, stakeholders, risks and context. We need both.

    Definition of Done is not a definition of quality

    The Definition of Done is an important mechanism. It can establish shared expectations for code review, testing, documentation, deployment and other necessary activities. But it may be given more meaning than it can carry.

    A Definition of Done usually tells us whether the team has completed the expected work correctly. It does not necessarily tell us whether:

    • the product solves the right problem;
    • the complete user experience is coherent;
    • the architecture remains sustainable;
    • operational risks are acceptable;
    • the intended outcome is being achieved.

    A strong Definition of Done contributes evidence. It is not, by itself, a complete reference model for product quality.

    Quality extends from intent to outcome

    Perhaps the deeper mistake is that we have confused a delivery process with a quality model. The delivery process may look like this:

    Product Owner → story → implementation → testing → done

    A broader quality perspective looks more like this:

    Intent → product definition → implementation → integrated product → operation → outcomes → learning

    This wider view introduces questions at several levels:

    • Did we understand the problem and intended outcome?
    • Did we translate that understanding into clear and assessable expectations?
    • Did we build the solution correctly and sustainably?
    • Does the integrated product exhibit the quality characteristics required in realistic use?
    • Does it create the intended value after release?
    • What have we learned that should change the product—or our understanding of quality?

    Testing contributes to this system, but it does not contain the whole system. Product Ownership contributes to it, but one role cannot represent every relevant quality perspective. Stories contribute to it, but they cannot describe every emergent property of the complete product. The Definition of Done contributes to it, but process completion does not prove fitness for purpose.

    This is a leadership question

    Teams can improve stories, acceptance criteria, refinement, testing and their Definition of Done. But a holistic quality approach extends beyond the authority of a single team.

    It may require alignment between strategy, product management, engineering, architecture, operations, security, support and other stakeholders. It requires explicit choices about quality characteristics, risks, evidence and decision-making.

    Only leadership can create the conditions in which these perspectives form one coherent quality system. The leadership question is therefore not only:

    Are teams following the expected quality practices?

    It is also:

    Does the organisation have a sufficiently complete model of quality in the first place?

    A broader working hypothesis

    Quality begins before testing. But it also extends beyond testing, beyond individual stories and beyond the Definition of Done.

    The familiar delivery sequence remains valuable. It helps teams convert product intentions into tested increments. We should not abandon it. We should stop mistaking it for the complete quality model. A collection of accepted stories is not automatically an acceptable product.

    A collection of accepted stories is not automatically an acceptable product. Product quality cannot be secured at the end of delivery.

    It must be governed from intent to outcome.