Field Notes from Quality & Delivery Transformations

Blog

  • Field Note: An Assessment Is a Snapshot. Improvement Requires Reassessment.

    I have usually thought of an assessment as a one-time intervention. We define a reference model, collect evidence, judge the current state and identify improvements. The result is a snapshot. That is useful for diagnosis.

    But it is not enough when the purpose is improvement. Once change begins, the subject of the assessment starts to move. Practices change. Responsibilities shift. New evidence appears. Some improvements work, others do not. New weaknesses become visible. The original assessment gradually becomes a description of a state that no longer exists.

    An assessment may be a snapshot. Assessment should be a recurring capability.

    The first assessment establishes a baseline

    A thorough initial assessment remains valuable. It clarifies:

    • what is being assessed;
    • what good looks like;
    • the current state;
    • important strengths, weaknesses and risks;
    • priorities for improvement.

    Without that baseline, improvement can become a collection of disconnected actions. But the baseline is only the beginning.

    Improvement creates new evidence

    Every improvement action is also a hypothesis. We expect that changing a process, role, tool, structure or behaviour will lead to a better result.

    That expectation should be tested. Did the change actually happen? Did behaviour improve, or only documentation? Did the intended result appear? Did the improvement create new problems elsewhere?

    Without reassessment, these questions are often answered through impressions or completion reports.

    But completing an improvement is not the same as achieving improvement. Reassessment brings the organisation back to evidence.

    Reassessment does not mean repeating everything

    Continuous assessment does not require a full formal assessment every sprint or every month. That would quickly become bureaucratic. Reassessment can happen at different depths. A lightweight review may examine recent evidence and a few critical indicators. A periodic deeper assessment may revisit the complete reference model. A major change, failure or shift in context may trigger an earlier reassessment.

    The principle is simple:

    Reassess often enough to learn whether improvement is actually happening.

    The reference model may also change

    Reassessment is not merely updating a score against a fixed model. As understanding improves, the definition of good may also evolve. Some expectations may prove irrelevant. New capabilities may become important. The organisation may discover that its original target state was incomplete or misguided.

    Reassessment should therefore ask two questions:

    Are we moving closer to what we defined as good?

    And:

    Is our definition of good still the right one?

    From report to improvement loop

    A static assessment often ends with findings and recommendations.

    A continuous assessment process closes the loop:

    reference model

    → evidence

    → judgement

    → decision

    → improvement

    → new evidence

    → reassessment

    The assessment should therefore identify not only what needs to improve, but also:

    • what evidence will show progress;
    • when reassessment will occur;
    • what should trigger an earlier review;
    • which assumptions remain uncertain.

    Assessment then becomes part of the improvement system rather than an event before improvement begins.

    The real purpose of reassessment

    The purpose is not to prove that the plan was followed. It is to discover whether the plan was right. Reassessment helps the organisation detect ineffective improvements early, adapt priorities, expose unintended consequences and remain focused on outcomes rather than activity.

    An assessment tells us where we are. Reassessment tells us whether we are actually moving—and whether we are still moving in the right direction.

  • Field Note: Quality Is the Integrity of the Chain

    We usually discuss software quality somewhere in the middle of the product lifecycle. Requirements have been defined. Work has entered the backlog. Engineering has started. Testing, Quality Assurance and Quality Engineering help ensure that the software is built and delivered well.

    But the product’s quality story began earlier. It began when the organisation formed an intention: solve a problem, serve a customer, enter a market, reduce a cost or create a business outcome.

    And the story does not end when the product is released. It ends—if it ends at all—when we can understand whether the product produces the outcome for which it was created.

    This suggests a broader definition:

    Quality is the integrity of the chain from business intent to business outcome.

    The chain

    The chain may be expressed simply:

    Business intent

    → product decisions

    → engineering delivery

    → operational reality

    → business outcome

    Each link translates something into something else.

    Business ambition is translated into a product direction. Product direction is translated into capabilities, priorities and quality expectations. These are translated into engineering decisions and working software. The software enters an operational environment where customers, users, integrations, data and real workloads affect its behaviour. That behaviour contributes—or fails to contribute—to the intended outcome. Quality depends on the integrity of all these translations.

    How the chain breaks

    A product may fail because the original intent was unclear. It may fail because the organisation misunderstood the customer need or selected the wrong route to its ambition. It may fail because the right intent was translated into the wrong product. It may fail because engineering realised the product poorly. It may function as designed but prove unreliable, difficult to use, expensive to operate or impossible to scale. It may operate successfully and still fail to produce adoption, efficiency, customer value or commercial results.

    These failures look different, but they share something important:

    Somewhere between intent and outcome, the chain lost integrity.

    This expands the meaning of a quality failure. A defect is a quality failure. But so is building a technically excellent capability that customers do not need. So is improving performance when the real obstacle to enterprise adoption is permissions, auditability or integration. So is meeting every acceptance criterion while failing to create the intended outcome.

    Different disciplines protect different parts

    Testing, Quality Control, Quality Assurance and Quality Engineering remain essential. But they do not protect the complete chain. Testing guards against not knowing how the product behaves. Quality Control guards against accepting outputs that do not conform to defined expectations. Quality Assurance guards against relying on processes and controls that cannot consistently produce dependable results. Quality Engineering guards against treating quality as something that can be inspected into the product at the end. Each protects an important part of the chain.

    Product-quality assessment asks a more complete question:

    Does the available evidence support the judgement that the product is right, good enough for its intended context and capable of producing its intended outcome?

    It therefore guards against false confidence. Tests may pass. Requirements may be met. Delivery metrics may be healthy. The system may operate as designed. The organisation may still be wrong about the product.

    The role of Product Ownership

    Someone must preserve continuity of intent as the product moves through the chain. That does not mean one person owns every quality decision. But Product Ownership has a central responsibility for maintaining coherence between:

    • the business ambition;
    • customer and stakeholder needs;
    • product decisions;
    • quality expectations;
    • engineering work;
    • operational evidence;
    • product learning.

    Product Ownership should not merely feed work into engineering. It should help ensure that what emerges from engineering can still be traced to the purpose that justified the investment—and that evidence from operation changes future product decisions.

    The role of governance

    Product-quality assessment creates a basis for judgement. Governance determines what happens next. Leadership must decide:

    • which outcomes and risks matter;
    • what evidence is sufficient;
    • which trade-offs are acceptable;
    • who may accept remaining risk;
    • whether to release, invest, redirect or stop.

    This is why the complete view of quality cannot be delegated to testing or contained within delivery. It reaches into product strategy, investment and accountability.

    The economic meaning of integrity

    When the chain lacks integrity, the organisation pays in several ways. It pays the cost of being wrong when it invests in the wrong capability, assumption or market response. It pays the cost of failure through defects, incidents, disruption and lost trust. It pays the cost of operation through support demand, workarounds and recurring interventions. It pays the cost of future change when complexity, weak maintainability and unclear intent make every next improvement slower and more expensive.

    When the chain retains its integrity, the organisation increases the likelihood that engineering effort becomes product value and that product value becomes business outcome. That may be the most complete business definition of quality:

    Quality is the integrity of the chain from business intent to business outcome.

    Everything else helps us understand where that chain can break, what evidence reveals its condition and how the organisation should respond.

  • Field Note: Product Quality Can Only Be Fully Understood in Operation

    Product quality must be assessed before release. Testing, reviews and demonstrations help us decide whether the product is ready.

    But that judgement is always provisional. Before release, we assess the product under selected conditions. After release, reality adds evidence we could not fully create in advance.

    Pre-release evidence is necessary, but incomplete

    Testing can show whether expected behaviour is present, whether important risks have been explored and whether quality characteristics such as performance, security or resilience meet defined expectations.

    But testing always uses models. We select scenarios, create representative data, simulate load and define expected outcomes.

    All of this is essential. But it cannot reproduce the full complexity of real use. Actual users behave differently from test users. Real data contains unexpected patterns. Dependencies fail in unfamiliar combinations. Support demand reveals confusion that no acceptance test exposed. A product may pass every planned assessment and still teach us something important after release.

    Operation produces a different kind of evidence

    Once the product is in use, new evidence appears:

    • incidents and escaped defects;
    • helpdesk calls and customer complaints;
    • performance under real load;
    • usage and abandonment patterns;
    • manual workarounds;
    • operational effort;
    • customer and business outcomes.

    This evidence does not merely tell us whether engineering made mistakes. It tells us how the product behaves in its actual context. A repeated support question may reveal weak usability. A production incident may expose an architectural assumption that was never explicit. Low adoption may show that a correctly implemented feature does not solve the intended problem. High operational effort may reveal maintainability or supportability problems. Reality makes the product more understandable.

    The team’s responsibility continues after release

    In many agile and DevOps environments, the development team does not hand the product over and disappear. The same people who design, build and test the system also help operate, support, maintain and improve it. That creates a direct quality-learning loop. Incidents reveal weaknesses in design, testing or operational preparation. Difficult fixes expose maintainability problems. Repeated manual interventions show where operability or automation is insufficient. Production behaviour tests assumptions made during engineering.

    This evidence should matter deeply to the team. It is not merely reporting about a product they once delivered. It is feedback on the quality of their own engineering decisions. The team that creates the product should also learn from how that product behaves in reality. Done may end the implementation cycle. It should not end the team’s interest in quality.

    Monitoring is not yet assessment

    Production systems generate dashboards, logs, alerts and incident records. But observations do not interpret themselves. A green dashboard does not prove that the product is good. A metric may remain within its threshold while users still struggle.

    Operational evidence must be interpreted against:

    • intended outcomes;
    • stakeholder needs;
    • quality expectations;
    • acceptable risks;
    • operating context.

    Monitoring produces evidence. Assessment turns that evidence into a quality judgement.

    The reference model must survive release

    A product-quality reference model cannot be used only before delivery. It must remain relevant in operation. The expectations defined earlier should help us interpret what happens in reality.

    At the same time, operational evidence should be able to change the reference model. An incident may reveal a missing quality characteristic. Customer behavior may challenge an assumption. A support pattern may expose a need that was never understood. A change in scale may make yesterday’s acceptable quality level inadequate today.

    The reference model must therefore be living. It provides the basis for assessment, but it also evolves through assessment.

    Quality learning follows the lifecycle

    The Definition of Done marks an important engineering boundary. Release marks another. Neither marks the end of quality. Before release, we ask whether the available evidence gives us enough confidence to proceed. After release, we ask what reality reveals about the product in actual use. These are different stages of the same quality-learning process.

    Product quality must be assessed before release, but it can only be fully understood in operation. A mature team prepares the product carefully before release—and remains willing to learn what reality reveals afterwards.

  • Field Note: From Shared Responsibility to Shared Quality Judgment

    Agile approaches have argued for years that quality is a shared responsibility. This was an important correction to the idea that quality belongs to testers at the end of delivery. Product Owners, analysts, architects, developers, testers and specialists all contribute to the quality of the product.

    But shared responsibility does not automatically create shared understanding. Different roles may all take quality seriously while meaning different things by it:

    • The Product Owner may focus on value and customer outcomes.
    • The architect may focus on resilience, security and maintainability.
    • Developers may focus on technical sustainability.
    • Testers may focus on risks, behaviour and evidence.
    • Operations may focus on stability and recoverability.
    • Users may focus on whether the product actually helps them accomplish their work.

    None of these perspectives is wrong. Product quality is multidimensional.

    The problem is that the perspectives often remain fragmented across separate artefacts, conversations and professional practices. They may only be brought together when a release decision must be made—or when something fails in production. Everyone may be responsible for quality while nobody has a coherent view of the whole.

    Shared responsibility is not shared judgment

    Responsibility describes who contributes to the work. Judgment asks a different question:

    Does the combined evidence support the conclusion that the product is good enough for its intended purpose?

    That conclusion cannot be reached by adding together completed quality activities. Code review, automated testing, security analysis, usability research and operational monitoring may all be valuable. But their meaning depends on the quality needs, risks and context of the product.

    The organisation therefore needs more than shared participation. It needs a shared basis for reasoning about quality.

    A shared object of reasoning

    A product-quality reference model could provide that basis.

    It would not assign all quality responsibility to one role. Instead, it would describe the quality space within which different roles contribute. It might make explicit:

    • which quality characteristics matter;
    • why they matter in this product context;
    • which stakeholders and scenarios are affected;
    • what risks arise when they are weak;
    • what evidence is relevant;
    • how adequacy should be judged.

    The reference model would allow different professional perspectives to remain distinct while still contributing to one product-quality judgment:

    • Architecture decisions could be connected to resilience or scalability needs.
    • Engineering practices could be connected to the evidence they produce.
    • Operational incidents could be connected to assumptions about reliability or usability.
    • Customer feedback could be connected to intended outcomes.

    The different roles would no longer contribute only to their own documents and controls. They would contribute to a shared quality profile of the product.

    This may be one of the most important functions of a reference model:

    It makes quality a shared object of reasoning.

    Shared does not mean undifferentiated

    There is a danger in saying that quality belongs to everyone. When everyone is responsible in the same vague way, important accountabilities may disappear.

    Shared quality work still requires different contributions:

    • Product Ownership maintains the connection between product intent, priorities and quality trade-offs.
    • Architects explain how design decisions support—or endanger—important quality characteristics.
    • Developers create the product and much of the technical evidence.
    • Specialists contribute depth in areas such as security, usability, data and operations.
    • Quality professionals contribute assessment expertise: challenging the completeness of the model, identifying relevant evidence, exposing uncertainty and supporting a defensible judgment.
    • Business decision-makers remain accountable for accepting residual risks associated with release, rollout or continued investment.

    These responsibilities are related, but they are not interchangeable. Shared responsibility should not erase differentiated accountability.

    From coordinated activity to coordinated reasoning

    Scrum helps make quality part of the work of the whole team. Scaled approaches add mechanisms for coordinating quality across teams, architectures and integrated solutions.

    But coordinated delivery does not necessarily produce a coherent product-quality judgment. Teams may satisfy their own acceptance criteria and Definitions of Done while system-level concerns remain unresolved.

    A product may be locally complete but still lack sufficient resilience, accessibility, operability or end-to-end coherence.

    A reference model could help distinguish:

    • what can be assessed within one team or backlog item;
    • what requires integrated product evidence;
    • what can only be judged in operation;
    • who must contribute to the final decision.

    The result would be more than coordinated quality activity. It would be coordinated quality reasoning.

    A further step for agile quality

    Agile methods helped move quality away from a separate downstream phase. They made quality part of creating the product.

    A product-quality reference model may enable the next step: making explicit how different perspectives, evidence and accountabilities combine into a credible judgment of fitness for purpose.

    The proposition is therefore not only:

    Everyone is responsible for quality.

    It is:

    Shared responsibility for quality requires a shared model of what quality means, differentiated accountability for contributing to it, and a shared basis for judging whether it has been achieved.

    That is the shift from shared responsibility to shared quality judgment.

  • Field Note: Product Ownership Has a Business Side. Where Are Its Tools?

    Product Ownership carries broad business accountability.

    The Product Owner is expected to understand customers, engage stakeholders, maximize value, establish product goals, make trade-offs, order the backlog and inspect outcomes. These responsibilities sound substantial.

    But when we look for practical tools to perform them, the picture becomes less clear. “Understand the customer” is not a customer-analysis technique. “Manage stakeholders” is not a method for reconciling conflicting interests. “Maximize value” does not explain how value should be defined, compared or assessed. “Inspect and adapt” does not tell us which evidence matters or how it should influence product direction.

    The Product Owner is told what to think about. But where are the tools for doing the work?

    Engineering has a visible operating model

    Scrum gives engineering delivery a clear structure:

    Product Backlog → Sprint → Increment → inspection and adaptation

    The roles, events, artifacts and commitments make the work visible. A team can discuss whether backlog items are ready, whether the Sprint Goal is clear, whether the Increment is Done and what should change next. The Product Owner appears at the beginning of this flow through the backlog.

    But the backlog is not the beginning of Product Ownership. Before something becomes a story, a much broader translation should already have taken place. Someone must connect:

    • business and product strategy;
    • customer and stakeholder needs;
    • intended outcomes;
    • assumptions and risks;
    • quality expectations;
    • business and operational constraints;
    • dependencies and wider product context;

    to something engineering can understand and act upon. The story is the visible engineering-facing output. The reasoning that should produce it remains largely hidden.

    Scrum hides the business perspective behind the Product Owner

    Scrum gives Product Ownership broad accountability but only a thin formal operating structure outside backlog and delivery. Advanced Product Owner training adds useful stances, exercises and techniques, but these do not necessarily form an explicit lifecycle model for translating business intent into engineering work and operational evidence back into business learning.

    SAFe provides more—but spreads it out

    SAFe makes more of the business perspective visible. It introduces concepts such as Product Management, customer centricity, design thinking, roadmaps, epics, features, non-functional requirements, Lean Business Cases and outcome hypotheses. That is more operational support than Scrum provides on its own.

    But the broader Product Ownership capability is also distributed across many roles, levels and artifacts. Strategy may sit with portfolio leadership. Product Management may own vision and features. Product Owners manage team backlogs. Business Owners represent broader outcomes. Architects address technical direction. Operations and support hold important evidence after release. The pieces exist.

    What is harder to see is the model that keeps them coherent. Who maintains the complete connection between:

    • why the product exists;
    • who it serves;
    • what quality means;
    • which assumptions and risks matter;
    • what engineering is asked to build;
    • what evidence the organisation receives;
    • what production teaches us;
    • what should change next?

    Scrum hides much of the business perspective behind one Product Owner. SAFe exposes more of it, but risks fragmenting it across the organisation.

    Why Product Owners are pulled downstream

    The most practical tools and recurring activities available to Product Owners are usually engineering-facing:

    • backlog refinement;
    • story clarification;
    • prioritisation;
    • Sprint Planning;
    • answering team questions;
    • accepting completed work.

    These activities are concrete, visible and urgent. They are also supported by the delivery rhythm.

    The upstream work is less structured. So is the work after Done and release. How should a Product Owner translate strategy into a coherent product perspective? How should production incidents, support calls, customer behavior and outcomes be interpreted? How should that learning change product priorities and future engineering work?

    Without equally practical tools for these responsibilities, the role is naturally pulled towards the part of the system where the work is clearest. Product Owners may not choose to become backlog administrators. The operating model pulls them there.

    The backlog is not enough

    The backlog is an essential engineering-facing artifact. It organizes and prioritizes work. It supports refinement and delivery. But it cannot carry the complete business perspective of the product. It is not designed to preserve all relevant knowledge about purpose, stakeholders, outcomes, quality, risks, context, evidence and learning. Yet in many organisations, it is the primary practical instrument provided to the Product Owner.

    Product Ownership will remain centred on backlog administration as long as the backlog is the only operational tool we provide.

    In search of a broader model

    Perhaps Product Ownership needs a more explicit way to maintain the product perspective across the lifecycle.

    Something that helps connect:

    business intent → product definition → engineering work → delivery evidence → production experience → outcomes → learning

    It may need to preserve intended outcomes, stakeholders, quality expectations, assumptions, risks, constraints, operating context and required evidence. Perhaps this could eventually take the form of a product reference model.

    We do not yet know exactly what that model should contain or how it should work. But the gap is becoming visible. Product Ownership has a business side. It carries broad accountability for that business perspective.

    What it still lacks are the practical tools to keep that perspective coherent before, through and after engineering.