Field Notes from Quality & Delivery Transformations

Blog

  • Blog Retrospective #2

    I completed the next twenty posts, bringing the total to forty. Since I agreed with myself to pause for a retrospective after every twenty posts, here is the second one. It feels as though I have completed an initial theoretical loop around product quality assessment.

    During this cycle, the investigation moved through several connected questions:

    • What distinguishes assessment from testing, QA and quality engineering?
    • What must exist before meaningful assessment is possible?
    • What belongs in a product reference model?
    • Who owns and maintains that model?
    • How should evidence support judgement?
    • How does assessment continue after deployment?

    The final field notes brought many of these questions together in Product Quality Assessment as a Continuous Discipline.

    What Surprised Me?

    The biggest surprise was the growing importance of the product reference model. It began as the reference against which a product could be assessed. It gradually became something closer to a shared description of what the product is intended to achieve, what good looks like, which constraints matter and what evidence is needed.

    A second surprise was the emergence of Product Ownership as the continuity mechanism connecting business intent, engineering interpretation, release judgement and operational learning.

    A third surprise was that the field notes began reinforcing one another. Ideas that appeared speculative in isolation became more credible when placed into a coherent chain.

    What Became Clearer?

    Several propositions now seem reasonably stable:

    • Quality is the integrity of the chain from business intent to realised outcome.
    • Testing can be understood as a form of assessment, but product quality assessment is broader than testing.
    • Meaningful assessment requires an explicit reference model.
    • The reference model must guide implementation as well as evaluation.
    • Assessability and evidence need to be designed in.
    • Product quality cannot be fully understood at release. Operational evidence must complete the judgement.

    These are still working propositions, but they increasingly appear to form a coherent whole.

    What Remains Unproven?

    The theory has not yet been validated as an operational model. Important questions remain:

    • Can a useful product reference model be maintained without becoming bureaucratic?
    • What is its minimum viable form?
    • Can product quality assessment become part of normal delivery rather than a separate governance process?
    • Does it lead to better findings, decisions and priorities in practice?

    The business-intent side also needs more development. Much of the thinking so far has focused on engineering, testing, evidence and reference models.

    What Comes Next?

    The next step is probably not another rapid expansion of the theory. I want to begin consolidating the existing thinking around one master theme:

    Quality from Business Intent to Outcome

    Supporting themes may include:

    1. Testing as Assessment
    2. The Product Reference Model
    3. Product Ownership and Quality Judgement
    4. Continuous Product Quality Assessment

    These should be treated as evolving synthesis pages rather than finished statements.

    For the field notes, I first want to propose a conceptual table of contents for the product reference model. After that, I expect to return to assessment itself: the assessment model, the relationship between evidence and judgement, and perhaps a few conceptual capability models.

    I also want to revisit a historical assessment and see whether the emerging approach produces better or simply different findings.

    Conclusion

    The first retrospective asked what was happening to the writing process. This retrospective asks what has happened to the subject itself.

    Product quality assessment began as an attempt to describe testing differently. It expanded into questions about reference models, Product Ownership, evidence, operational learning and governance.

    The result is not yet a proven method. But it is becoming a coherent discipline. The next step is to find out whether it can also become concrete, usable and valuable in practice.

  • Field Note: Product Quality Assessment as a Continuous Discipline

    Product quality is often assessed at particular moments:

    • A requirement is reviewed.
    • A feature is tested.
    • A release decision is made.
    • An incident is investigated.
    • Customer feedback is analyzed.

    Each activity may produce useful evidence. But when they remain disconnected, no one continuously maintains the overall basis for judging the product. If quality is the integrity of the chain from business intent to business outcome, product quality assessment cannot be reduced to a single test phase or release gate. It must become a continuous discipline.

    Assessment Begins With Intent

    Before a product can be assessed, there must be a basis for judgment. What is the product intended to achieve? For whom? Under which conditions? Which qualities matter? Which constraints must be respected? What evidence would indicate success or failure? Together, these expectations form the product reference model.

    Requirements and acceptance criteria may be part of that model, but they are not the whole of it. The model may also include intended outcomes, stakeholder needs, quality characteristics, risks, assumptions, operating conditions and evidence requirements.

    Without such a model, testing may still find defects. But it becomes much harder to judge whether the product as a whole is fit for its intended purpose.

    The Model Must Feed Development

    The reference model should not be created only so that someone can assess the finished product. It must also guide the people building it. Quality expectations have design implications. Performance expectations influence architecture. Auditability requires traceability. Resilience requires detection and recovery mechanisms. Business outcomes require ways to observe whether those outcomes occurred.

    The question is therefore not only whether an expectation can be tested later. It is whether the product is being designed to satisfy the expectation and produce the evidence needed to assess it. Assessment begins before implementation because the basis for later judgment influences what must be built.

    Evidence Accumulates Across the Lifecycle

    Testing provides essential evidence, but it provides only part of the picture. Reviews, analysis, demonstrations, security checks and performance tests may contribute evidence before release. Monitoring, incidents, support cases, user behavior and business results contribute evidence after release. Some qualities can be assessed with confidence before deployment. Others can only be fully understood in operation.

    Continuous product quality assessment brings these sources together rather than treating them as separate activities belonging to different functions. The purpose is not to gather as much evidence as possible. It is to gather the evidence needed to make responsible judgments about the product.

    Judgment Must Be Explicit

    Evidence does not make decisions by itself. Someone must interpret what the evidence means in relation to the reference model, the remaining uncertainty and the risks involved. Is the product ready to release? Are known limitations acceptable? Did the product create the intended outcome? Has new operational evidence changed what the organization believes about it?

    Product Ownership safeguards the coherence of the product reference model. Specialists contribute evidence and professional judgment. Leadership sets strategic direction, constraints and acceptable risk. Assessment integrates these perspectives into a product-quality judgment.

    The Cycle Does Not End at Release

    A release is not the conclusion of product quality assessment. It is the point at which new forms of evidence become available.

    Real use may reveal that assumptions were wrong. Customers may value different outcomes than expected. Operational conditions may expose risks that pre-release testing could not reproduce. A product may function correctly and still fail to produce the intended business result.

    This evidence should not only lead to fixes. It should update the product reference model itself. The continuous cycle becomes:

    intent → reference model → development → evidence → judgment → operation → learning → revised intent

    The product changes. Its context changes. What the organization knows about it changes. The basis for judging it must change as well.

    A Discipline, Not an Additional Phase

    Product quality assessment is not a new phase wrapped around testing. It is the continuous discipline of maintaining the basis for product-quality judgment from intent to outcome. Testing remains essential. QA and quality engineering remain valuable. Product Ownership, operations and leadership all retain their distinct responsibilities. But their contributions become more useful when they support one coherent and continuously updated judgment of the product.

    Quality is not established once. It is assessed, learned from and reassessed throughout the life of the product.

  • Field Note: From Testability to Assessability

    Testability is a familiar quality concern. A testable product makes it easier to create conditions, observe behavior and determine whether the product works as expected. Interfaces can be controlled. States can be inspected. Failures can be reproduced. Relevant outputs can be observed. Without testability, testing becomes slower, more expensive and less reliable.

    But if quality is understood as something broader than conformance to specified behavior, testability may not be enough. A product may be testable and still be difficult to assess.

    Testing Produces Only Part of the Evidence

    Testing can provide evidence about many important product qualities. It can show whether functionality behaves as expected, whether performance meets defined thresholds and whether known risks have been addressed before release.

    But some quality claims cannot be judged fully through testing alone:

    • Can users complete their actual work successfully?
    • Does the product create the intended business outcome?
    • Does it remain reliable under real operating conditions?
    • Can incidents be understood and recovered from?
    • Are decisions and changes traceable?
    • Do larger customers use the product differently from smaller ones?

    These questions require evidence from operation, users, support, monitoring and business results.

    The broader question is therefore not only:

    Can this product be tested?

    It is also:

    Can its important quality claims be assessed?

    Assessability Extends Testability

    Assessability is the product’s ability to provide the evidence needed for a meaningful quality judgment. Testability is part of this. So are observability, inspectability, traceability and measurability.

    A product is more assessable when we can understand:

    • what it is doing;
    • why it behaved as it did;
    • whether important conditions were met;
    • how users experienced it;
    • whether intended outcomes occurred;
    • what changed when results differed from expectations.

    This evidence does not appear automatically:

    • If auditability matters, relevant actions and decisions must be traceable.
    • If resilience matters, failures and recovery must be observable.
    • If performance matters, representative measurements must be possible.
    • If business outcomes matter, product behavior must be connected to suitable outcome measures.
    • If usability matters, there must be ways to observe real user success rather than only confirm that screens function.

    Evidence Has Design Implications

    intent → quality claim → required evidence → design implication → assessment

    This means that evidence should not always be considered after the product has been built. The ability to produce evidence may itself need to be designed into the product.

    The chain becomes:

    Suppose the product reference model states that a service must recover reliably after a failure. That claim implies evidence about detection, interruption, recovery time and data integrity. Producing that evidence may require monitoring, logging, controlled failure mechanisms and clear state information. Without those capabilities, the expectation may remain valid but difficult to judge.

    The same applies beyond technical qualities. If the intended outcome cannot be measured or observed, the organization may never know whether the product delivered the value that justified building it.

    Quality Must Be Assessable by Design

    Testability asks whether evidence can be produced through testing. Assessability asks whether sufficient evidence can be produced to judge the product throughout its lifecycle. That includes evidence before release, but also evidence from real use and operation. A product that cannot produce evidence about its important qualities cannot be assessed with confidence.

    Designing for assessability does not mean instrumenting everything or measuring every possible outcome. It means beginning with the important claims in the product reference model and asking:

    What would we need to know in order to judge this?

    Sometimes the answer will be a test. Sometimes it will be a log, a metric, an audit trail, an operational signal or feedback from users.

    Testability helps us determine whether the product can be tested. Assessability helps us determine whether the product can be understood.

  • Field Note: Who Owns the Product Reference Model?

    The idea of a product reference model creates an immediate question:

    Who owns it?

    If the model describes what the product is intended to achieve, the conditions under which it must operate, the quality expected of it and the evidence needed to judge it, ownership cannot remain implicit. Many people may contribute to the model. Someone must still own whether it forms a coherent whole. That responsibility naturally belongs to Product Ownership.

    Ownership Does Not Mean Writing Everything

    A Product Owner cannot personally define every part of the product reference model.

    Architects contribute technical constraints. Security specialists contribute controls and risks. Developers contribute feasibility and implementation knowledge. Testers contribute evidence needs and ambiguity detection. Operations contributes knowledge of production reality. Customers and users contribute expectations and desired outcomes.

    Leadership also contributes by setting strategic direction, investment boundaries, customer commitments and acceptable levels of risk. All of these inputs may be valid. But a collection of valid inputs is not yet a coherent product model. Someone must ensure that they fit together.

    Leadership Sets the Boundaries

    Leadership defines the space in which the product is expected to operate. It determines strategic direction, business outcomes, investment limits, policy constraints and acceptable risk.

    These decisions shape the product, but they do not automatically translate into something that can guide development and assessment. That translation belongs closer to the product. Leadership defines the boundaries. Product Ownership determines what the product must become within those boundaries.

    Leadership also has to provide the conditions needed to meet its expectations. A product cannot be expected to be secure, resilient, scalable and compliant without sufficient time, competence, authority and investment. Constraints without enabling conditions become governance theatre.

    Product Ownership Owns Integrity

    The Product Owner does not need to be the greatest expert in every dimension of the product. The Product Owner does need to ensure that the different dimensions form one coherent whole.

    That means connecting business objectives to product outcomes, making constraints visible, resolving important trade-offs, recognizing risks and ensuring that quality expectations and evidence needs are explicit. The central responsibility is therefore not authorship. It is integrity. The Product Owner owns the integrity of the product reference model.

    Specialists remain responsible for the validity of their contributions. The architect must provide credible architectural guidance. The security specialist must provide credible security expertise. The tester must identify meaningful evidence. The developer must explain technical implications.

    But those contributions should not remain separate professional perspectives. They must still describe one intended product.

    The Product Owner as Custodian

    The product reference model is not finished when the initial requirements are written. It changes as the organization learns. Operational incidents may reveal false assumptions. Larger customers may expose scalability limitations. Regulation may change. User behavior may challenge expectations. Business priorities may shift.

    Someone must ensure that the model changes with the product and its context. This is why the Product Owner is best understood as the custodian of the product reference model. A custodian does not create every element. A custodian ensures that the whole remains coherent, current and useful.

    Leadership defines direction, constraints and acceptable risk. Specialists ensure that their contributions are credible. Product Ownership ensures that it all still describes one product.

  • Field Note: A Product Reference Model Must Feed Development

    A reference model can easily become something we use after the fact. We define what good looks like, compare reality against it and identify gaps. That is useful for assessment.

    But if the model only becomes relevant when the product is being evaluated, it arrives too late. The same understanding that supports judgement should also support creation.

    A product reference model must not only help us assess the product. It must also help us build it.

    The danger of an assessment-only model

    Suppose the product reference model says that the product must be scalable, auditable, secure, configurable and easy to integrate. If those expectations are only checked near release, they function as acceptance criteria for work that has already been designed and implemented.

    By then, the architecture may be difficult to change. The wrong assumptions may already be embedded in the solution. The team may discover that important evidence cannot be produced because observability, testability or traceability were never designed in.

    A late reference model can reveal gaps. It cannot prevent them.

    The model should influence earlier decisions

    If the reference model expresses the intended product, then it should influence:

    • product strategy;
    • roadmap decisions;
    • architecture;
    • requirements and stories;
    • acceptance criteria;
    • quality scenarios;
    • test design;
    • observability;
    • release preparation;
    • operational feedback.

    It should help people ask better questions before implementation begins.

    Which users and stakeholders are affected?

    Which qualities are important in this context?

    Which assumptions are we making?

    What could make the product unsuitable even if the feature works?

    What evidence will we need later to judge whether the intended outcome has been achieved?

    These are not only assessment questions. They are development questions.

    Requirements are translations of the model

    Requirements, stories and acceptance criteria remain necessary. But they should be understood as selected translations of the wider product reference model into engineering work. They do not contain the complete product understanding. A story may describe a behaviour.

    The reference model provides the reason, context, quality expectations, constraints, risks and intended outcome behind it. That wider understanding helps engineering make better decisions when the requirement is incomplete or when several technically valid solutions exist. It also reduces the risk that testers have to reconstruct missing product intent after implementation.

    Evidence should be designed in

    If the organisation knows what it will need to assess, it can design the necessary evidence into the product and delivery system. Performance expectations can shape load testing and monitoring. Reliability expectations can shape resilience engineering and incident measures. Usability expectations can shape research and usage analytics. Business outcomes can shape product instrumentation.

    The reference model therefore influences not only what is built, but how the organisation will learn whether it works.

    A shared object of reasoning

    The model should not belong exclusively to Product Ownership, architecture, QA or testing. It should become a shared object through which business and engineering reason about the same product.

    Product Ownership helps maintain the connection to intent and outcomes. Engineering uses the model to guide implementation and technical trade-offs. Testing and assessment use it to determine what evidence is needed. Operations and support use real-world experience to challenge and update it.

    The model moves in both directions: intent guides development
    and operational evidence changes future intent.

    More than a ruler

    A ruler tells us whether something meets a standard after it has been produced. A useful product reference model should do more. It should guide decisions while the product is being shaped, help generate the evidence needed for assessment and evolve when reality challenges earlier assumptions.

    A product reference model that only measures the product is incomplete. Its real value appears when it helps the organisation create, assess and evolve the right product.