Blog

Systems fail when continuity is broken. When we think about continuity, we usually think in technical terms: backups, version control, redundancy or disaster recovery. Those preserve systems.

But there is another kind of continuity that is just as important:

The continuity of understanding.

A few years ago, after my father passed away, I inherited many of his belongings. Some objects immediately meant something to me. His field glasses, for example, reminded me of my childhood. They carried a story, and because of that story they retained their meaning.

Many other objects did not. They may have been just as important to him, but I simply didn’t know why. Without their stories, they became little more than anonymous possessions. The objects survived. Their context did not.

Software systems suffer from exactly the same problem. Production systems often survive for decades. The source code is still there. The database still exists. Diagrams, tickets and documentation may still be available. Yet the people who understood why the system was designed that way have long since moved on.

A peculiar integration remains, but nobody remembers which dependency required it. A workaround survives, but the incident that justified it has been forgotten. A business rule is still enforced, but nobody knows why it was introduced. The system survives. Its context does not.

The next generation inherits the equivalent of a room full of possessions: many artifacts, some clearly important, others seemingly meaningless, with very little explanation of the stories that connect them.

Without those stories, maintenance becomes archaeology. Changes become risky because nobody can distinguish deliberate design from historical accident.

That experience has changed the way I think about documentation. Documentation does not preserve knowledge. At best, it preserves information. Knowledge survives only when enough context is preserved for future people to reconstruct the original meaning.

Knowledge is not preserved by storing documents. Knowledge is preserved by maintaining enough context that future people can reconstruct the original meaning.

That context rarely consists of technical details alone. Sometimes it is the problem that triggered an idea. Sometimes it is the trade-off that was consciously accepted. Sometimes it is a story. Sometimes it is simply the answer to a single question:

Why?

Perhaps that is the real purpose of documentation. Not to describe every detail. But to preserve enough understanding that the next person can continue the story instead of starting over.

We often think that an object’s identity is determined solely by the object itself. It isn’t. The boundary around an object profoundly influences how we perceive the object within it.

A photograph does not change when you replace its frame. Yet a black frame can make the colors appear richer, while a wide white mat can make the same photograph feel more spacious or more like a piece of art. A television does not become larger when Philips Ambilight projects light onto the wall. Yet the image appears to extend beyond the physical screen because the boundary of the display has changed.

The same principle appears in industrial design. While experimenting with different straps on a Victorinox I.N.O.X. wristwatch, I noticed something unexpected. The watch itself—the steel case, dial and movement—remained exactly the same. Only the strap changed.

On the steel bracelet, the watch is perceived as one continuous piece of engineering—a substantial object made of steel. The steel bracelet and the case compete for attention, together forming a single architectural object. Replacing the bracelet with a black leather strap completely changes the visual hierarchy. The leather quietly recedes into the background, allowing the eye to focus almost entirely on the distinctive hexagonal steel case. The watch has not changed. Yet it feels like a different object. The boundary determines what our attention considers to be the object.

Music demonstrates the same principle in time rather than space. A skilled musician knows that silence is not the absence of music—it is part of the composition. Bass players, for example, will sometimes stop playing briefly to create tension. The temporary void changes how the audience experiences what follows. When the bass returns, listeners instinctively experience a sense of release. The notes have not become better. The silence has changed how they are perceived. In music, silence is the boundary that defines the notes.

Good design often understands this instinctively. It is not only the product that is designed; it is also the context in which the product is experienced. Architecture uses pedestals, lighting and empty space to make sculptures more powerful. Graphic design uses margins and whitespace to make information easier to understand.

Perhaps this is a more general design principle:

The boundary does not merely surround the object. It defines the object.

When trying to improve something, we often redesign the object itself. Sometimes the better question is:

Can I change the boundary instead?

If this principle applies equally to photography, architecture, music, industrial design and product design…

…what might it mean for software?

Product quality is often assessed at particular moments:

• A requirement is reviewed.
• A feature is tested.
• A release decision is made.
• An incident is investigated.
• Customer feedback is analyzed.

Each activity may produce useful evidence. But when they remain disconnected, no one continuously maintains the overall basis for judging the product. If quality is the integrity of the chain from business intent to business outcome, product quality assessment cannot be reduced to a single test phase or release gate. It must become a continuous discipline.

Assessment Begins With Intent

Before a product can be assessed, there must be a basis for judgment. What is the product intended to achieve? For whom? Under which conditions? Which qualities matter? Which constraints must be respected? What evidence would indicate success or failure? Together, these expectations form the product reference model.

Requirements and acceptance criteria may be part of that model, but they are not the whole of it. The model may also include intended outcomes, stakeholder needs, quality characteristics, risks, assumptions, operating conditions and evidence requirements.

Without such a model, testing may still find defects. But it becomes much harder to judge whether the product as a whole is fit for its intended purpose.

The Model Must Feed Development

The reference model should not be created only so that someone can assess the finished product. It must also guide the people building it. Quality expectations have design implications. Performance expectations influence architecture. Auditability requires traceability. Resilience requires detection and recovery mechanisms. Business outcomes require ways to observe whether those outcomes occurred.

The question is therefore not only whether an expectation can be tested later. It is whether the product is being designed to satisfy the expectation and produce the evidence needed to assess it. Assessment begins before implementation because the basis for later judgment influences what must be built.

Evidence Accumulates Across the Lifecycle

Testing provides essential evidence, but it provides only part of the picture. Reviews, analysis, demonstrations, security checks and performance tests may contribute evidence before release. Monitoring, incidents, support cases, user behavior and business results contribute evidence after release. Some qualities can be assessed with confidence before deployment. Others can only be fully understood in operation.

Continuous product quality assessment brings these sources together rather than treating them as separate activities belonging to different functions. The purpose is not to gather as much evidence as possible. It is to gather the evidence needed to make responsible judgments about the product.

Judgment Must Be Explicit

Evidence does not make decisions by itself. Someone must interpret what the evidence means in relation to the reference model, the remaining uncertainty and the risks involved. Is the product ready to release? Are known limitations acceptable? Did the product create the intended outcome? Has new operational evidence changed what the organization believes about it?

Product Ownership safeguards the coherence of the product reference model. Specialists contribute evidence and professional judgment. Leadership sets strategic direction, constraints and acceptable risk. Assessment integrates these perspectives into a product-quality judgment.

The Cycle Does Not End at Release

A release is not the conclusion of product quality assessment. It is the point at which new forms of evidence become available.

Real use may reveal that assumptions were wrong. Customers may value different outcomes than expected. Operational conditions may expose risks that pre-release testing could not reproduce. A product may function correctly and still fail to produce the intended business result.

This evidence should not only lead to fixes. It should update the product reference model itself. The continuous cycle becomes:

intent → reference model → development → evidence → judgment → operation → learning → revised intent

The product changes. Its context changes. What the organization knows about it changes. The basis for judging it must change as well.

A Discipline, Not an Additional Phase

Product quality assessment is not a new phase wrapped around testing. It is the continuous discipline of maintaining the basis for product-quality judgment from intent to outcome. Testing remains essential. QA and quality engineering remain valuable. Product Ownership, operations and leadership all retain their distinct responsibilities. But their contributions become more useful when they support one coherent and continuously updated judgment of the product.

Quality is not established once. It is assessed, learned from and reassessed throughout the life of the product.

Testability is a familiar quality concern. A testable product makes it easier to create conditions, observe behavior and determine whether the product works as expected. Interfaces can be controlled. States can be inspected. Failures can be reproduced. Relevant outputs can be observed. Without testability, testing becomes slower, more expensive and less reliable.

But if quality is understood as something broader than conformance to specified behavior, testability may not be enough. A product may be testable and still be difficult to assess.

Testing Produces Only Part of the Evidence

Testing can provide evidence about many important product qualities. It can show whether functionality behaves as expected, whether performance meets defined thresholds and whether known risks have been addressed before release.

But some quality claims cannot be judged fully through testing alone:

• Can users complete their actual work successfully?
• Does the product create the intended business outcome?
• Does it remain reliable under real operating conditions?
• Can incidents be understood and recovered from?
• Are decisions and changes traceable?
• Do larger customers use the product differently from smaller ones?

These questions require evidence from operation, users, support, monitoring and business results.

The broader question is therefore not only:

Can this product be tested?

It is also:

Can its important quality claims be assessed?

Assessability Extends Testability

Assessability is the product’s ability to provide the evidence needed for a meaningful quality judgment. Testability is part of this. So are observability, inspectability, traceability and measurability.

A product is more assessable when we can understand:

• what it is doing;
• why it behaved as it did;
• whether important conditions were met;
• how users experienced it;
• whether intended outcomes occurred;
• what changed when results differed from expectations.

This evidence does not appear automatically:

• If auditability matters, relevant actions and decisions must be traceable.
• If resilience matters, failures and recovery must be observable.
• If performance matters, representative measurements must be possible.
• If business outcomes matter, product behavior must be connected to suitable outcome measures.
• If usability matters, there must be ways to observe real user success rather than only confirm that screens function.

Evidence Has Design Implications

intent → quality claim → required evidence → design implication → assessment

This means that evidence should not always be considered after the product has been built. The ability to produce evidence may itself need to be designed into the product.

The chain becomes:

Suppose the product reference model states that a service must recover reliably after a failure. That claim implies evidence about detection, interruption, recovery time and data integrity. Producing that evidence may require monitoring, logging, controlled failure mechanisms and clear state information. Without those capabilities, the expectation may remain valid but difficult to judge.

The same applies beyond technical qualities. If the intended outcome cannot be measured or observed, the organization may never know whether the product delivered the value that justified building it.

Quality Must Be Assessable by Design

Testability asks whether evidence can be produced through testing. Assessability asks whether sufficient evidence can be produced to judge the product throughout its lifecycle. That includes evidence before release, but also evidence from real use and operation. A product that cannot produce evidence about its important qualities cannot be assessed with confidence.

Designing for assessability does not mean instrumenting everything or measuring every possible outcome. It means beginning with the important claims in the product reference model and asking:

What would we need to know in order to judge this?

Sometimes the answer will be a test. Sometimes it will be a log, a metric, an audit trail, an operational signal or feedback from users.

Testability helps us determine whether the product can be tested. Assessability helps us determine whether the product can be understood.