Product quality must be assessed before release. Testing, reviews and demonstrations help us decide whether the product is ready.
But that judgement is always provisional. Before release, we assess the product under selected conditions. After release, reality adds evidence we could not fully create in advance.
Pre-release evidence is necessary, but incomplete
Testing can show whether expected behaviour is present, whether important risks have been explored and whether quality characteristics such as performance, security or resilience meet defined expectations.
But testing always uses models. We select scenarios, create representative data, simulate load and define expected outcomes.
All of this is essential. But it cannot reproduce the full complexity of real use. Actual users behave differently from test users. Real data contains unexpected patterns. Dependencies fail in unfamiliar combinations. Support demand reveals confusion that no acceptance test exposed. A product may pass every planned assessment and still teach us something important after release.
Operation produces a different kind of evidence
Once the product is in use, new evidence appears:
- incidents and escaped defects;
- helpdesk calls and customer complaints;
- performance under real load;
- usage and abandonment patterns;
- manual workarounds;
- operational effort;
- customer and business outcomes.
This evidence does not merely tell us whether engineering made mistakes. It tells us how the product behaves in its actual context. A repeated support question may reveal weak usability. A production incident may expose an architectural assumption that was never explicit. Low adoption may show that a correctly implemented feature does not solve the intended problem. High operational effort may reveal maintainability or supportability problems. Reality makes the product more understandable.
The team’s responsibility continues after release
In many agile and DevOps environments, the development team does not hand the product over and disappear. The same people who design, build and test the system also help operate, support, maintain and improve it. That creates a direct quality-learning loop. Incidents reveal weaknesses in design, testing or operational preparation. Difficult fixes expose maintainability problems. Repeated manual interventions show where operability or automation is insufficient. Production behaviour tests assumptions made during engineering.
This evidence should matter deeply to the team. It is not merely reporting about a product they once delivered. It is feedback on the quality of their own engineering decisions. The team that creates the product should also learn from how that product behaves in reality. Done may end the implementation cycle. It should not end the team’s interest in quality.
Monitoring is not yet assessment
Production systems generate dashboards, logs, alerts and incident records. But observations do not interpret themselves. A green dashboard does not prove that the product is good. A metric may remain within its threshold while users still struggle.
Operational evidence must be interpreted against:
- intended outcomes;
- stakeholder needs;
- quality expectations;
- acceptable risks;
- operating context.
Monitoring produces evidence. Assessment turns that evidence into a quality judgement.
The reference model must survive release
A product-quality reference model cannot be used only before delivery. It must remain relevant in operation. The expectations defined earlier should help us interpret what happens in reality.
At the same time, operational evidence should be able to change the reference model. An incident may reveal a missing quality characteristic. Customer behavior may challenge an assumption. A support pattern may expose a need that was never understood. A change in scale may make yesterday’s acceptable quality level inadequate today.
The reference model must therefore be living. It provides the basis for assessment, but it also evolves through assessment.
Quality learning follows the lifecycle
The Definition of Done marks an important engineering boundary. Release marks another. Neither marks the end of quality. Before release, we ask whether the available evidence gives us enough confidence to proceed. After release, we ask what reality reveals about the product in actual use. These are different stages of the same quality-learning process.
Product quality must be assessed before release, but it can only be fully understood in operation. A mature team prepares the product carefully before release—and remains willing to learn what reality reveals afterwards.