Marcel van Oosterwijk

Category: Assessment

Quality, testing, assessment, evidence, judgment, reference models, and the development and evaluation of quality and capability.

  • Practice Note: What If Requirements Are Reference Models?

    As an experiment, I have been exploring whether system testing can be reframed as a specialised form of quality assessment.

    The motivation is not to replace testing, but to see whether a generic assessment model can also explain traditional testing. An assessment typically requires two things: an assessment model that describes how evidence is gathered and evaluated, and a reference model that defines the criteria against which the object is assessed.

    Interestingly, the testing process maps surprisingly well to the assessment model. Test design becomes evidence planning, test execution becomes evidence collection, and test reporting becomes assessment reporting.

    But the more interesting insight emerged on the reference-model side.

    If testing is an assessment, what constitutes its reference model?

    The first obvious candidate is requirements. This reframes requirement analysis from understanding what to test into something more fundamental: constructing and validating the reference model against which the product will be assessed.

    Viewed this way, many familiar requirement problems start to look different. Are the requirements complete enough to assess the product? Are important quality attributes missing? Do stakeholders agree on the criteria for success? Are release decisions based on expectations that were never captured in the reference model?

    Perhaps some of the bottlenecks we experience in requirements work are not really testing problems at all. They are reference-model problems.

    That was not the insight I was looking for when I started this experiment, but it may turn out to be the more interesting one.

  • Practice Note: What Would We Assess a System Against?

    In an earlier field note, I explored the idea that testing might be understood as a form of assessment. Testing generates evidence. Assessment uses evidence to reach a judgment. But this raises a more difficult question:

    If we want to assess the quality of a system, what would we assess it against?

    For a capability assessment, the answer is relatively clear. A reference model describes the capability we expect to find. An assessment model explains how evidence should be collected, interpreted and judged.

    For system quality, the equivalent reference model is less obvious. Requirements and acceptance criteria may seem to provide the answer. They describe what the system should do and the conditions under which it can be accepted.

    But are they sufficient as a reference model for system quality? I do not think they are.

    Requirements are not a complete model of quality

    Requirements describe selected expectations. They may address functionality, performance, security, usability and other quality concerns. In a well-developed specification, they can provide a substantial basis for testing and acceptance.

    But requirements are still the result of selection. Some expectations are documented. Others remain implicit. Some stakeholders are represented more strongly than others. Some concerns are recognised early, while others become visible only during operation.

    A system can satisfy its documented requirements and still be difficult to operate, hard to maintain, inaccessible, fragile under unusual conditions, unsuitable for future growth or inappropriate for its business context.

    This does not necessarily mean that the requirements were badly written. It means that requirements describe what has been specified, not necessarily everything that matters.

    Acceptance criteria make selected expectations testable

    Acceptance criteria make expectations concrete. For example:

    Ninety-five per cent of transactions must complete within two seconds.

    This gives us something observable and testable. We can determine whether the system meets the threshold. But it does not answer the wider questions:

    • Why are two seconds acceptable?
    • Which transactions and operating conditions are included?
    • What happens to the slowest five per cent?
    • Will the threshold remain adequate as demand grows?
    • Does it reflect actual user and business needs?

    The criterion defines a threshold. It does not explain the significance of that threshold.

    The same limitation applies to qualities that are difficult to reduce to a pass-or-fail statement. A system may be maintainable for its current team but difficult for another supplier to take over. It may be secure against the threats that were tested but exposed through dependencies outside the test scope. Acceptance criteria can express parts of these expectations. They cannot easily represent the entire quality picture.

    A reference model also reveals what is missing

    Testing against acceptance criteria primarily confirms whether specified expectations have been met. A system-quality assessment must also ask whether the relevant expectations were specified in the first place. A reference model should prompt questions such as:

    • Have the relevant quality characteristics been considered?
    • Are the important stakeholder groups represented?
    • Have operational and lifecycle concerns been included?
    • Are dependencies and consequences of failure understood?
    • Have future changes in scale and use been considered?
    • Which assumptions remain unvalidated?

    This leads to an important distinction:

    Acceptance criteria confirm what has been specified. A reference model also helps reveal what has not been specified.

    That may be one of its most valuable contributions. It enables us to challenge the completeness of the test basis rather than only derive tests from it.

    A taxonomy is not yet a reference model

    Established quality models describe characteristics such as functional suitability, reliability, performance efficiency, usability, security, compatibility, maintainability and portability. These provide a useful starting point.

    But a list of characteristics is not automatically an assessment reference model. For each quality area, a usable model should help us understand what the characteristic means, which questions should be asked, what strong or weak quality might look like, what evidence could support a conclusion and how the operating context affects the judgment.

    Without this, the model tells us where to look but not how to judge what we find. It remains a classification system rather than an assessment foundation.

    Reference model and assessment model

    The distinction between a reference model and an assessment model is important. The reference model describes the quality space against which the system is assessed. The assessment model describes how the assessment is performed.

    The reference model might identify recoverability as an important aspect of reliability. The assessment model would explain how to determine whether recoverability is adequate, perhaps through recovery tests, operational exercises, incident data, architecture analysis and interviews.

    The reference model tells us that recoverability matters. The assessment model tells us how to reach a defensible conclusion about it.

    A possible structure could be:

    • quality characteristic;
    • sub-characteristic;
    • assessment question;
    • evidence requirement;
    • indicator or measure;
    • acceptance criterion.

    For example:

    • Reliability
      • Recoverability
        • Can acceptable service be restored after failure?
          • Evidence from recovery tests and operational exercises
            • Recovery time, data loss and restoration success
              • Service restored within 30 minutes with no more than five minutes of data loss

    The acceptance criterion remains important, but it becomes one element within a wider reasoning framework.

    System quality is contextual

    A reference model for system quality cannot describe only the intrinsic properties of the system. It must also address fitness for context. The same system may be suitable for one purpose and unsuitable for another. A service may be adequate for an internal pilot but unacceptable for a business-critical process.

    The assessment therefore needs two connected perspectives. The first concerns the qualities the system possesses: functionality, reliability, security, usability, maintainability, operability and performance.

    The second concerns whether those qualities are adequate for the intended context: business criticality, stakeholder needs, user groups, operational environment, expected scale, risk appetite, regulatory obligations, dependencies and consequences of failure.

    A technically strong system can still be unfit for a particular context. A technically modest system can be entirely adequate for limited, low-risk use. Quality cannot be judged independently of purpose.

    The model must support judgment and uncertainty

    Acceptance criteria are commonly propositions that can be verified. A reference model must support a broader judgment.

    It may need to conclude that a quality characteristic is strong, weak, adequate only under certain conditions, acceptable for limited use, insufficiently evidenced or dependent on unverified assumptions.

    It must also distinguish between the quality of the system and confidence in the conclusion.

    A system may appear reliable because no serious failures have been observed. But if it has never been exposed to realistic load or failure conditions, confidence in that conclusion should be low.

    A failed test can provide strong evidence of weakness. A passed test may provide weak evidence of quality when the environment, data or scope were unrepresentative.

    The model must therefore connect quality characteristics not only to criteria, but also to evidence and uncertainty.

    Perhaps the real limitation is not the test process

    When testing is criticized for being too narrow, attention often turns to the process. We conclude that testing should begin earlier, include more automation, involve more stakeholders or extend into production. These may all be valid improvements.

    But perhaps testing often appears narrow because the model against which the system is judged is narrow. When requirements and acceptance criteria define the entire quality horizon, testing can only confirm expectations that have already been made visible.

    A stronger reference model would change the questions we ask before it changes the tests we perform. It would expose missing quality concerns, broaden the evidence we consider and help us distinguish between meeting criteria and being fit for purpose.

    The challenge may therefore not merely be to improve the test process. It may be to develop a sufficiently rich reference model for system quality.

    Because before we can assess whether a system is good, we need a credible answer to a more fundamental question:

    What, exactly, are we assessing it against?

  • Practice Note: Evidence Is Not Interpretation

    One of the most common weaknesses I encounter in assessments is surprisingly simple. Evidence and interpretation become mixed together. At first glance, this may seem harmless. In practice, it makes assessments harder to challenge, harder to validate, and ultimately less trustworthy.

    The Problem

    Consider the statement:

    Test automation is insufficient.

    Is this evidence? Or is it a conclusion? Most people instinctively treat it as an observation. In reality, it is an interpretation. The statement already contains a judgment. It implies a comparison against some expectation, objective, or benchmark.

    The natural response is:

    Insufficient according to whom?

    Compared to what?

    Based on what evidence?

    Without answers to those questions, the statement remains an opinion rather than an assessment finding.

    Separating the Layers

    A stronger assessment makes each step of the reasoning explicit.

    Evidence

    • 12% of regression tests are automated.
    • Automation execution requires manual preparation.
    • Critical business flows are excluded from automation coverage.

    Interpretation

    • The automation capability provides limited support for regression testing.

    Conclusion

    • Release cycles remain highly dependent on manual testing effort.

    Recommendation

    • Increase automation coverage of critical business flows.

    Each layer serves a different purpose. And more importantly, each layer can be challenged independently.

    Why This Matters

    When evidence and interpretation are merged, disagreements become difficult to resolve. A stakeholder may reject a finding without being able to explain why. The assessor may defend a conclusion without clearly identifying the supporting evidence. The discussion becomes subjective.

    When evidence and interpretation are separated, the conversation becomes more productive. A stakeholder can challenge:

    • the evidence,
    • the interpretation,
    • the conclusion,
    • or the recommendation.

    Each disagreement becomes explicit. Each assumption becomes visible.

    Traceability of Reasoning

    One useful way to think about assessment is as a chain of reasoning.

    Evidence leads to interpretation. Interpretation leads to conclusions. Conclusions lead to recommendations. A robust assessment makes this chain traceable.

    If a recommendation is challenged, it should be possible to walk backwards through the reasoning and understand how it was derived. Likewise, if new evidence emerges, its impact on conclusions should be clear. The reasoning should be visible, not hidden.

    Beyond Expert Opinion

    Many assessments rely heavily on expert judgment. There is nothing inherently wrong with expertise. However, expertise becomes significantly more valuable when the reasoning behind it is transparent.

    A strong assessment does not merely say:

    Trust me.

    It says:

    Here is the evidence, here is how I interpreted it, here is the conclusion that follows.

    The assessor’s role is not to replace reasoning. It is to make reasoning explicit.

    A Working Hypothesis

    This leads me to a simple hypothesis:

    Effective assessments require explicit separation of evidence and interpretation, with clear traceability between them.

    The quality of an assessment is not determined solely by the quality of its conclusions. It is determined by the visibility of the reasoning that produced them. After all, an assessment is more than a collection of opinions. It is an argument supported by evidence.

  • Practice Note: Who Assesses the Assessor?

    In recent years, I have spent a significant amount of time helping organizations assess their capabilities.

    Quality assurance.

    Testing.

    Data quality.

    Processes.

    Governance.

    Organizations routinely evaluate how well these capabilities perform and where improvement is needed.

    Yet a question occurred to me:

    Who assesses the assessor?

    The Invisible Capability

    When conducting an assessment, we tend to focus on the subject being assessed. We discuss the quality of testing. The maturity of data management. The effectiveness of governance. The assessment itself is often treated as a neutral instrument. A means to an end.

    But what if assessment is not merely an activity? What if assessment is a capability in its own right? If so, it should be subject to the same scrutiny as any other capability.

    Separating the Subject from the Assessment

    A useful distinction is the separation of a reference model from an assessment model. The reference model describes what good looks like. The assessment model describes how we evaluate against it. Most discussions stop there.

    However, there is a third question:

    How do we know the assessment itself is any good?

    A poor assessment can produce misleading conclusions, inappropriate recommendations, and misguided investments. In other words, the quality of improvement initiatives is often limited by the quality of the assessment that precedes them.

    Assessment as a Professional Discipline

    One reason this question is rarely asked may be that assessment is often viewed as an extension of domain expertise. If someone understands testing, they assess testing. If someone understands data quality, they assess data quality.

    Yet experience suggests that understanding a domain and assessing a domain are different capabilities. Domain expertise helps us understand what is happening. Assessment expertise helps us determine what it means. The best assessments typically emerge from collaboration between both forms of expertise.

    What Makes a Good Assessment?

    If assessment is a capability, it should be possible to describe what good assessment looks like.

    Some characteristics immediately come to mind:

    • Clear scope and objectives
    • Structured methodology
    • Evidence-based conclusions
    • Traceability from findings to recommendations
    • Balanced stakeholder engagement
    • Appropriate challenge of assumptions
    • Consideration of alternative explanations
    • Actionable recommendations

    Most professionals would recognize these as indicators of assessment quality. Yet they are rarely evaluated explicitly.

    Learning from Other Disciplines

    Other professions routinely evaluate their own methods. Researchers subject their work to peer review. Auditors perform quality reviews. Architects conduct architecture reviews. Scientists continuously challenge the validity of their findings.

    The assessment profession should be no different. Assessments should be open to challenge, critique, and improvement. In fact, the strongest assessments are often those that have survived the most scrutiny.

    A Working Hypothesis

    This leads me to a simple hypothesis:

    Assessment is a capability in its own right and should therefore be assessed like any other capability.

    Organizations routinely evaluate testing, quality, governance, and architecture. Perhaps it is time to evaluate assessment as well.

    After all, if we believe in assessment as a mechanism for improvement, it seems reasonable to ask:

    Who assesses the assessor?

  • Practice Note: What If Testing Is Just a Form of Assessment?

    Over the past few weeks, while working on assessment models for QA capability and Data Quality, I stumbled upon an idea that has been difficult to shake off.

    The observation itself is surprisingly simple: the structure of a QA capability assessment and a system test is essentially the same. The only thing that changes is the object being assessed.

    At first, this felt like an interesting analogy. The more I thought about it, however, the more it began to feel like something more fundamental.

    Starting with System Testing

    Most of us who have worked in testing have been taught a familiar model:

    • Requirements define what the system should do.
    • Test cases verify those requirements.
    • Defects are reported.
    • A release decision is made.

    Nothing controversial there.

    But viewed through a different lens, the process can be described in another way:

    • Requirements define a reference model.
    • Testing is an assessment process.
    • Test results are evidence.
    • Defects are findings.
    • The test report is an evaluation.
    • The release recommendation is decision support.

    In other words, system testing can be described as a quality assessment of a software system. That may sound like a semantic distinction, but it has interesting consequences.

    Looking Beyond Testing

    Around the same time, I was working with assessment models for QA capability and Data Quality.

    Those assessments typically look like this:

    • Define a reference model.
    • Gather evidence through interviews, reviews and metrics.
    • Identify gaps.
    • Evaluate current capability.
    • Recommend improvements.

    Structurally, the process is identical. The only difference is that the object being assessed is no longer a software system. Instead of assessing a product, we assess:

    • A QA organization
    • A data platform
    • An operating model
    • A security posture
    • An architecture

    The pattern remains the same:

    Object → reference model → assessment activities → evidence → findings → evaluation → decision support

    Once you see it, it becomes difficult to unsee.

    Is QA Already Solving This?

    One could argue that Quality Assurance and later Quality Engineering have already attempted to broaden testing into something larger. There is certainly truth in that:

    Testing evolved from defect detection. QA expanded the focus toward process quality. Quality Engineering expanded it further toward prevention, automation and quality built into delivery. Yet even Quality Engineering remains largely anchored in the vocabulary of testing:

    • Test automation
    • Test coverage
    • Test strategy
    • Test management
    • Test environments

    The underlying mental model often remains:

    Software → testing → quality

    What if the model should instead be:

    Object → assessment → evidence → quality evaluation

    That is a subtle but important shift.

    A Different Perspective on Testing

    If assessment becomes the primary concept, testing becomes one assessment technique among many.

    Consider the following examples:

    ObjectAssessment Technique
    Source CodeUnit Testing
    Software SystemSystem Testing
    ArchitectureArchitecture Review
    Data PlatformData Profiling
    Security PosturePenetration Testing
    QA OrganizationInterviews and Document Review

    Different techniques, same assessment pattern.

    This perspective also changes the questions we ask.

    Instead of how many test cases have we executed, we ask: have we collected sufficient evidence?

    Instead of what is our test coverage, we ask: which quality attributes have been assessed?

    Instead of how many defects remain, we ask: what is our confidence level and residual risk?

    These are fundamentally different conversations. They are also conversations that executives tend to understand much better than traditional testing metrics.

    A Broader Quality Assessment Framework?

    Perhaps the most interesting implication is organizational rather than technical. Many consulting organizations offer services such as:

    • System testing
    • Data quality assessments
    • Security reviews
    • Architecture assessments
    • QA capability assessments

    These are often positioned as separate offerings with separate methodologies.

    But what if they are all instances of the same underlying discipline? A generic quality assessment framework could potentially provide:

    • A common assessment model
    • A common reference model structure
    • A common evidence model
    • A common reporting approach
    • A common maturity model

    The assessed object changes. The assessment pattern does not.

    An Open Question

    I do not yet know whether this idea is genuinely new or merely a reframing of concepts that already exist within Quality Engineering.

    What I do know is that it has changed the way I think about testing. For years, I viewed testing as a discipline concerned with finding defects.

    Today, I increasingly see testing as one technique for generating evidence about quality. That may sound like a small distinction. I suspect it is not.

    If the observation holds, then perhaps testing is not the overarching discipline after all. Perhaps quality assessment is. And testing is simply one of its most visible techniques.