Marcel van Oosterwijk

Author: Marcel

  • Practice Note: How Testing Became Quality Management Without Leaving Testing Behind

    Testing has changed considerably over the years.

    What once focused primarily on evaluating software products has expanded into test automation, quality assurance, quality engineering, process improvement, and increasingly organizational capability development.

    We have moved upstream, becoming involved in requirements, architecture, development practices, and organizational ways of working.

    Yet we still tend to describe much of this work as testing.

    Perhaps we have expanded the discipline without reconsidering what the discipline has become.

    Following Quality Upstream

    Testing provides information about products.

    When something doesn’t work as expected, testing helps us discover it, investigate it, and understand its consequences.

    But many problems discovered through testing originate elsewhere.

    Ambiguous requirements, architectural decisions, inadequate development practices, or insufficient organizational capabilities may all become visible through testing.

    Improving testing alone does not necessarily address these problems.

    So we started moving upstream.

    We became involved in requirements reviews, development practices, risk management, process improvement, and eventually organizational assessments.

    The underlying motivation remained the same: improving our ability to achieve quality.

    But the subject of our attention gradually changed.

    We were no longer concerned only with evaluating products. We were increasingly concerned with understanding and improving the organizational mechanisms that produce them.

    From Testing to Quality Management

    This development introduced a collection of overlapping concepts.

    Quality Assurance, Quality Control, Quality Engineering, Test Management, and Quality Management are frequently used in ways that make their responsibilities difficult to distinguish.

    Perhaps the problem is not primarily terminology.

    Perhaps we have been trying to accommodate an increasingly broad set of responsibilities within a discipline originally organized around testing.

    A useful distinction might be:

    • Testing investigates products and provides evidence about their behaviour and characteristics.
    • Quality Control evaluates whether products satisfy relevant quality expectations.
    • Quality Assurance provides confidence that the ways of working used to achieve quality are adequate.
    • Quality Engineering develops and applies engineering practices that help achieve quality.
    • Quality Management provides the organizational mechanisms for directing, evaluating, and improving the ability to achieve quality.

    These are not necessarily competing disciplines.

    They address different aspects of the same challenge.

    And testing remains an essential part of that challenge.

    Assessing More Than Products

    Organizations routinely evaluate their products.

    They test software, inspect deliverables, monitor performance, and investigate defects.

    But how systematically do they evaluate the organizational capabilities responsible for producing those products?

    When testing repeatedly reveals poor requirements, do we assess the organization’s requirements engineering capability?

    When defects originate in architectural decisions, do we examine the architecture capability?

    When problems recur across projects, do we investigate the organizational mechanisms that should enable learning and improvement?

    Often, we address the immediate problem without systematically evaluating the capability behind it.

    This suggests an interesting extension of Quality Management.

    Just as product assessment provides evidence about products, capability assessment could provide evidence about the organization’s ability to achieve intended outcomes.

    And just as product development responds to findings about products, capability development could respond to findings about organizational capabilities.

    Quality Management would then concern itself not only with whether quality is achieved, but also with whether the organization has the capabilities required to achieve it repeatedly.

    Without Leaving Testing Behind

    None of this makes testing less important.

    Testing remains one of our most valuable sources of evidence about whether products behave as intended.

    It also provides an important perspective on the consequences of decisions made elsewhere.

    But testing cannot compensate indefinitely for weaknesses throughout the organization.

    Nor should it be expected to.

    Moving upstream does not mean abandoning testing. It means recognizing that the information testing provides can lead us toward questions that testing alone cannot answer.

    And perhaps that is how testing gradually led us toward Quality Management.

    Not because testing became something else, but because understanding quality required us to look beyond testing.

    Quality is created throughout the organization. Quality Management makes that creation intentional, coordinated, and improvable.

  • Practice Note: When the KPI Prevents the Capability

    Organizations often have ambitions to develop their capabilities.

    They establish communities of practice, appoint capability owners, develop methodologies, and encourage knowledge sharing.

    At the same time, they may measure individual performance through KPIs that actively discourage these activities.

    Consider a consulting organization.

    A consultant who spends all their available time delivering to customers performs well according to utilization metrics.

    Another consultant spends part of that time developing methodologies, mentoring colleagues, or creating reusable knowledge.

    The first generates more directly attributable revenue.

    The second may be contributing more to the organization’s future ability to generate revenue.

    Yet according to the KPI, the first consultant is performing better.

    Measuring the Wrong Contribution

    There is nothing inherently wrong with measuring utilization. Consulting organizations depend on billable work.

    The problem arises when a metric designed to measure one aspect of performance becomes a general measure of individual contribution.

    Utilization measures billable time. It does not measure the value someone creates for the organization.

    A consultant delivering an assignment creates value through their own work.

    A consultant developing an approach that enables twenty colleagues to deliver better assignments creates value through the work of others.

    The latter contribution may be substantially greater, but it is harder to observe, attribute, and measure.

    And what is difficult to measure is often difficult to reward.

    The Capability Paradox

    Capability development requires investment.

    Someone must examine existing practices, identify improvements, develop better approaches, and help others adopt them.

    All of this takes time that could otherwise have been spent on billable work.

    The organization wants its capabilities to improve, but evaluates the people responsible according to criteria that reward them for spending their time elsewhere.

    Capability development becomes something to do when there is time left over.

    And in an organization optimizing utilization, there may be very little time left over.

    The organization has made capability development dependent on the availability of a resource it is simultaneously trying to eliminate: unused capacity.

    The resulting behaviour is entirely predictable.

    Yet the organization may interpret the lack of capability development as insufficient engagement, ownership, or initiative.

    Perhaps people are simply responding exactly as the organization has instructed them to.

    From Productivity to Leverage

    Individual productivity concerns what someone accomplishes through their own work.

    Organizational capability concerns what the organization can accomplish repeatedly, independently of particular individuals.

    Developing that capability means making knowledge, methods, and practices available beyond their original creators.

    Its value is often indirect, distributed, and delayed.

    A methodology developed today may improve dozens of assignments over several years.

    But those improvements will appear in the performance of the consultants using it, not necessarily in the performance indicators of the person who developed it.

    The organization benefits without necessarily recognizing where the benefit originated.

    If we measure individual contribution primarily through direct output, how do we recognize contributions that improve the output of others?

    What Are We Actually Optimizing?

    The obvious response might be to introduce additional KPIs for capability development.

    But counting methodologies, knowledge-sharing sessions, or reusable assets does not necessarily tell us whether organizational capabilities have improved.

    We would simply be measuring more activities.

    Perhaps the more important question is whether the organization’s management mechanisms support what it is trying to achieve.

    Maximizing utilization and developing organizational capability are different objectives. They compete for the same resources, and the organization must decide how to balance them.

    That balance cannot be established through strategic declarations alone. It must be reflected in how work is prioritized, capacity is allocated, and contributions are evaluated.

    Otherwise, capability development remains an ambition that the organization’s own management system works against.

    Perhaps the question is not whether we have the right KPIs, but whether we understand what our KPIs are preventing us from achieving.

  • Exploration: What Happens to the Knowledge Worker When AI Removes the Bottlenecks?

    I learned something useful from performance testing many years ago. When you remove a bottleneck from a system, another bottleneck usually appears.

    That does not mean the optimization failed. The performance of the whole system has increased. The next constraint simply became visible because the previous one no longer dominated.

    I wonder if this is a useful way to think about AI and knowledge work.

    Much of knowledge work is built around expensive activities: finding information, reading it, comparing it, writing, analyzing, programming, documenting, communicating and reviewing. Organizations have developed roles, processes, queues and handovers around those constraints.

    AI is now reducing the cost of several of those activities at once. The obvious conclusion is that fewer people will be needed. That may be true in some cases. But another possibility is that the bottleneck simply moves.

    If software implementation becomes dramatically faster, development capacity may stop being the main constraint. Then other constraints become more visible: deciding what is worth building, understanding customer needs, integrating changes, assuring quality, absorbing change in operations, or learning from what actually happens after release.

    The system may still become much faster overall. That matters because a role is not the same thing as the activities currently associated with it. A Product Manager may spend a lot of time managing a backlog because implementation capacity is scarce. A technical writer may spend a lot of time producing documents because creating and maintaining representations is expensive. A Quality Manager may spend a lot of time collecting and summarizing evidence because that work is slow and fragmented.

    If AI removes those frictions, the activity may shrink without the underlying function disappearing.

    That raises a more interesting question:

    If AI can perform many of the activities associated with a profession, were those activities actually the function of the profession, or merely the mechanisms through which that function happened to be realized?

    For a Product Manager, the deeper function may be understanding context and deciding what should change and why. For a technical writer, it may be preserving and making technical understanding available. For a Quality Manager, it may be maintaining enough understanding of quality, risk and evidence for the organization to make informed decisions. AI may therefore push roles upstream toward their underlying purpose.

    There is also another effect: when something becomes cheaper, we often use more of it.

    If software becomes cheaper to build, more software may become economically viable. If analysis becomes cheap, questions that were never worth investigating may suddenly be worth answering. If quality evidence can be gathered continuously, assurance may expand rather than simply become cheaper.

    So the future may not be today’s amount of knowledge work performed by fewer people.

    It may be far more knowledge work being performed, with different constraints determining where human attention is most valuable.

    And that is where the performance-testing analogy becomes useful again.

    The next bottleneck might be judgment. Or trusted evidence. Or customer attention. Or organizational ability to absorb change. Or something AI will also reduce later.

    The important thing is that removing one constraint does not tell us that the system is finished. It tells us where to look next.

    So perhaps the more useful question is not:

    Will AI replace the knowledge worker?

    But:

    Which bottlenecks currently shape this role, what happens when AI removes them, and what function remains when the old mechanisms are no longer necessary?

  • Blog Retrospective #5

    After the previous retrospective, the pace of new Practice Notes slowed down considerably.

    At first glance, that might look as if the inquiry itself had slowed down. In reality, almost the opposite happened. I was simply spending less time adding new notes and more time working with what had already accumulated.

    The Practice Notes started feeding temporary synthesis notes, more structured theory, standard assessment and development approaches, LinkedIn posts, and articles for my employer’s blog.

    Retrospective #4 had already suggested that the notes might become the primary body of evidence, while papers, models and guidance became derived representations. What I had not yet understood was that the relationship would not be one-way. 

    Those derived representations started giving something back.

    Trying to synthesize several notes exposed gaps and tensions between them. Turning theory into a standard approach forced abstract ideas to become usable. A LinkedIn post forced an argument to survive without all the context around it. And once those compressed ideas entered public conversations, people from other disciplines began challenging them, extending them, or asking questions I had not considered.

    Some of those questions eventually returned as new Practice Notes.

    The flow had changed. Experience still generated notes, but the notes now also generated synthesis, applications and conversations, and those in turn generated new experience and new questions. The Body of Knowledge was no longer merely accumulating. It had begun to behave like a learning system.

    I think that also explains why it has become increasingly difficult to say what the Body of Knowledge is actually about.

    The blog began in a reasonably recognizable territory. My original questions concerned software quality, assessment, evidence, organizational capabilities and the models we use to understand them. Even then, I described the scope as broader than testing alone, extending toward systems thinking and the conceptual models used to understand complex organizations. 

    But every cycle of inquiry seems to expose another relationship beyond the previous boundary.

    Quality led into risk and evidence. Requirements led into intent. Product development led into operations and continuity. Documentation led into the preservation of understanding. Organizational capabilities led into stewardship. AI raised the possibility that future understanding may not need to be stored primarily in predefined documents at all.

    Even an Exploration of Vivaldi’s Winter Largo followed a strangely familiar path. The interesting question was not merely which notes the bass voice played, but what function those notes performed within the music.

    That recurring habit has become more visible to me lately. When I encounter a mechanism, practice or artifact, I tend to move upstream from it.

    Why does this requirement exist? Why is this criterion in the Definition of Done? Why do we create this document? Why does this capability need to exist?

    The question is not simply a search for root cause. It is an attempt to recover the relationship between mechanism, function, intent and outcome.

    This also helps explain why several recent threads seem to converge around the idea of stewardship. A document, requirement, model or report may be useful, but it is still only a representation. Different artifacts may contain different claims about the same product, while different specialists each hold part of the relevant understanding.

    The challenge then shifts. Instead of trying to preserve one perfect artifact, perhaps we need to preserve enough coherence around the subject itself that people can continue to understand, develop and assess it as circumstances change.

    That idea led to one formulation I keep returning to:

    Contribution follows expertise; stewardship preserves coherence.

    Quality has not disappeared from the work as the scope widened. If anything, the latest notes show how strongly it still anchors the inquiry.

    The recent note on Ready and Done began with a very ordinary Sprint-end problem: tests being negotiated away because work needs to become Done. But asking why those controls existed changed the meaning of the situation. Skipping a test was no longer merely a delivery shortcut. It could be understood as accepting additional risk without the evidence that the test was supposed to provide.

    Following that line of thought into refinement produced another observation: refinement is not only about reducing uncertainty about what to build. It is also about reducing uncertainty about what could go wrong.

    That is still recognizably quality thinking, but it no longer feels bounded by QA. It reaches into product management, risk, decision-making, operations and learning.

    Perhaps that is why Quality Management feels closer to the territory I am moving through, while still not quite describing all of it.

    Looking back across the Body of Knowledge, I increasingly see one part concerned with how we intervene in systems and another concerned with how we understand them. But even that distinction is becoming difficult to maintain. We need understanding in order to intervene responsibly. The intervention produces consequences. Those consequences give us new evidence. The evidence changes our understanding, which should then influence the next intervention.

    The two strands keep folding back into each other.

    And perhaps this is why the subject itself seems harder to define now than it did twenty or forty notes ago.

    Not despite the Body of Knowledge becoming a learning system, but because of it.

    Every cycle seems to reveal relationships that were not visible before. That does not necessarily mean that the subject itself is objectively expanding. It means that my understanding of its boundary keeps changing as the inquiry develops.

    That distinction matters. Retrospective #4 already contained an important warning: as the theory became more ambitious, it became increasingly important to distinguish observation from interpretation and speculation, and not to make an experience prove more than it actually did. 

    So I am not ready to give the Body of Knowledge a new grand title.

    Quality Assurance is clearly too narrow. Quality Management gets closer. Systems thinking, organizational capability, assessment, knowledge management, product management and organizational learning all seem to touch parts of it.

    For now, one question seems to describe the territory better than any label:

    How do we understand changing systems well enough to intervene responsibly, learn from the consequences, and preserve that understanding over time?

    I do not know whether that is what the Body of Knowledge is ultimately about.

    But I think it is now a fair question to ask.

    When I started the blog, I wrote that writing was not merely a way to communicate conclusions. It was a way to discover what I think. 

    That still feels right.

    The difference is that the mechanism has become larger than writing. The Practice Notes feed synthesis, theory, approaches and public discussion. Those representations encounter other contexts and other people. What comes back changes the Body of Knowledge again.

    The previous retrospective ended with the observation that the theory had started talking back.

    Now it seems to be asking a more difficult question:

    What is it actually a theory of?

    And perhaps, for now, it is better that I do not know.

  • Retrospective #5: When the Body of Knowledge Started Learning

    After the previous retrospective, the pace of new Practice Notes slowed down considerably.

    At first glance, that might look as if the inquiry itself had slowed down. In reality, almost the opposite happened. I was simply spending less time adding new notes and more time working with what had already accumulated.

    The Practice Notes started feeding temporary synthesis notes, more structured theory, standard assessment and development approaches, LinkedIn posts, and articles for my employer’s blog.

    Retrospective #4 had already suggested that the notes might become the primary body of evidence, while papers, models and guidance became derived representations. What I had not yet understood was that the relationship would not be one-way. 

    Those derived representations started giving something back.

    Trying to synthesize several notes exposed gaps and tensions between them. Turning theory into a standard approach forced abstract ideas to become usable. A LinkedIn post forced an argument to survive without all the context around it. And once those compressed ideas entered public conversations, people from other disciplines began challenging them, extending them, or asking questions I had not considered.

    Some of those questions eventually returned as new Practice Notes.

    The flow had changed. Experience still generated notes, but the notes now also generated synthesis, applications and conversations, and those in turn generated new experience and new questions. The Body of Knowledge was no longer merely accumulating. It had begun to behave like a learning system.

    I think that also explains why it has become increasingly difficult to say what the Body of Knowledge is actually about.

    The blog began in a reasonably recognizable territory. My original questions concerned software quality, assessment, evidence, organizational capabilities and the models we use to understand them. Even then, I described the scope as broader than testing alone, extending toward systems thinking and the conceptual models used to understand complex organizations. 

    But every cycle of inquiry seems to expose another relationship beyond the previous boundary.

    Quality led into risk and evidence. Requirements led into intent. Product development led into operations and continuity. Documentation led into the preservation of understanding. Organizational capabilities led into stewardship. AI raised the possibility that future understanding may not need to be stored primarily in predefined documents at all.

    Even an Exploration of Vivaldi’s Winter Largo followed a strangely familiar path. The interesting question was not merely which notes the bass voice played, but what function those notes performed within the music.

    That recurring habit has become more visible to me lately. When I encounter a mechanism, practice or artifact, I tend to move upstream from it.

    Why does this requirement exist? Why is this criterion in the Definition of Done? Why do we create this document? Why does this capability need to exist?

    The question is not simply a search for root cause. It is an attempt to recover the relationship between mechanism, function, intent and outcome.

    This also helps explain why several recent threads seem to converge around the idea of stewardship. A document, requirement, model or report may be useful, but it is still only a representation. Different artifacts may contain different claims about the same product, while different specialists each hold part of the relevant understanding.

    The challenge then shifts. Instead of trying to preserve one perfect artifact, perhaps we need to preserve enough coherence around the subject itself that people can continue to understand, develop and assess it as circumstances change.

    That idea led to one formulation I keep returning to:

    Contribution follows expertise; stewardship preserves coherence.

    Quality has not disappeared from the work as the scope widened. If anything, the latest notes show how strongly it still anchors the inquiry.

    The recent note on Ready and Done began with a very ordinary Sprint-end problem: tests being negotiated away because work needs to become Done. But asking why those controls existed changed the meaning of the situation. Skipping a test was no longer merely a delivery shortcut. It could be understood as accepting additional risk without the evidence that the test was supposed to provide.

    Following that line of thought into refinement produced another observation: refinement is not only about reducing uncertainty about what to build. It is also about reducing uncertainty about what could go wrong.

    That is still recognizably quality thinking, but it no longer feels bounded by QA. It reaches into product management, risk, decision-making, operations and learning.

    Perhaps that is why Quality Management feels closer to the territory I am moving through, while still not quite describing all of it.

    Looking back across the Body of Knowledge, I increasingly see one part concerned with how we intervene in systems and another concerned with how we understand them. But even that distinction is becoming difficult to maintain. We need understanding in order to intervene responsibly. The intervention produces consequences. Those consequences give us new evidence. The evidence changes our understanding, which should then influence the next intervention.

    The two strands keep folding back into each other.

    And perhaps this is why the subject itself seems harder to define now than it did twenty or forty notes ago.

    Not despite the Body of Knowledge becoming a learning system, but because of it.

    Every cycle seems to reveal relationships that were not visible before. That does not necessarily mean that the subject itself is objectively expanding. It means that my understanding of its boundary keeps changing as the inquiry develops.

    That distinction matters. Retrospective #4 already contained an important warning: as the theory became more ambitious, it became increasingly important to distinguish observation from interpretation and speculation, and not to make an experience prove more than it actually did. 

    So I am not ready to give the Body of Knowledge a new grand title.

    Quality Assurance is clearly too narrow. Quality Management gets closer. Systems thinking, organizational capability, assessment, knowledge management, product management and organizational learning all seem to touch parts of it.

    For now, one question seems to describe the territory better than any label:

    How do we understand changing systems well enough to intervene responsibly, learn from the consequences, and preserve that understanding over time?

    I do not know whether that is what the Body of Knowledge is ultimately about.

    But I think it is now a fair question to ask.

    When I started the blog, I wrote that writing was not merely a way to communicate conclusions. It was a way to discover what I think. 

    That still feels right.

    The difference is that the mechanism has become larger than writing. The Practice Notes feed synthesis, theory, approaches and public discussion. Those representations encounter other contexts and other people. What comes back changes the Body of Knowledge again.

    The previous retrospective ended with the observation that the theory had started talking back.

    Now it seems to be asking a more difficult question:

    What is it actually a theory of?

    And perhaps, for now, it is better that I do not know.