Field Notes from Quality & Delivery Transformations

Blog

  • Field Note: Who Assesses the Assessor?

    In recent years, I have spent a significant amount of time helping organizations assess their capabilities.

    Quality assurance.

    Testing.

    Data quality.

    Processes.

    Governance.

    Organizations routinely evaluate how well these capabilities perform and where improvement is needed.

    Yet a question occurred to me:

    Who assesses the assessor?

    The Invisible Capability

    When conducting an assessment, we tend to focus on the subject being assessed. We discuss the quality of testing. The maturity of data management. The effectiveness of governance. The assessment itself is often treated as a neutral instrument. A means to an end.

    But what if assessment is not merely an activity? What if assessment is a capability in its own right? If so, it should be subject to the same scrutiny as any other capability.

    Separating the Subject from the Assessment

    A useful distinction is the separation of a reference model from an assessment model. The reference model describes what good looks like. The assessment model describes how we evaluate against it. Most discussions stop there.

    However, there is a third question:

    How do we know the assessment itself is any good?

    A poor assessment can produce misleading conclusions, inappropriate recommendations, and misguided investments. In other words, the quality of improvement initiatives is often limited by the quality of the assessment that precedes them.

    Assessment as a Professional Discipline

    One reason this question is rarely asked may be that assessment is often viewed as an extension of domain expertise. If someone understands testing, they assess testing. If someone understands data quality, they assess data quality.

    Yet experience suggests that understanding a domain and assessing a domain are different capabilities. Domain expertise helps us understand what is happening. Assessment expertise helps us determine what it means. The best assessments typically emerge from collaboration between both forms of expertise.

    What Makes a Good Assessment?

    If assessment is a capability, it should be possible to describe what good assessment looks like.

    Some characteristics immediately come to mind:

    • Clear scope and objectives
    • Structured methodology
    • Evidence-based conclusions
    • Traceability from findings to recommendations
    • Balanced stakeholder engagement
    • Appropriate challenge of assumptions
    • Consideration of alternative explanations
    • Actionable recommendations

    Most professionals would recognize these as indicators of assessment quality. Yet they are rarely evaluated explicitly.

    Learning from Other Disciplines

    Other professions routinely evaluate their own methods. Researchers subject their work to peer review. Auditors perform quality reviews. Architects conduct architecture reviews. Scientists continuously challenge the validity of their findings.

    The assessment profession should be no different. Assessments should be open to challenge, critique, and improvement. In fact, the strongest assessments are often those that have survived the most scrutiny.

    A Working Hypothesis

    This leads me to a simple hypothesis:

    Assessment is a capability in its own right and should therefore be assessed like any other capability.

    Organizations routinely evaluate testing, quality, governance, and architecture. Perhaps it is time to evaluate assessment as well.

    After all, if we believe in assessment as a mechanism for improvement, it seems reasonable to ask:

    Who assesses the assessor?

  • Field Note: Continuity

    A few months ago, my mother passed away. Recently, a question about our family history popped into my head. My immediate reaction was simple: I’ll ask Mom. Then I remembered, I couldn’t. The answer disappeared with her.

    A Library of One

    What struck me was not the loss itself. I had already experienced that. What struck me was the realization that my mother had carried a unique body of knowledge. Stories. Relationships. Explanations.Family history. Context. The answers to questions nobody knew would one day be asked. For years, that knowledge was always available. A simple question was enough.

    Then suddenly it wasn’t. Not because the knowledge lacked value. Because the person who carried it was no longer there.

    The Fragility of Knowledge

    This is not unique to families. Every person carries a library. Years of experience. Lessons learned. Mistakes made. Patterns observed. Insights developed.

    Much of this knowledge exists nowhere else. It remains tacit. Unwritten. Unrecorded. Accessible only through conversation. When a person leaves, part of that library often disappears as well.

    Thinking About My Own Library

    The experience made me reflect on my own situation. Over the course of a career, I have accumulated a large collection of observations, mental models, and insights. Some are personal. Some are professional. Some are useful only to my family. Others may be useful to people I will never meet.

    Like everyone else, I will eventually leave this world. The question is not whether that happens. The question is what happens to the knowledge I carry when it does.

    Two Different Forms of Continuity

    I find myself thinking about continuity in two different ways.

    The first is personal. I would like my son to have access to as much of my experience as possible. Not because I expect him to follow the same path. But because every parent hopes to pass on some of what they have learned.

    The second is broader. I would like useful ideas, observations, and lessons learned to remain available to others. Not because they are mine. But because they might help someone else think, learn, or solve a problem.

    Why Field Notes Matter

    For years, capturing knowledge required significant effort. Books had to be written. Articles had to be published. Most insights never made the journey from intuition to publication.

    Today, the friction is much lower. A thought can become a conversation. A conversation can become a field note. A field note can become part of a growing body of knowledge. Each note is a small act of continuity. A small attempt to ensure that something learned does not simply disappear.

    Beyond Immortality

    People sometimes speak about legacy or immortality. I find myself less interested in those ideas.

    The opposite of mortality is not immortality. It is continuity. I do not need my name to survive. What I hope survives are the useful things. The lessons. The observations. The insights. The things that might still help someone after I am gone.

    A Working Hypothesis

    This leads me to a simple hypothesis:

    Human lives are finite. Continuity is not.

    Every conversation, story, lesson, article, field note, and book is an attempt to carry something forward. Not forever. But long enough to matter.

    Perhaps that is one of the most meaningful things we can do with the knowledge we accumulate during a lifetime.

    Not keep it. Pass it on.

  • Field Note: Uncovering Hidden Ideas

    When people discuss AI-assisted writing, the conversation often focuses on productivity. Writing becomes faster. Editing becomes easier. Publishing becomes cheaper. While all of this is true, I increasingly believe it misses the most important effect. The greatest value of reduced friction is not efficiency.

    It is revelation.

    The Myth of Fully Formed Ideas

    We often imagine that ideas already exist inside our heads. The challenge is simply expressing them clearly. In this view, writing is a transmission process. An existing thought is transferred from mind to paper.

    My experience suggests something different. Many valuable ideas do not exist in a fully formed state. They begin as vague intuitions. Patterns. Questions. Half-recognized connections. At this stage, the idea is not ready to be written. In many cases, it is not even ready to be articulated.

    Ideas Need Space to Develop

    For an idea to mature, it must be explored. Questions need to be asked. Assumptions need to be challenged. Alternative interpretations need to be considered. Connections need to be discovered.

    Historically, this process carried significant friction. As a result, many ideas never progressed beyond an initial intuition. Not because they lacked value. But because they never received enough attention to become visible.

    Hidden Rather Than Lost

    Looking back, I suspect that many of our best ideas are not lost. They remain hidden. They exist as possibilities rather than conclusions. As intuitions rather than models. As observations rather than theories.

    The problem is not that we fail to generate ideas. The problem is that most ideas never receive enough exploration to reveal what they might become.

    What Changes When Friction Disappears

    When the effort required to explore an idea becomes negligible, something remarkable happens. Ideas can be followed immediately. Questions can be pursued. Connections can be tested. Hypotheses can be challenged. Thoughts that would previously have remained unfinished can evolve.

    The result is not merely more writing. The result is the emergence of ideas that would otherwise have remained hidden.

    A Working Hypothesis

    This leads me to a hypothesis:

    Lower friction does not simply make it easier to express ideas. It makes it easier to discover them.

    The most valuable outcome is not faster publication. It is the revelation of thoughts that would never have become fully visible without the opportunity to explore them.

    Perhaps the greatest contribution of AI is not that it helps us write what we already know. Perhaps it helps us discover what we think.

  • Field Note: Agile Report Writing

    After reflecting on how recent assessment reports evolved, I realized that the process felt surprisingly familiar. Not because it resembled traditional report writing. Because it resembled Agile software development.

    The Waterfall Model of Reporting

    Most report writing follows a waterfall mindset. The process is typically assumed to be:

    1. Gather information
    2. Analyze findings
    3. Draw conclusions
    4. Write report
    5. Review report
    6. Deliver report

    The underlying assumption is that understanding precedes writing. First we figure out what we think. Then we document it. Writing is treated as the final step. The report becomes a container for conclusions that already exist. This model feels natural. It is also how many assessment reports have traditionally been produced.

    What Actually Happened

    My recent experience was very different. The report did not emerge from a completed analysis. The analysis evolved together with the report.A finding would be challenged. A recommendation would be reconsidered. An alternative explanation would emerge. A new perspective would change the interpretation. The assessment model itself might evolve.

    The report became less of a documentation activity and more of a learning process. The conclusions were not fully understood before writing began. They emerged during the process.

    A Familiar Pattern

    This should sound familiar to anyone who has worked with Agile software development. Traditional software development assumed that requirements could be understood completely before implementation started. Agile challenged that assumption.

    The Agile insight was simple:

    • Understanding emerges through iteration.
    • Requirements evolve.
    • Design evolves.
    • Architecture evolves.
    • The product evolves.
    • The team learns while building.

    The final solution cannot be fully understood at the start because the learning happens during development.

    The Same Principle Applied to Reporting

    I increasingly believe the same principle applies to assessment reports and other forms of knowledge work.

    The traditional reporting model assumes:

    Understand first, write later.

    An Agile reporting model assumes:

    Write, challenge, refine, learn, and repeat.

    The report becomes a vehicle for exploration rather than documentation. Each draft becomes an experiment. Each critique becomes feedback. Each revision increases understanding. The objective is no longer to document existing conclusions. The objective is to discover better conclusions.

    AI Changes the Economics

    Historically, this approach was difficult. Each review cycle required significant effort. Feedback was expensive. As a result, most reports received only a limited number of iterations.

    AI changes that equation. Ideas can be challenged immediately. Alternative interpretations can be explored continuously. Reasoning can be stress-tested throughout the process. The number of feedback loops increases dramatically. What was previously impractical becomes routine.

    A Working Hypothesis

    This leads me to a hypothesis:

    Reports should be developed the way Agile teams develop software.

    Not through a sequence of analysis followed by documentation. But through continuous cycles of exploration, feedback, learning, and refinement. The report is not the final step in the thinking process. The report is part of the thinking process.

    Perhaps the most important lesson from Agile was never about software. Perhaps it was about learning. And if that is true, then assessment reports, strategy documents, presentations, and even books may benefit from the same principle.

    They are not written once understanding is complete. They are iterated into existence.

  • Field Note: Reports Are Iterated Into Existence

    When I started using AI in assessment work, I assumed the primary benefit would be faster writing. I was wrong. The biggest change was not in writing. It was in iteration.

    Traditionally, an assessment report follows a familiar pattern:

    1. Information is collected.
    2. Findings are analyzed.
    3. A report is written.
    4. One or two review cycles take place.
    5. The report is updated.
    6. The report is delivered.

    In this model, the report is largely created before feedback arrives. Feedback improves the report, but only at the margins. The quality of the result depends heavily on the quality of the initial analysis. That is how I have worked for most of my career.

    A Different Process

    Recently, I noticed something fundamentally different. Instead of developing findings in isolation and reviewing them later, I began testing ideas continuously.

    1. A finding would emerge.
    2. I would challenge it.
    3. An alternative explanation would be explored.
    4. Underlying assumptions would be questioned.
    5. Evidence would be re-examined.
    6. The wording would change.
    7. The model would evolve.
    8. The recommendation would be refined.
    9. Then the cycle would start again.

    Not once. Not twice. Sometimes hundreds of times. The report was no longer something I wrote and then reviewed. The report developed through the review process itself.

    The Report Does Not Exist Yet

    The most interesting realization was that many of the strongest conclusions were not present at the beginning. They were discovered along the way.

    The traditional view assumes that the consultant already possesses the insight and simply needs to express it clearly.

    My experience was different. The initial insight was often incomplete. Sometimes it was partially wrong. Sometimes it pointed in the right direction without explaining why. The real value emerged through repeated challenge and refinement. The final conclusion was often something that neither existed nor was fully understood at the start. It emerged through iteration.

    Feedback as a Discovery Mechanism

    This changed how I think about feedback. Feedback is usually viewed as a quality assurance activity. A way to catch mistakes. A way to improve wording. A way to strengthen an already completed deliverable.

    But feedback can also be a discovery mechanism. Each challenge forces deeper thinking. Each alternative explanation reveals hidden assumptions. Each refinement creates opportunities for new insights to emerge. The purpose is not merely to improve an existing idea. The purpose is to discover a better one.

    Why AI Changes the Equation

    The significance of AI is not that it writes reports. The significance is that it makes hundreds of critique cycles economically feasible.

    Historically, each review cycle required another person, another meeting, another scheduling exercise, and another investment of time. As a result, feedback was scarce.

    Today, an idea can be challenged immediately. The cost of critique approaches zero.The number of iterations increases dramatically. And with enough iterations, something interesting happens.The deliverable stops being drafted and polished. It starts being discovered.

    A Working Hypothesis

    This leads me to a hypothesis that I find increasingly difficult to ignore:

    High-quality reports are not written and then reviewed.

    They are iterated into existence.

    The value does not come from producing a perfect first draft. It comes from creating an environment where ideas can be challenged, refined, and transformed repeatedly. The report is not the starting point. The report is what remains after hundreds of iterations have shaped the thinking behind it.