Blog Retrospective #6

I have now reached 120 posts.

In the previous retrospective, I observed that the Body of Knowledge had begun behaving like a learning system. Practice Notes generated theory, theory generated approaches, and those approaches generated new observations that could flow back into the Practice Notes.

During the latest twenty posts, that process continued.

But something else happened.

The inquiry increasingly turned towards the reliability of the understanding we produce.

Not just what we know about products, capabilities and organizations, but how we know it, what our evidence actually supports, and whether our models might themselves be wrong.

What Happened?

The latest cycle explored several apparently different subjects.

Some notes examined transformation, projects and interventions. Others explored product reference models, contextualization, organizational knowledge and AI. A few returned to everyday observations, including watches and model railway automation.

The final group emerged directly from developing a standard approach to capability assessment.

That work raised questions about evidence, contradictions, assessment confidence, sampling and the quality of assessments themselves.

At first glance, these might seem like separate lines of inquiry.

But I think they share an underlying concern.

How do we establish that the mechanisms we use actually achieve their intended purpose?

A project can deliver successfully without producing the intended change.

An intervention can be completed without improving organizational capability.

A product can be extensively documented without being properly understood.

An assessment can contain considerable evidence without producing a justified conclusion.

The presence of a mechanism doesn’t establish that the mechanism works.

That observation has appeared throughout the blog. During this cycle, it became particularly difficult to ignore.

From Reference Models to Context Layers

One development I hadn’t anticipated was the connection between reference models and organizational knowledge.

Earlier work on reference models focused primarily on assessment.

A reference model provides a conceptual structure against which we can examine a subject. Contextualization helps us interpret that structure in relation to a particular organization, capability or product.

But what if contextualized reference models could serve another purpose?

Organizations already possess enormous Bodies of Knowledge. They contain documents, decisions, observations, assessments and accumulated experience.

AI can make much of this information accessible.

But accessibility isn’t necessarily understanding.

Knowing that several documents mention requirements management doesn’t automatically establish how requirements management relates to organizational capabilities, product characteristics, decisions or outcomes.

Perhaps contextualized reference models could provide part of the structure needed to interpret those relationships.

That led to the idea of context layers: representations connecting generic concepts to particular organizational subjects and the evidence available about them.

An interrogable organization might then be able to ask questions that require information from many different sources.

Not merely retrieving documents, but investigating relationships.

There is still an important open question.

How much context must be explicitly modeled, and how much can AI reliably reconstruct when needed?

I don’t know.

But the possibility changes how I think about reference models.

They might be useful not only as instruments for assessment, but also as part of the infrastructure through which an organization understands itself.

When the Assessment Approach Started Asking Questions

Perhaps the most interesting development came from working on the capability Assessment Approach.

The approach was intended to operationalize ideas already developed in the Practice Notes and Continuity work.

But making those ideas operational exposed questions that deserved attention in their own right.

One concerned contradictions.

What should an assessor do when interviews, documentation, observed behaviour and outcomes disagree?

The obvious response is to reconcile the evidence.

But perhaps the disagreement itself reveals something important.

Why does management believe a process works when outcomes repeatedly suggest otherwise?

Why does a team achieve excellent results despite apparently weak practices?

Perhaps the contradiction exposes a weakness in our understanding rather than a weakness in the evidence.

Another question concerned confidence.

An assessment might conclude that a capability is mature, but how strongly does the evidence support that conclusion?

A Level 4 capability assessment based on limited evidence is different from a Level 2 assessment supported by extensive observation.

The assessed condition and our confidence in that condition are different dimensions.

Then came sampling.

An assessment can examine evidence rigorously and still produce a misleading conclusion if the evidence represents only a favourable part of the organization.

We might establish that one team performs effectively without establishing that the organization can reproduce that performance elsewhere.

Finally, the assessment itself became the subject of inquiry.

An assessment report may claim to be evidence-based.

But what justifies that claim?

Were the right questions asked? Was the evidence representative? Were contradictions investigated? Were conclusions stronger than the evidence warranted?

That produced Who Assesses the Assessment?

By the time I finished that note, I was beginning to wonder whether I would ever dare conduct another assessment.

When Disciplined Curiosity Becomes Indiscipline

That reaction was useful.

Each additional question about assessment quality seemed reasonable.

But taken together, they threatened to make assessment impossibly demanding.

Every conclusion could be challenged.

Every sample could be questioned.

Every interpretation could have alternatives.

And every assessment of an assessment could presumably require another assessment.

At some point, the pursuit of justified understanding can become an obstacle to using that understanding.

This brought me back to Disciplined Curiosity.

The purpose of assessment is not to eliminate uncertainty.

It is to develop sufficiently justified understanding for the decisions we need to make.

The necessary rigor depends on the purpose of the assessment, the consequences of getting it wrong and the resources available.

Further investigation has a cost.

So does acting on insufficient understanding.

Professional judgment includes knowing when additional inquiry is worthwhile.

Perhaps that is something the Assessment Approach still needs to express more clearly.

Moving Upstream

Another recurring theme became more explicit during this cycle.

A professional can perform an assignment correctly without achieving the purpose that motivated it.

A tester can thoroughly test the wrong requirements.

A project manager can deliver a project that produces no meaningful change.

A consultant can implement an improvement that doesn’t address the underlying problem.

This led to the question of whether moving upstream is itself a professional quality.

Perhaps professional competence involves more than executing an assignment.

It also involves understanding why the assignment matters and recognizing when successful execution may not achieve its intended purpose.

The AI-related notes added another perspective.

When AI removes some of the friction involved in implementation, more attention can move towards intent, evidence and judgment.

That doesn’t make implementation knowledge irrelevant.

But it may change where human attention creates the most value.

Looking back, moving upstream has been a recurring movement throughout the blog.

Testing led to quality.

Quality led to organizational capabilities.

Capabilities led to organizational purpose.

Projects led to interventions and their effects.

Documentation led to understanding.

And assessment results led to questions about the justification for those results.

The subjects changed.

The underlying curiosity remained remarkably consistent.

What Surprised Me?

The biggest surprise was how naturally the Assessment Approach generated new Practice Notes.

I had expected the approach to apply existing ideas.

Instead, operationalizing those ideas exposed distinctions that hadn’t previously received sufficient attention.

That is encouraging.

It suggests that the relationship between Practice Notes, theory and application can work in both directions.

But I also noticed a potential danger.

The ideas are becoming increasingly coherent.

Almost every new observation can be connected to something already present in the Body of Knowledge.

That makes synthesis easier.

It also creates the risk that the framework becomes too good at explaining everything.

If every observation can be interpreted as supporting the theory, how would we recognize evidence that challenges it?

The recent note Contradiction Is Information applies just as much to this Body of Knowledge as it does to organizational assessments.

Perhaps the next important development won’t be another concept that fits neatly into the existing framework.

Perhaps it will be an observation that doesn’t fit.

Looking Back at the Whole Blog

At 120 posts, the scope of the inquiry has changed considerably.

It began largely with questions about software quality, testing and assessment.

Those questions led towards organizational capabilities and the conditions required to produce reliable outcomes.

From there, the inquiry expanded towards products, portfolios, Subject Stewardship and Continuity.

Then came organizational knowledge, the preservation of decisions and evidence, and the possibility of making accumulated knowledge interrogable.

The latest cycle has increasingly examined the quality of the understanding produced by those mechanisms.

There is an interesting symmetry here.

The early notes questioned whether testing activities were sufficient to establish product quality.

The recent notes question whether assessment activities are sufficient to establish justified understanding.

Both challenge the assumption that performing an activity demonstrates that its purpose has been achieved.

And both lead to questions about the capabilities and conditions needed to produce the intended outcome.

Perhaps the blog hasn’t moved as far from its original subject as it sometimes appears.

The subject of Quality Management has simply become broader.

What Concerns Me?

I see two concerns worth carrying into the next cycle.

The first is that abstraction may be moving faster than practical application.

Some of the newer ideas, particularly around context layers and interrogable organizational knowledge, remain propositions rather than demonstrated approaches.

They need to encounter reality.

The second is that the growing coherence of the Body of Knowledge may make it increasingly difficult to challenge its assumptions.

I don’t want the Practice Notes to become a collection of arguments explaining why the existing theory is correct.

They should remain a place where observations can challenge the theory, including observations that make earlier conclusions uncomfortable.

That is also why I think the distinction between observations, interpretations and hypotheses deserves continued attention.

Not everything that follows logically from an idea has been demonstrated in practice.

What Next?

I don’t think the next step necessarily requires another major theoretical synthesis.

I would rather see what happens when some of these ideas are applied.

What would a contextualized product reference model reveal about an actual product?

Would context layers improve an AI assistant’s ability to investigate organizational questions?

Would the Assessment Approach expose weaknesses or contradictions that a conventional assessment might overlook?

And perhaps most importantly, what would happen when one of these approaches failed to work as expected?

Those failures might generate more valuable Practice Notes than another successful conceptual synthesis.

A Final Observation

The first retrospectives were largely concerned with the writing process and the ideas emerging from it.

Later retrospectives examined how the Practice Notes began connecting, generating theory and feeding applications.

This retrospective has turned towards something different.

The Body of Knowledge is increasingly examining the mechanisms through which it develops understanding.

That feels like progress.

But it also creates a responsibility.

If I argue that organizations should challenge their assumptions, investigate contradictions and evaluate the evidence supporting their conclusions, the same expectations should apply to my own work.

The objective cannot simply be to develop an increasingly coherent theory.

It must remain possible for experience to change that theory.

Perhaps the most important question for the next retrospective is not what new ideas have emerged.

It is what I have learned that forced me to reconsider something I previously believed.