This morning I woke up with a rather practical question.
Since my laptop is getting older but is still considered rather capable, I wondered how suitable it is to run a language model locally on this machine. And I wondered what a local LLM could be used for. A specific use case came up quickly:
Organizations accumulate enormous amounts of material over time: architecture descriptions, requirements, decision records, source repositories, project documentation, incident histories, assessments, reports, presentations, and wikis. Could a local LLM collect, analyze, and synthesize such material into a body of knowledge that could then be interrogated?
We had already explored the idea that this might reduce the need to keep producing new documents simply to carry existing insights into another context. Instead of anticipating every future question and writing a document to answer it, perhaps we could maintain the knowledge and construct the answer when somebody actually asks.
There is an attractive side effect to this. Documents start aging almost as soon as they are published. If the underlying body of knowledge changes instead, the same question asked six months later can produce an answer based on what is known six months later. The emphasis shifts from periodically republishing knowledge to maintaining something that can continuously be interrogated.
Reconstructing a Product Reference Model
I had already been thinking about this in relation to Product Reference Models. An existing product tends to leave a substantial trail of evidence behind it: architecture, requirements, code, decisions, incidents, operational information, and many other artifacts. Rather than manually recreating an understanding of the product from scratch, perhaps AI could work across those existing sources and help reconstruct a Product Reference Model.
The interesting part would not simply be collecting links. A useful model would also need the relationships between those sources and, particularly, the rationale for their inclusion. Why does this architecture repository matter? What does this incident history tell us about the product? Why is an old decision record still relevant? Where do different sources contradict each other, and where are there white spots for which no useful evidence appears to exist?
A list of artifacts gives us information. The relationships and rationale between them begin to give us understanding.
Such a Product Reference Model would not necessarily need to become another physical document. It could remain connected to its underlying sources and be synthesized when required. If a production incident exposed a previously unknown dependency, or a new business need resulted in changes to architecture and code, the evidence available to the model would change with it. The next interrogation could therefore reflect a different understanding of the product without somebody first having to rewrite a reference document.
From products to capabilities
That thought made me look differently at another Reference Model I had been working with.
In parallel, I had been developing a concept Quality Assurance Capability Reference Model. Unlike the rather distributed Product Reference Model I was imagining, this one is very tangible: a spreadsheet with multiple views of the capability, including dimensions and maturity characteristics, stakeholder perspectives, an evidence landscape, indicators, anti-patterns, and example practices. It is intended to support both capability assessment and capability development.
If a Product Reference Model could become interrogable, why couldn’t this one?
An assessor could ask what stronger capability looks like in a particular dimension, which stakeholder perspectives might be relevant, or what evidence could challenge an emerging interpretation. Someone working with capability development could interrogate the same model from a different perspective. The spreadsheet might remain useful for maintaining the model, but it would no longer have to be the primary way in which people consume it.
There is an additional twist here. Assessment and development do not only use the Reference Model; they also generate evidence about the subject it describes. An intervention may succeed, fail, or work for entirely different reasons than expected. Repeated assessments may reveal a pattern that the existing model does not explain particularly well.
So a Reference Model is not simply a stable body of knowledge from which answers are retrieved. It participates in a loop. The model helps us interpret evidence, while evidence can support, extend, or challenge the model. The Reference Model therefore becomes part of a learning loop.
And that started to connect the local-LLM thought experiment with another idea we had already been approaching: stewardship.
From management to stewardship
Management disciplines such as product management and portfolio management necessarily spend considerable attention on the things associated with their subject: decisions, processes, plans, repositories, ownership structures, and other artifacts. But even when those things are managed well, the organization can still gradually lose its accumulated understanding of the subject itself.
Stewardship suggests a different concern:
Can the organization continue to understand a subject as people leave, evidence accumulates, and reality changes?
That question exposes something slightly awkward. Where does persistent organizational understanding actually exist?
Some of it exists in people’s heads, some in documents and systems, some in evidence, and a great deal of it exists in the relationships between those things. We had already suspected that Reference Models might play a foundational role in stewardship, but I think the reason for that is now becoming clearer.
Reference Model = Understanding?
We normally describe a Reference Model as a representation of our understanding.
Perhaps that distinction is unnecessary.
If a Reference Model maintains what matters about a subject, how things relate, why they matter, what evidence supports them, where uncertainty remains, and how all of that changes as we learn, then what separate thing are we calling “the understanding”?
Perhaps the Subject Reference Model is the maintained organizational understanding of the subject.
That does not mean every fact or artifact needs to be contained inside it. The Product Reference Model demonstrates why. Much of its detailed knowledge can remain in the sources to which it refers. Knowing that a source matters, understanding why it matters, and maintaining its relationships to other parts of the subject can itself be part of the model.
A Capability Reference Model may encode much more of its domain knowledge explicitly. The physical forms are different because the subjects are different, but the fundamental function may be the same.
A Product Reference Model is maintained product understanding. A Capability Reference Model is maintained capability understanding.
More generally:
Subject Reference Model = Subject Understanding
And that connects to a problem we have encountered from an entirely different direction.
Understanding debt
We have been using the term understanding debt for something that documentation alone does not seem to solve. Organizations are extraordinarily good at accumulating information while gradually losing the context, relationships, and rationale that make the information understandable.
The architecture document remains, but nobody remembers why the architecture was chosen. The old system remains, but nobody is entirely sure what depends on it. A practice continues long after its original purpose has disappeared from organizational memory. A decision can still be found, but the alternatives that were considered and the reasons they were rejected are gone.
Nothing necessarily went undocumented. There may be more documentation than ever.
What disappeared was the understanding that connected it.
If the Subject Reference Model is maintained understanding, then understanding debt becomes easier to describe. It is the gap between what the organization needs to understand about a subject and what its maintained Subject Reference Model actually allows it to understand.
Seen that way, Reference Models become rather more important than useful assessment frameworks, maturity models, or knowledge structures. They become a mechanism for allowing organizational understanding to persist beyond the people who happen to hold it at a particular moment.
AI may make Reference Models more important, not less
It would be easy to assume that increasingly capable AI makes Reference Models less important. If an AI can read everything, why bother maintaining a model?
I suspect the opposite may be true.
Giving AI access to everything gives it information: twenty years of documents, code, incidents, presentations, abandoned decisions, obsolete descriptions, and contradictory claims. Without some maintained conception of what matters and how it fits together, the AI has to reconstruct an understanding from that history every time we ask a question.
A stewarded Reference Model gives that information meaning and continuity. AI, in turn, may make it practical to reconstruct such models from existing evidence, interrogate them rather than continually publishing new documents, and help keep them aligned with new evidence as the organization learns.
The interesting role of AI may therefore not be producing more organizational knowledge at all. It may be helping us maintain the understanding underneath it.
If Subject Reference Model = Subject Understanding, then stewardship of Reference Models is stewardship of organizational understanding.
And that may make the Reference Model our best antidote yet to understanding debt.
Leave a Reply