Imagine assessing an organization’s testing capability.
Management explains that testing is integrated throughout development. Team members confirm that testing activities are part of every sprint. Process documentation describes how testing should be performed.
Everything appears consistent.
Then we examine the actual work.
Testing frequently happens at the end of the sprint. Defects are discovered late. Releases are delayed because important problems remain unresolved.
Now we have a contradiction.
What should we do with it?
We could conclude that the organization isn’t following its defined process.
But perhaps the more interesting question is:
Why does the organization believe testing is integrated when its actual behaviour suggests otherwise?
The contradiction may tell us more about the organization than the original evidence did.
When Evidence Doesn’t Agree
Assessments often involve collecting information from different sources.
We interview people, examine documentation, observe behaviour and investigate outcomes.
Ideally, these sources help us develop a coherent understanding of the subject being assessed.
But sometimes they disagree.
Management describes one reality. Employees describe another. Documentation describes a third. Actual outcomes suggest something else entirely.
It’s tempting to treat these inconsistencies as problems to resolve.
Perhaps someone misunderstood the question. Perhaps the documentation is outdated. Perhaps one observation was unrepresentative.
Any of these explanations might be correct.
But we shouldn’t assume that disagreement is merely noise.
A contradiction may reveal something that none of the individual observations could tell us.
The Process That Works Without Being Followed
Consider another example.
An organization has established a detailed development process.
An assessment discovers that one team routinely deviates from it.
At first glance, this appears to indicate weak process compliance.
But the team consistently delivers good results.
Now we have another contradiction.
The prescribed process isn’t being followed, yet the intended outcomes are being achieved.
Perhaps the team has developed a more effective way of working.
Perhaps the prescribed process contains unnecessary activities.
Perhaps the team succeeds because an exceptionally capable individual compensates for weaknesses elsewhere.
Or perhaps its apparent success depends on conditions that won’t last.
We don’t know yet.
But simply recording noncompliance would miss the opportunity to understand what is happening.
The contradiction gives us a reason to investigate.
The Danger of Making Evidence Agree
There is a subtle risk in the way we synthesize assessment findings.
We want to produce a coherent conclusion.
Conflicting observations complicate that task.
So we may be tempted to reconcile them into a single explanation.
Management’s description becomes the intended process. Employees’ accounts become variations in implementation. Unfavourable outcomes become exceptions.
The resulting assessment may sound perfectly reasonable.
But have we explained the contradictions?
Or merely organized them into a narrative that makes them less visible?
Coherence is desirable, but it isn’t proof of correctness.
An explanation can be internally consistent while failing to account for important observations.
Sometimes the most valuable assessment finding is precisely that our current explanation doesn’t adequately describe what we observe.
Contradiction as a Starting Point
This suggests a different approach.
When significant evidence contradicts our current understanding, we should become curious.
Why do these observations disagree?
Are they describing different conditions?
Do different parts of the organization operate differently?
Are people reporting intended behaviour rather than actual behaviour?
Has something changed that our existing understanding hasn’t captured?
Or is our original interpretation simply wrong?
The objective isn’t to pursue every inconsistency indefinitely.
Some contradictions are trivial. Others result from unreliable evidence or differences in terminology.
But significant contradictions deserve investigation, particularly when they challenge conclusions that would otherwise influence important decisions.
This is where assessment becomes more than checking conformity against a reference model.
It becomes an exercise in disciplined curiosity.
What Contradictions Reveal
A contradiction doesn’t automatically tell us which observation is correct.
Two conflicting accounts may both be partly correct.
A process may work under normal conditions but fail under pressure.
A capability may be effective in one team but absent elsewhere.
A product may satisfy its documented requirements while failing to achieve its organizational purpose.
In each case, the contradiction may indicate that we have overlooked an important distinction.
Perhaps our assessment is too general.
Perhaps we’re combining observations from different contexts.
Perhaps the reference model needs to be contextualized differently.
Or perhaps we have mistaken a shared organizational belief for demonstrated capability.
The contradiction doesn’t provide the answer.
It tells us that our current understanding may not be sufficient.
When the Model Is Wrong
There is another possibility worth considering.
What if the contradiction isn’t between different pieces of evidence?
What if it’s between the evidence and the model we’re using to interpret it?
A reference model describes concepts and relationships we believe are relevant to understanding a subject.
But the model is not reality.
Suppose an assessment repeatedly encounters effective practices that don’t fit the model’s expectations.
We could interpret those practices as deviations.
Or we could ask whether the model adequately represents the capability we’re trying to understand.
This doesn’t mean abandoning the model whenever evidence becomes inconvenient.
It means recognizing that models are themselves open to examination.
A useful reference model should help us interpret observations.
But observations should also be capable of challenging the model.
Otherwise, assessment risks becoming a mechanism for confirming what we already believe.
Learning From What Doesn’t Fit
Perhaps one of the most important qualities of an assessment is not how much evidence it collects, or how consistently it applies its criteria.
It is whether the assessment remains capable of discovering that its initial understanding was wrong.
Confirming evidence can strengthen an explanation.
Contradictory evidence can reveal where that explanation breaks down.
Both matter.
But they contribute differently to our understanding.
If we only seek agreement, we may produce assessments that are coherent, consistent and reassuring.
If we investigate meaningful contradictions, we may discover something we didn’t know to look for.
Contradiction is not necessarily a problem with the evidence.
Sometimes, it is the evidence that our understanding needs to change.
Leave a Reply