Metric-less Improvement

A Failure Pattern of Measurement Gap and Broken Learning Loop

Summary

Metric-less Improvement is a Failure Pattern in which improvement activities continue without measurement, and only the sense of "it should be getting better" accumulates.

In this Pattern, the improvement motivation or effort itself is not denied. Rather, in an environment where measurement is lacking, continuing improvement itself becomes the most rational behavior, and this addresses the structure in which learning does not hold as a result.

Context

In projects and teams, improvement is constantly demanded.

For performance, quality, development experience, speed, and so on, some awareness of problems is shared, and retrospectives and improvement measures are continuously implemented.

However, in many cases, improvement proceeds without a common indicator showing "what changed by how much." Measurement is costly and requires consensus, so in situations demanding short-term results it is easily postponed.

As a result, improvement is discussed as impression rather than fact, and the activity itself tends to become the goal.

Forces

The main dynamics that generate this Pattern are as follows:

High cost of measurement
Defining and collecting indicators requires time and agreement, so in situations demanding short-term results it is easily postponed.
Pressure of expectations for improvement
Doing nothing is regarded as stagnation, so executing improvement measures is easily valued even when effectiveness is unclear.
High uncertainty
In domains where results do not appear immediately, being unable to measure is itself accepted as an assumption.
Asymmetry of success and failure
While the impact of failed improvement is limited, the accountability for not improving anything becomes heavy.

Failure Mode

Because measurement does not exist, the correspondence between improvement measures and outcomes cannot be verified.

As a result, the following forms of breaking proceed simultaneously:

Presence or absence of effect is not distinguished
Measures that had an effect and those that did not are treated equally.
Judgment converges toward a sense of acceptance
Whether improvement is appropriate is determined by impressions and atmosphere, and comparison or refutation does not hold.
The next improvement does not connect to the previous outcome
Continuing or stopping measures is not based on observation, and the same trial and error is repeated.

Improvement continues, but judgment and behavior are not updated.

Consequences

The learning loop does not hold, and the same trial and error repeats
(Part II: Why It Breaks — Broken Learning Loop)
Judgment criteria are not shared, and decision avoidance is rationalized
(Part II: Why It Breaks — Decision Avoidance)
Procedures and rules proliferate in proportion to improvement activities
(Part I: What Breaks — Responsibility / Operation)
Because actual change cannot be explained, trust is gradually lost
(Part I: What Breaks — State / Boundary)

Countermeasures

The following are not a list of solutions, but counter-patterns for changing dynamics with minimal intervention against Failure Mode.

Rather than whether improvement is appropriate, make explicit what we are trying to observe as an assumption for judgment
Prioritize comparability over precision, and reduce it to a form that can be observed as deltas
Take as the goal not "continuing" improvement, but a state where the judgment to stop can be made

Resulting Context

Improvement is still necessary, and not everything can be parsed numerically.

However, when measurement exists even minimally, the relationship between improvement and outcomes becomes discussable.

As a result, improvement becomes learning rather than an event, and connects to the next judgment.

Appendix: Conceptual References

Feedback, Measurement & Learning
Theoretical background of structures in which observation and judgment updating do not connect.
Systems Thinking & Constraints
Background of dynamics in which local improvement does not lead to overall learning.
Responsibility & Decision
Background of structures in which absence of measurement rationalizes decision avoidance.

Appendix: References

W. Edwards Deming, Out of the Crisis, 1982.
Stafford Beer, Cybernetics and Management, 1959.
Stafford Beer, The Heart of Enterprise, 1979.
Eric Ries, The Lean Startup, 2011.

Summary​

Context​

Forces​

Failure Mode​

Consequences​

Countermeasures​

Resulting Context​

See also​

Appendix: Conceptual References​

Appendix: References​