Systems, Decisions and Robust Design

A reading-and-practice path through cybernetics, systems dynamics and failure studies for seeing behaviour as the output of structure and designing so that being wrong is survivable.

Systems thinking begins with a reversal. Do not ask who is to blame for a behaviour. Ask what structure would produce this behaviour no matter who occupied the seats. The second half is design: build so that when your understanding is wrong, the damage stays small and the error shows.

Core vocabulary

Meadows's Thinking in Systems gives the plainest introduction. A stock is an accumulation: water in a bathtub, money in an account, unread messages. A flow changes a stock. A feedback loop carries information about a stock back to the flows. A balancing loop pushes toward a target, as a thermostat does. A reinforcing loop amplifies, as compound interest or a bank run does.

Delays between action and result cause many oscillations. Meadows's own example is a hotel shower that "took at least a minute to respond" to her twists of the tap. With only delayed information, she writes, "you will overshoot and undershoot". Boundaries are choices about what to include, and a wrong boundary hides the real cause.

Her essay on leverage points ranks the places to intervene. The paradigm behind a system, and the goals and rules that grow from it, beat the numbers people usually adjust. Stocks, flows and loops are Established as a framework. The exact ranking is Provisional: a practitioner's synthesis, not a tested result.

The founders

Wiener (Cybernetics, 1948) treated control and communication in animals and machines as one subject. A system steers by comparing its state to a goal and correcting the error.
Ashby (An Introduction to Cybernetics, 1956) stated the law of requisite variety. A regulator copes only if it has at least as many responses as the disturbances have distinct effects. Demonstrated within its formal setting. Applying it loosely to organisations is analogy, useful but not proven.
Simon (The Sciences of the Artificial) placed designed things between an inner environment and an outer one. Real agents "satisfice": they accept a good-enough option because search is costly.
Bertalanffy (General System Theory) proposed that similar organising principles recur across biology, engineering and society. How far that unity goes is Aporetic.

Beyond "root cause"

The phrase "root cause" is sometimes right and often misleading. Many real failures involve several individually acceptable conditions interacting. Redundant parts can share a hidden dependency, such as one power source, one shared configuration or one vendor. Then the backup fails with the primary.

Incentives distort metrics too. When a measure becomes a target, people optimise the measure. This is often called Goodhart's law (a saying, not a theorem). Keep fact and analogy apart here. The bathtub model is a fact about accounting identities. Calling an organisation "a feedback system" is a lens.

Why systems fail

Doerner's The Logic of Failure reports experiments in which people ran simulated towns and villages. They failed in repeatable ways. They ignored side effects, changed one variable at a time without watching delays, and gave up on goals too early.

Perrow's Normal Accidents (1984) argued that some systems will eventually produce accidents no operator could easily foresee. They are both tightly coupled (little slack between steps) and interactively complex (opaque, with unexpected interactions). Established as an influential framework drawn from case studies. Leveson's systems-theoretic approach (Engineering a Safer World) disputes some of its claims, so the argument is still live.

Vaughan's study of the 1986 Challenger launch decision shows how a group can slowly accept deviation as normal. For financial panics, Kindleberger's history traces a recurring pattern: rising credit, a shock, then a rush to liquidity.

Coordination and institutions

Schelling's Micromotives and Macrobehavior shows how mild individual preferences can add up to sharp collective patterns nobody intended. Hayek's 1945 essay argues that the knowledge an economy needs is dispersed among individuals, and prices carry it. Adam Smith's "invisible hand" (Wealth of Nations, Book IV, ch. 2) makes a related point about unintended order.

Ostrom's Governing the Commons answers a different pair of rivals. She documents communities that managed shared resources through locally evolved rules. That challenges the claim that only privatisation or central control works.

On networks, Granovetter (1973) found that weak ties carry novel information between clusters. Watts and Strogatz (1998) showed that a few long-range links shorten paths across a network. The mathematics of nodes and edges is in Discrete Mathematics and Graphs, and feedback and queues in Systems Thinking.

Diagnosis and design

Diagnosis reasons backward from an outcome to causes: "why did this happen?" Design reasons forward from requirements and failure modes. It asks what structure will keep producing the desired result despite variation and faults. The skills overlap, but they are different habits. Practice for the first is in Evidence-Based Troubleshooting.

Design begins by asking, for each component, "what happens when this fails or lies?" A related lesson from Stale Information and Dependable Reports: any stored copy of the world, such as a cache or a dashboard, can be out of date while still looking fine, and must be designed accordingly.

How much more information is worth

Bounded rationality means you will never have all the facts. The useful question is whether more information could change the decision, at a cost worth paying. Three factors set the answer: reversibility (an undoable change needs less certainty), stakes, and cost of delay. Reasoning under such limits is the subject of Judgment Under Uncertainty.

Robust design

When your model may be wrong, prefer choices that work acceptably across several possible worlds:

Redundancy with independent failure modes.
Staged rollout, so a faulty change reaches few users first.
Backups that are actually restored in tests.
Rollback plans prepared before the change.
Segmentation, so a failure or intrusion cannot spread everywhere.
Graceful degradation, so the system loses features before it loses everything.

These are engineering tools and also epistemic tools, since each one admits that the designer might be mistaken. After any action, verify. Did the symptom actually go away? Did anything else change? Does the improvement persist? An improvement that coincides with your change does not yet prove the change caused it.

What keeps a system open to correction

Every healthy system has a way to detect error and act on it. Carry one question into every institution you join or build: what keeps this system from closing itself to correction? Candidates include honest measurement, protected dissent, blameless review of failures, and leaders who value correction over looking right. The same attitude runs through Peirce's fallibilism, taken up in Reasoning for What Ends, and the wider questions of Technology, AI and Human Judgment.

Where this could be wrong

Strongest objection. "Find the structure, not a culprit" can dissolve accountability. If the system wrote the script, nobody answers for anything. Meadows herself allows an exception. She calls changing the players a low-level intervention, except at the top, where one player can change the system's goal.

Best reply. Looking for structure does not deny choice. A blameless review still asks who knew what, and why acting on it seemed reasonable. It shifts the aim from punishment to prevention. People still answer for what they did; the review just does not stop there.

What would settle it. Comparisons of structural reviews with blame-focused ones, measured by repeat failures. This page has not checked that evidence, so the claim stays Provisional.

Try this

One diagram, three cases. Draw a causal-loop diagram for a financial panic, a technical outage and a public tragedy of your choice. Show stocks, flows, reinforcing and balancing loops, and delays. Mark where information was delayed, where coupling was tight and where incentives pointed the wrong way. Compare the three diagrams for shared structure.
Blameless postmortem. Take a recent failure from your own life or work. Write what happened, what made the action seem reasonable at the time, and which structural change would make it less likely. Name no culprit.
Failure inventory. List five components of something you depend on. Write what happens when each one fails silently.