Invert industrial automation design around failure first

By Amara Osei

The proposed failure-first exercise asks teams to name 10 ways one automation cell can fail on a bad shift, then protect the 3 scenarios that cause the most harm. That sequence, rather than a feature list or polished control narrative, becomes the design brief.

What does inversion change?

The method has a lineage. Carl Jacobi told his students, “Invert, always invert.” Charlie Munger later made the idea familiar outside mathematics: ask what would make you fail, then refuse those things. In a plant, that discipline beats another slide of happy-path logic.

Dr. Krisztián Galánthai Fekete asks operators, maintenance staff and engineers: “If this line must fail, how will it fail?” The answer must fit the actual floor: this product, this line, and a Friday night shift with fewer hands and a half-empty spare-parts locker.

Before anyone draws a sequence or names a programmable-logic-controller (PLC) tag, the team writes a short, deliberately ugly set of failure scenes. It is not a risk register left in a shared folder. It is something people can picture.

Representative scenes include:

• A sensor stays high after wash-down.
• A conveyor keeps running after a jam because the detector was temporarily bypassed.
• A recipe download partly succeeds, leaving the batch half old and half new.
• A valve opens while a broken feedback wire makes the human-machine interface (HMI) show closed.

The team ranks those scenes by frequency and damage, then designs against the worst first. Features that do not protect a named failure can wait; optimism, apparently, has no schedule.

Putting the list on the design-review wall changes the meeting. Instead of touring screens, the group walks through ways the line can injure people, scrap product or stop a shift.

A named failure outranks a feature request.

How much room should the design leave?\n

Inversion tells the team what to avoid. A margin of safety tells it how much room to leave when that reasoning is wrong.

Benjamin Graham used the phrase in investing: leave a gap below an estimated value so an error does not wipe out the result. The engineering analogy is blunt: engineers do not design a bridge rated for 10 tonnes to fail at 10.5.

On an automated line, the margin can look like waste during calm production:

• An interlock (a control rule that blocks an unsafe step) rejects a cycle before the part is damaged.
• A short buffer (holding space) absorbs a brief upstream stop without a cascade.
• A second confirmation protects a critical state when sensors get dirty.
• A trusted safe stop gives operators a better choice than jumping a wire.

The same review should ask what guarantees failure and where the design can absorb error without injury or scrap. If the second answer rests on hope, the team has expressed a wish, not specified a margin.

The margin must absorb a mistake, not merely describe one.

Who recovers a failed line?

Hardware and software fail. People recover them, or they do not.

The fastest operators do not memorize every alarm code. They understand what each interlock protects, which state is safe to leave the line in, and which quick fix creates a worse failure later. Their recovery judgment therefore belongs in the design, not merely in a training manual.

When operators help write the failure list, they recognize the jam-bypass story under pressure. Training should follow that list, not only the happy path: walk the bypasses that must never become permanent, explain each stop in the words the night shift uses, and leave a written path back to a known-good state (a verified safe configuration).

Recovery design belongs to operators, not only engineers.

When does a temporary override become permanent?

Fekete recalls a night crew whose “temporary” override had lived for weeks. It silenced a noisy sensor on a busy day, but no failure list named that path. Afterward, every review asked a sharper question: “Which temporary changes are still alive, and which failure do they hide?”

The next override received an expiry date on the whiteboard. Bureaucracy, for once, did useful work.

Temporary changes need an owner and an end date.

Can one cell test the method?

One cell or one sequence is enough to test the method:

• Write 10 ways the automation can fail during a bad shift, using plant language rather than vendor language.
• Mark the 3 that hurt most, then design or revise the system against those first.
• Name one margin for each: an extra confirmation, buffer, interlock or clear safe state.
• Have one operator walk the recovery for each of the 3, and change the design until that walk is boring.

The method supplies no accuracy class, bandwidth, resolution, sampling rate, calibration interval or standard revision, and it reports no measured product performance. It offers operational guidance, not a vendor datasheet claim or quantified reliability result.

The evidence a team can inspect is the ranked list, chosen margins and completed recovery walks. Their value appears only when the method survives contact with a short-handed night shift.

The walkback must be boring because the fault mode is real.

What must adoption prove?

Failure-first design does not make automation pessimistic; it makes it honest. Failure modes on a real floor are easier to see than a perfect success path drawn in advance, and trust comes from showing that failure shaped the design.

If teams institutionalize the method, each review should leave a ranked failure list, named margins, operator recovery walks and visible temporary overrides. Without those artifacts, inversion becomes another slide in the shared folder, precisely what the method rejects.

Until that becomes routine, the method is advice. Required, tested and boring, it is engineering. So: will teams require an operator to complete the recovery walk for each of the 3 worst failures before accepting the cell for routine operation?