Fallbacks, Authority, and Failure Paths

A system is not fully designed when only the success path is defined.

Ask what happens when:

• input validation fails;
• the model is unavailable;
• latency exceeds the decision deadline;
• confidence falls into an uncertain range;
• monitoring detects drift;
• the human override path is unavailable.

These are not edge cases to ignore until deployment. They are part of the system behavior.

Safe failure is an architectural choice

A fallback might:

• refuse automatic action;
• use a simpler non-AI rule;
• preserve the last known safe state;
• request human review;
• stop the process;
• continue with reduced functionality.

The correct choice depends on the application. A recommendation system and a moving robot should not share the same failure policy simply because both contain neural networks.

Suppose a cloud model normally answers in 80 ms, but the application requires a decision within 150 ms. A network incident pushes response time to 900 ms.

A weak design waits indefinitely because “the model is accurate.”

A stronger design treats the latency budget as part of the contract and enters a documented fallback when the deadline is exceeded.

Human authority must be real, not decorative

Saying “a human is in the loop” is not enough. Ask:

• What information does the human receive?
• How much time do they have?
• Can they actually override the action?
• Is the override tested?
• What happens if the human is unavailable?
• Is responsibility clearly assigned?

Human control is a system component with its own failure modes.

Before the lab

Choose one component from the system map and describe one failure that the model evaluation from Week 14 would not detect.