Chaos Engineering in Production: Designing Fault-Injection Experiments with Controlled Blast Radius and Automated Rollback

Abstract

Distributed systems fail in combinations no designer anticipated, and the failures that matter most are precisely the ones that integration tests never exercise: a dependency that slows rather than stops, a region that partitions, a retry storm that turns a small fault into an outage. Chaos engineering confronts this directly by injecting controlled failures into running systems, including production, to discover weaknesses before an uncontrolled failure does. The discipline is often misread as breaking things at random. It is the opposite: a scientific method applied to system resilience, in which an engineer states a hypothesis about steady-state behaviour, injects a specific fault, and measures whether the system holds up. This paper presents chaos engineering as an experimental practice and concentrates on the two mechanisms that make it safe enough to run on live traffic. Blast-radius control limits the fraction of users and infrastructure an experiment can affect, beginning with a single host and escalating only as confidence grows. Automated rollback continuously compares the live system against its expected behaviour and aborts the experiment the instant the deviation exceeds a threshold, bounding the worst case. We trace the practice from its origin in the Netflix Simian Army to today's platforms, describe how to design an experiment, and argue that deliberately injecting failure is now a core technique of reliability engineering rather than a stunt.