🌿Retries without backoff turn a blip into an outage
A single failed request retried immediately is harmless. Ten thousand clients retrying a struggling service in lockstep is a denial-of-service attack you wrote yourself.
This is the retry storm: a service slows, timeouts fire, every caller retries at the same moment, and that synchronized wave of load finishes off whatever was still standing. The retries don't help the system recover - they prevent it from recovering.
The fix isn't "retry less." It's to make retries disagree about when to fire. Exponential backoff spreads them out; jitter (randomized delay) breaks the lockstep so the waves never re-form. Add a retry budget - a cap on the fraction of traffic allowed to be retries - so a broad outage can't be amplified into a bigger one.
And retries are only safe when the operation is idempotent. Otherwise you're not recovering, you're duplicating.
Related
Linked from
- 🌿 Distributed Systems
An evolving map of hardwon lessons about building systems that span more than one machine…
- 🌳 Idempotency is a property, not a mechanism
Idempotency describes an outcome applying an operation N times equals applying it once.…
- 🌿 At-least-once delivery makes deduplication your job
Most message systems promise atleastonce delivery: your handler will see every message and…
- 🌿 A circuit breaker protects the caller, not the callee
It's tempting to think a circuit breaker shields a struggling downstream service from load.…
- 🌿 A queue is a loan against future capacity
Putting a queue in front of an overloaded service feels like relief spikes disappear, callers…
- 🌿 Timeouts are a design decision, not a default
An unset timeout is not "no timeout." It's the longest timeout in your stack inherited from…