Building resilient systems without excess complexity
Patterns and trade-offs for designing systems that stay reliable under change without over-engineering every layer.
Resilience is usually framed as a list of mechanisms: retries, circuit breakers, bulkheads, queues. In practice the hard part is deciding which of them you actually need, because every mechanism you add becomes a component that can fail on its own.
Start from the failure you have seen
The cheapest resilience work comes from failures you have already observed. Before adding a new layer, write down the last five incidents and ask which mechanism would have changed the outcome. Most teams discover that two or three patterns cover the majority of their real failures.
Budget complexity like you budget latency
Every abstraction has a carrying cost: onboarding, debugging, and upgrade paths. Treat complexity as a budget with a fixed size. Spending it on a message queue means not spending it on a second cache tier.
Make degradation the default path
Systems that degrade well share one property: the degraded path is exercised continuously, not only during incidents. Serving cached results for 1% of traffic all the time is more reliable than a fallback that runs for the first time at 3am.
What to keep
- One retry policy, applied consistently, with jitter and a hard cap
- A single place that owns timeouts per dependency
- Health signals that reflect user-visible behavior, not process liveness
- A documented degraded mode for every user-facing surface
Everything else should earn its place by pointing at a failure you have actually seen.
Topics