Defence in Depth
Design as if each control has already failed, and prefer controls that work when someone forgets.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
If one control fails, what is left — and is the next layer actually independent of the first?
After a near miss, the team wants to know which parts of the system would have contained the damage, and where a single mistake is still enough.
Add more controls. A WAF at the edge, a scanner in CI, another validation library, an extra review step. More layers means more safety.
Layers that fail for the same reason are one layer. Two validation libraries called from the same middleware are both bypassed by the route that skips the middleware.
- Layers that fail for the same reason are one layer. Two validation libraries called from the same middleware are both bypassed by the route that skips the middleware.
- Controls that depend on vigilance decay at exactly the moment they matter: during an incident, under deadline pressure, on the endpoint added by someone new.
- Every added layer costs latency, operating effort and alerts. Past a point the cost is paid in attention, and attention is the resource an incident consumes fastest.
- A visible control produces confidence out of proportion to what it does. "We have a WAF" has ended more security discussions than it has stopped attacks.
What is actually happening
- Depth is about independence, not count. Two controls help when the failure of the first does not imply the failure of the second — different mechanism, different layer, ideally a different team's change to break.
- Layers do different jobs. Prevent stops the action; contain limits what a successful action reaches; detect tells you it happened; recover restores state. A stack of four preventive controls has no containment and no detection.
- The strongest layers are structural: an API that cannot express the unsafe operation, a database user without the privilege, a network that cannot route to the destination. They hold when nobody remembers the rule, which is the only condition worth designing for.
- The weakest are procedural: a checklist, a review convention, a wiki page. They are not worthless — they are how new rules start — but they degrade with team growth and turnover, and they should be converted into structure as soon as the rule is stable.
- Every control has a failure mode of its own, and the direction matters: a control that fails open under load or timeout adds nothing precisely when the system is stressed (Fail Open vs Fail Closed in Security Engineering).
Four jobs, not four copies of one job
The useful audit is not "how many controls do we have" but "which of the four jobs does each one do". Teams routinely find they have four preventive controls, no containment and no detection — which means every failure is total and none of them is noticed.
Read the table against one concrete threat at a time. For SQL injection: parameterized queries prevent, a restricted database user contains, database error-rate alerting detects, and point-in-time recovery restores. Four layers, four mechanisms, four different ways to fail — that is depth.
| Job | Example for injection | Example for a leaked credential | Fails when |
|---|---|---|---|
| Prevent | Parameterized queries only; no raw string path (SQL Injection) | Non-printable secret type; field allow-list in the logger | Someone reaches an escape hatch, or adds a code path outside the rule |
| Contain | Database user with no DDL and access to one schema | Short-lived, narrowly scoped credentials per service | One shared privileged credential across services or environments |
| Detect | Alert on SQL syntax errors and on unexpected query shapes | Canary credential search over log storage; provider-side use alerts | Alerts go to a channel with no owner |
| Recover | Point-in-time restore, audited write history | Rotation runbook that is rehearsed, not written | Rotation has never been tested and takes six hours the first time |
One attempt, four independent layers
Trace a single hostile request through a system that has depth, and notice that each barrier is owned by a different mechanism: application code, database grants, network policy, and the observability stack. No single change by any one person removes more than one of them.
That property — a different change is needed to break each layer — is the practical definition of independence, and it is the one to check when someone proposes a new control.
Which layer to add next
The question is never "is another layer good". It is "which failure is currently unlayered, and what is the cheapest independent control for it". Answering it in that order stops the drift towards tool accumulation.
Note that the cheapest answer is often not a new system: converting a procedural rule into a structural one adds a layer without adding an operational surface, because the enforcement lives in code that already runs.
What does this control cover that nothing else does?
when A rule exists and is enforced by review — a lint rule, a type, a repository API can enforce it instead.
cost Engineering time and some lost flexibility. Almost always the best value available.
when Prevention exists but a successful bypass reaches everything: one DB user, no egress policy, no tenant scoping.
cost Configuration surface and a new class of misconfiguration outage.
when A control exists and you would not know if it stopped working.
cost Alert volume and an owner. Without the owner this is not a layer.
when The first is genuinely likely to fail and the second fails differently.
cost Latency and complexity, and correlated failure if it reads the same inputs — check that before building it.
when A layer nobody can explain, or one whose alerts are routinely ignored.
cost The argument you will have about removing it, against the attention it returns.
How to build it
Most important first.
- For each control, write the sentence: "if this fails silently, what still stops the damage?" If the answer is nothing, you have a single point of failure regardless of how many other controls exist elsewhere.
- Prefer making the unsafe thing unrepresentable over detecting it. A repository whose query methods take parameters only, a fetch client that always resolves and pins (SSRF — When the Backend Fetches a URL), a secret type that cannot be printed (Secrets in Logs).
- Put containment underneath every preventive control: least-privileged database users under parameterization, a credential-free sandbox under subprocess safety, network egress policy under URL validation, per-tenant scoping under authorization.
- Make controls fail closed, and be explicit where they cannot. If your authorization service is unavailable, requests are refused — and someone has decided that in advance rather than discovering it during an outage.
- Add detection to the layers you expect never to fire. An egress denial, an authorization denial on an internal service, a canary credential appearing in logs — these are high-signal precisely because they should be silent.
- Do not add a layer without naming the failure it covers and the cost it adds. Layers without owners become alerts nobody reads (Alert Fatigue: The Page Nobody Reads in Observability & Performance).
What can go wrong
- Correlated layers: an edge check and an application check that both read the same misparsed forwarding header.
- A control that fails open on timeout, so the layer disappears exactly when the system is under stress (Circuit Breakers misapplied to a security decision).
- Detection with no response path — the alert fires into a channel nobody owns, which is indistinguishable from having no detection.
- Compensating controls used to justify leaving the real defect: "the WAF blocks that pattern" as a reason not to parameterize the query.
- Layers added and never removed, so the system accumulates controls that nobody can explain, and the cost of change rises until the team stops changing things.
- Blast-radius containment defeated by convenience: one shared database user, one shared role, one shared secret across environments.
- The realistic goal is not preventing every breach; it is ensuring that a single mistake in a single component is survivable. That is what "how bad was it" turns on in an actual incident.
- Containment layers determine the difference between "one customer's record was exposed" and "the tenancy boundary did not exist". They are less satisfying to build than preventive ones and they decide the severity.
- Independence is worth testing directly: disable a control in a staging environment and see whether anything else notices. If nothing does, the depth is nominal (Security Testing Portfolio in Security Engineering).
- Every control on The Backend Security Checklist is one layer. This lesson is the instruction to assume each of them is already broken.
- "Defence in depth means more tools." It means independent layers with different failure modes. Three tools reading the same signal are one layer with three invoices.
- "We have a WAF, so injection is handled." A WAF is a pattern matcher at the edge with no knowledge of your data model. It is a detection and mitigation layer, never the fix.
- "Belt and braces is always worth it." Layers cost attention, and attention is finite. A layer nobody understands is a layer that will be bypassed in an incident.
- "Our internal network is a layer." A flat internal network is one boundary, not depth. Segmentation is what makes it a layer (Network Segmentation in Security Engineering).
- "The outer layer never fires, so the inner one is unnecessary." It fires rarely by design. Its value is the day the outer one is misconfigured.
Operating it
- Instrument each layer's denials separately. If the inner layer never denies anything, either the outer one is doing all the work or the inner one is unreachable — and you cannot tell which without the metric.
- Track fail-open events explicitly: every time a control was skipped because a dependency timed out, that is a security event with a count.
- Run a periodic review that removes controls as well as adding them. An unused layer with a maintenance cost is a liability.
- Use incident reviews to ask which layer stopped the damage, and whether it was the one you expected. The answer is frequently surprising and is the best available evidence about your real architecture.
- At small scale, procedural controls genuinely work — everyone is in every review. The failure comes later and suddenly, when the team crosses the size where that stops being true.
- At 10x services, layers must be inherited from a shared platform or each new service starts at zero. Independence also gets easier here: network policy and identity are genuinely separate from application code and owned by different people.
- At larger scale containment becomes the dominant concern, because prevention across hundreds of services is statistically hopeless: assume one of them is compromised and design the blast radius accordingly.
- Every layer costs latency, operational surface and cognitive load. Depth is not free and is not linear — the fifth layer usually costs more attention than it returns.
- Fail-closed is correct for security and produces outages when a control's own dependency is down. That is a deliberate availability trade, and it belongs in the SLO conversation rather than being discovered (Availability, SLOs and Error Budgets in Software Architecture).
- Structural controls constrain legitimate work. The repository that forbids raw SQL will block a valid query eventually; plan the reviewed escape hatch instead of pretending it will not be needed.
- Strong containment — separate credentials, separate schemas, separate egress rules per service — multiplies configuration. That configuration is itself a source of outages.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe independence principle is universal. Which specific layers are available is not — a single-tenant on-premise deployment and a multi-tenant SaaS have different containment options for the same application code.
- SCALE-SPECIFICBelow one team, review plus a small number of structural controls is proportionate; adding a full detection stack there consumes the attention that would have gone into the code. Above roughly ten services, containment and detection stop being optional because prevention across all of them cannot be verified by anyone.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.