Learn Backend Engineering

A long-lived process holding the things a client cannot be trusted with, reached over a protocol it does not control.

Every hop between a client and a response, and the fact that each one can fail independently.

The list of duties that cannot be delegated to a client, a framework or a managed service.

Eleven questions to ask of any feature, in an order that surfaces the expensive decisions early.

Everything crossing into your process is untrusted — including responses from services you own.

Which question belongs to which domain, so you look for answers in the right place.

Keeping request-serving instances free of durable state, and being precise about which state that means.

The seven things every HTTP server does between a listening socket and the last byte written, whatever framework sits on top.

A connection is not a request. Listen, backlog, accept and file descriptors decide what happens to traffic before your code exists.

Turning a byte stream with no message boundaries into a request — and why the parser is a security component, not a formality.

What `req` and `res` really are: a mutable view over a socket, with a body that has not been read and a response that has a point of no return.

A status code is an operational signal: it decides who gets paged, what retries, and whether the number is counted against you.

Reusing a connection removes a handshake from every request — and introduces a timeout you must coordinate with every hop.

A body is bytes arriving over time at a rate the client controls, which makes buffering a memory decision and limits a survival requirement.

Four ways a server serves many requests at once, and what each one makes cheap, expensive and dangerous.

One thread runs your JavaScript, a small pool runs some of the I/O, and knowing which is which explains most Node production behaviour.

One function that does not yield holds the only thread that makes progress, so every concurrent request pays for it.

Sync workers, threads, async and processes are four different services with the same source code — and the GIL explains which is which.

Running N copies of your service in one machine buys cores and isolation, and multiplies every per-process resource by N.

When a backend is infrastructure, no garbage collector and direct control of memory buy predictable tail latency — at a price paid in engineering time and safety.

A decision made once, changed rarely and paid for daily — decided by workload shape, ecosystem and who is on call, not by benchmarks.

A route table is a lookup structure over method and path; everything else about routing follows from which structure your framework chose.

A path parameter is an attacker-supplied string that happens to be positioned where you expected an identifier.

The least standardised part of an HTTP request, parsed differently by every stack, and the usual entry point for unbounded work.

When two routes can match one path, something decides which wins — and in half of all frameworks that something is the order of lines in a file.

API Design decides the versioning policy; this is what the policy costs inside a running process, and how to pay it without forking the codebase.

A handler is an adapter between HTTP and one application operation — parse, resolve caller, call, map, return — and everything else it does belongs somewhere it can be reused and tested.

Middleware is function composition around a handler, with a phase on the way in and a phase on the way out — not a list of things that run first.

The chain is a dependency graph flattened into a list; reordering it does not tidy the same behaviour, it produces different behaviour.

Authenticating first means an unauthenticated flood still costs you crypto and a user lookup; rate-limiting first means your key is an IP, which is coarse and evadable. Both are true, so you do both, with different keys.

One place that turns any failure below it into a response the client can act on and a log line you can investigate — plus the specific failures it will not catch.

Getting the correlation id, principal, tenant and deadline from the outermost middleware to a log line five layers down — without an extra argument on every function, and without a global that lies.

A concern belongs in middleware if it is uniform, transport-level, needed even when no handler runs, and cheap — which rules out several of the things most often put there.

A use case expressed as a plain function that knows nothing about HTTP — which is what lets a job, a CLI and an endpoint share it.

A named place for the queries your domain asks, so the definition of "active subscription" lives once instead of in eleven call sites.

A repository whose methods are one-line passthroughs to the ORM adds a file, a name and a hop, and removes no decision from anyone.

Four layers, one rule that matters — dependencies point inward — and a scale at which the whole thing is overhead.

Vertical slices, transaction scripts and hexagonal are not lesser versions of layering — they optimise for different changes, and one of them probably matches yours.

The 300-line handler is a real production problem for a specific reason: it puts a payment call inside a database transaction and nobody can see it.

Passing a dependency in instead of importing it is the whole idea; a container is one way to do the wiring and is not the idea.

Well-formed, permitted, and consistent are three different questions with three different enforcement points — and only the last one survives a race.

The only check that needs no state: is this payload the right shape, the right types, and small enough to look at?

Rules that need loaded state and an actor — and the fact that every answer they give is already stale.

The only check evaluated inside the write — which is why it is the only one that survives two requests arriving at the same instant.

A check that returns a boolean throws away what it learned; a check that returns a typed value hands the knowledge to every line below it.

Body, query, path, headers, cookies, files, webhooks, external responses — and the second-order case where your own database hands back something a request wrote.

Three layers reject for three reasons, so one 400 with a sentence in it is the wrong answer to all three.

A response is bytes on a socket. The encoder decides which of your runtime's types survive the trip and which quietly change shape.

Parsing untrusted bytes produces data of a known shape, not data you may trust — and the gap between those two is where mass assignment lives.

CPU proportional to the nodes you visit and garbage proportional to the bytes you produce — paid on every request, amortised by nothing.

The database row, the domain object and the API response answer to different owners and change for different reasons. Collapsing them is a decision, not a default.

Returning the row is publishing the schema. It is a contract you did not write, cannot see, and will be held to.

Turning an untrusted credential into an authenticated principal, once, early, in one place — and nothing more than that.

Store passwords with a slow, salted, purpose-built hash from a maintained library. Everything else in this lesson is a consequence of that sentence.

The server keeps the state and the client carries an opaque handle. Revocation is a delete; the cost is a lookup on every request.

Process memory, a database table, a shared cache or a distributed store — four answers with different scale ceilings and different things that happen when they fail.

A self-contained token removes the lookup by carrying its own claims — and removing the lookup is exactly what makes immediate revocation hard. That is the trade, and it is the whole lesson.

Three roles, two very different tokens, and one rule: use a maintained library, because the parts you would get wrong are the security parts.

A long-lived secret that identifies an application rather than a person — cheap to verify, easy to leak, and revocable only if you designed for it.

Every request carries a claim about what the caller may do. The backend is the only place that claim can be tested.

"Who are you" and "may you do this" are different questions with different answers, different failure modes and different blast radii.

A hidden button is not a control. Middleware, handler, service and query each enforce something different — and only one of them is a guarantee.

Roles group permissions so people can be granted a job, not a list. What roles cannot express is anything about the object.

Rules over attributes of the principal, the resource and the context — more expressive than roles, and correspondingly harder to reason about.

A user can be perfectly authenticated, hold exactly the right role, and still have no business touching project 123.

One deployment serving many customers, where the worst possible bug is showing one of them another one's data.

The tenant comes from the authenticated principal. Any other source — header, subdomain, path, body — is an authorization bypass with extra steps.

ORM, query builder, raw SQL and stored procedures as four points on a spectrum, chosen per query rather than per project.

Object method to generated SQL to database, plus the identity map, unit of work and lazy proxies that decide when statements are issued.

An honest ledger: real productivity on entity-shaped work, real opacity on query count, plans and complex reads.

One query for the list, one more for every row: 100 users become 101 statements, and the source code shows none of it.

The two general fixes for per-row queries — load the relation up front, or collect the ids and fetch once — and what each one over-fetches.

Composing SQL structurally in the host language: dynamic filters without string concatenation, and no mapping layer to explain.

When to write the statement yourself, how to keep it parameterized and findable, and what you take on when you do.

Schema change as a deployment problem: two code versions run at once, and some `ALTER TABLE` statements take a lock that stops the service.

The pool is what actually bounds your concurrency: 1,000 requests against 20 connections means 20 running and 980 waiting, silently, until they time out.

BEGIN, COMMIT and ROLLBACK as things your code controls: bound to one connection, ended by an error you did not expect, and retried when the database says so.

Which operations must commit together, which merely happen nearby, and why "the whole handler" is almost never the right answer.

Splitting a unit of work trades an atomicity guarantee for shorter locks — and buys you an intermediate state you now have to design.

A five-second payment call between BEGIN and COMMIT holds a pooled connection and every lock the transaction took, for five seconds, on every request.

The commit succeeds and the publish fails, or the publish succeeds and the commit rolls back. Two systems, no shared transaction, and no ordering that fixes it.

Write the event as a row in the same transaction as the state change, then publish it from a background reader — at-least-once, by design.

Two transactions take the same two locks in opposite orders, each waits for the other, and the database kills one of them — with an error your code has to expect.

A cache trades correctness-in-time for work avoided; everything else in this module is about controlling that trade.

Read the cache, miss, read the database, write the cache — and the four things that go wrong in those four steps.

The database changed. How does the cache find out? Four answers, each with a different failure when it does not.

A TTL is a staleness budget written as a number, plus the only invalidation mechanism that cannot fail.

One popular key expires, every in-flight request misses at the same instant, and all of them run the same expensive query at once.

In-process is faster and per-instance; shared is consistent and one more thing that can be down. The choice is about invalidation, not speed.

A cache buys you a consistency problem and an availability dependency. Sometimes it does not buy anything back.

Work that outlives the request that asked for it — and the four guarantees you give up to move it there.

Five questions that decide where work runs — and the reminder that deferring is a cost, not a default.

Enqueue, claim, process, ack, retry, dead-letter — the six-step lifecycle every queue implements, however it spells them.

Ordering, duplicates, retries, visibility timeouts and poison messages — the five properties that differ between every broker you will use.

Delivery will repeat, so the effect must not. How to make a worker safe to run twice, including concurrently.

Cron in a single process is a timer. Cron on three instances is three timers, and the job runs three times.

Somewhere for work that will never succeed, so that one poison message cannot consume the fleet — and a human who is expected to look.

When producers outrun consumers, something has to give. Backpressure is choosing what, instead of letting memory choose for you.

More workers help until the shared dependency saturates, at which point they make everything worse — including the requests still in the path.

The queue is growing. Four possible causes, and the recovery that is right for one of them makes two of the others worse.

A command asks for something to happen and has exactly one handler; an event states that something happened and has any number of consumers.

Publishing a fact instead of calling the next step, what that actually buys, and the honest list of what it costs.

Past-tense domain facts decouple; procedural names smuggle the consumer's behaviour into the producer.

A consumer is a program that will see every message twice, some out of order, and one that poisons it.

Database change to event to indexer to search engine — and the fact that when it breaks, nothing errors and search is quietly wrong.

Once work happens after the response, some reads are stale — and the engineering is in bounding it, showing it honestly, and knowing when it has stopped converging.

Eight questions every outbound call has to answer, and the fact that a dependency's availability becomes yours the moment you await it.

Never assume an external dependency returns. A call with no timeout is a resource leak waiting for a bad day.

"Retryable" and "safe to retry" are different properties, and confusing them is how a transient error becomes a duplicate charge.

Waiting longer between attempts stops you hammering a struggling dependency; randomising the wait stops every client from hammering it in unison.

After a dependency has failed enough, stop calling it: fail fast, protect your own capacity, and probe carefully for recovery.

Give each dependency its own bounded slice of your resources, so one slow dependency cannot consume every worker you have.

Deciding which caller has had enough, along which dimension — and why the counter's atomicity is the part that makes it correct.

Fixed window, sliding window, token bucket and leaky bucket — what each one allows, what it refuses, and the burst each permits.

The integration everyone builds first and treats casually: an external provider, at-least-once delivery, and a side effect that cannot be taken back.

A third party calls your API on its schedule, with its retry policy, and treats your endpoint as infrastructure it depends on.

Proving the request came from the provider — computed over the raw bytes, compared in constant time, bounded by a timestamp.

Providers retry, so duplicate delivery is the normal case — deduplicate on the provider event id, atomically.

The provider decides when to retry and does not promise order, so your handler must be correct for events that arrive late, twice, or backwards.

When you are the provider: delivering to endpoints you do not control, without letting a slow customer take down your service.

Doing the same thing twice must produce the same result as doing it once — because in a network, twice is not optional.

A client-generated identifier that is stable across retries of one intent and unique across different intents.

Client sends a key, the server claims it atomically, and the outcome is either process-and-store or return-the-stored-result.

Where keys live, how long they last, what scopes them, what is stored against them — and the atomic insert that makes concurrent use safe.

Queues can guarantee a message is delivered at least once; only your consumer can guarantee the business effect happens once.

Recognising that this work has already been done — atomically, at the right scope, within a bounded window.

Two requests, one row: where concurrency bugs actually live in a backend, and why they never appear in development.

Read a version, write only if it has not changed, and treat zero rows updated as a conflict — never as success.

SELECT ... FOR UPDATE serialises access to a row — and holds a lock, a connection and a transaction while you do it.

Do it in one statement the database evaluates indivisibly, instead of reading, deciding and writing from application code.

Work that starts without a limit does not fail gracefully — it consumes memory, connections and downstream capacity until something breaks.

When N concurrent requests need the same expensive result, do the work once and share it — the in-process answer to a stampede.

Every finite resource needs an explicit limit, or the system discovers its own — at the worst possible moment, in the worst possible way.

Eight kinds of failure, each with a different status, a different caller action and a different owner — instead of one 500 for everything.

A driver error becomes an application error becomes an API response — and each translation adds context while removing internals.

Stack traces, SQL fragments, internal hostnames and library versions in an error response are free reconnaissance for an attacker.

One identifier that follows a request through services, queues and workers — including the hop into a background job, which is where it is usually dropped.

Six questions every log line should help answer, and the one category of data that must never appear in one.

Log events as typed key-value records rather than sentences, because the consumer is a query engine, not a person reading a terminal.

Request rate, error rate, latency, in-flight requests, pool usage, queue depth, cache hit rate and dependency latency — eight numbers that make a service legible.

What a service must emit and propagate so one request's path across processes becomes a single readable timeline.

Three different questions with three different consequences — and a liveness check that fails on a dependency outage turns a bad hour into a much worse one.

The obvious path — client to backend to storage — makes your request handler a bandwidth-bound file mover, and it is still the right answer sometimes.

Your backend issues a signed, expiring permission slip; the client uploads directly to storage; your process never sees a byte — which is the benefit and the cost.

A bucket holds objects addressed by a key. That primitive — not any provider's SDK — is what you are actually programming against.

File size, privacy, scanning, processing and serving decide the architecture. There is no default that is right for all five.

Stored is not ready. Scanning, transcoding and thumbnailing are background work with a state machine, and skipping the state machine is how unscanned files get served.

Private files need a check on every read; public files need a CDN. Serving both through your API is the one option that is wrong for both.

The same artifact must run in dev, staging and production — so everything that differs between them is input, not source.

Credentials need a different storage, a different access path and a lifecycle — and rotation is the part every team skips.

Check everything the process needs at boot and refuse to start — rather than discovering the missing value at 3am on the first request that needs it.

Separating deploy from release, buying an instant off-switch — and accumulating a combinatorial mess if nobody removes them.

Business logic to unit tests, database behaviour to integration tests, contracts to contract tests, and only the critical flows to end-to-end.

A substitute engine with different SQL semantics gives you a green suite and a broken production — the failure the substitute exists to prevent.

Verify that a producer and its consumers still agree on the wire format, without running both systems at once.

Latency, throughput, concurrency, CPU, memory, database load and dependency load are seven different dimensions — a single "requests per second" number answers almost none of them.

VM, container, managed container, function, PaaS and Kubernetes compared by the engineering problem each one is solving, not by the marketing category.

An image is a frozen filesystem plus an entrypoint; the interesting part is what your process must do once it is PID 1 with no shell around it.

SIGTERM arrives, and the process has one job: stop taking new work, finish or cancel what it holds, release everything, and exit before it is killed.

Replacing instances a few at a time keeps the service up, and guarantees that two versions of your code run against one database at the same time.

Five steps that let a schema change survive a rolling deploy, because for the length of that deploy two versions of your code share one database.

Send a small slice of real traffic to the new version, compare it against the old on the same signals, and only then commit the fleet.

Run two complete environments, cut traffic from one to the other, and cut back if it is wrong — while remembering that the database was never duplicated.

A translation table for the eight things a backend needs from a cloud — with the column that matters most being what the analogy gets wrong.

What the application must do to be a well-behaved workload: honest probes, a termination sequence that races endpoint removal, resource requests that match reality, and no local disk.

A per-invocation execution model that removes supervision and adds cold starts, execution limits and connection pressure — excellent for some workloads and wrong for others.

The inventory, the migration order and the verification — turning a service that works on one instance into one where any instance can serve any request.

A bigger machine is simpler and has a ceiling; more machines have no ceiling and require statelessness, a load balancer and coordination you did not have before.

What your application owes the thing distributing traffic to it — an honest health signal, aligned timeouts, and no assumption about which instance gets what.

Pinning a client to one instance buys cache locality and hides instance-local state — and it costs you failover, rebalancing and clean scale-down.

Choosing a signal that actually reflects load — and understanding why CPU is the wrong one for a service that spends its time waiting.

Routing reads to a replica multiplies read capacity and introduces a window where the application can read data that is older than what it just wrote.

Offset pagination is easy and gets quadratically more expensive with depth; keyset pagination is cheap at any depth and gives up random access to page N.

The controls every service owes no matter what it does, and the layer each one has to live in.

Parameterized queries solve injection completely — and do nothing whatsoever for authorization.

The shell is a parser you did not intend to invoke. Pass an argument array, or do not spawn a process at all.

A fetch your server makes on a caller's behalf runs from inside your network with your identity. Blocklists lose; egress control holds.

Most of your running code was written by strangers. The controls are reproducible installs, a known time-to-patch, and a build that does not hand out credentials.

Logging a request object, an auth header or a webhook payload copies a credential into every system your logs reach.

Design as if each control has already failed, and prefer controls that work when someone forgets.

One deployable, in-process calls, real transactions across the whole domain — and costs that arrive at team boundaries, not at request volume.

One deployable with boundaries the build enforces: most of what services give you, without the network.

Independent deployment and independent failure, bought with every cost that arrives when a function call becomes a network call.

HTTP couples availability and returns an answer; messaging decouples availability and returns a promise. Neither is inherently more scalable.

A slow database becomes timeouts, becomes exhausted workers, becomes an outage in endpoints that never touched the database.

The feedback loops that turn a recoverable degradation into a system that cannot come back up.

Monolith, modular monolith, microservices, serverless and event-driven — when each is useful, what each costs, and how each fails.

Turning "the API got slow" into a named cause by narrowing the search space with evidence instead of guessing at fixes.

The decision flow from "slow" to a named bottleneck: split the time first, then follow the branch the evidence selects.

Thirteen failures that account for most backend incidents, each with the symptom that identifies it and the first diagnostic to run.

Ten patterns that predict production trouble — and, for each, the situation where the same pattern is the right answer.

The highest prior probability for a sudden change in behaviour belongs to the thing that just changed — usually yours.

Distinguishing a leak from ordinary heap growth, finding the reference that retains, and doing it on a live process.

How a reasonable retry policy turns a dependency's brief degradation into a sustained outage, and what bounds it.

The endpoint is slow, the database is calm, and the pool has waiters — the most commonly misdiagnosed backend incident.

The request path when a model sits in the middle: what is genuinely new, and what is the backend you already know.

The model choosing a tool is a client choosing an endpoint — and the server owes exactly what it always owed.

An agent acts on behalf of a user and must be limited to that user's permissions — not to the service account it happens to run under.

An agent loop terminates because you made it terminate — bounded in time, in steps, in tokens and in money.

What was asked, what the agent decided, which tools ran with which arguments, and what changed as a result.