Service Templates
The production-readiness checklist expressed as a template, so a new service starts with health checks, signals, shutdown, config validation, alerts, a runbook and an owner already in place.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What should a brand-new service already have on the day it is created, before anyone asks for it?
Everything a service needs to be operable is easy to add on day one and expensive to retrofit on day four hundred, and the incentive on day one is entirely towards shipping the feature.
Start from an empty project or copy the last service that worked. Add health checks, metrics and a runbook once the service is real.
Copying propagates whatever was wrong in the source. A health check that returns 200 unconditionally has been copied into more production services than any other single mistake (Probes: Readiness, Liveness and Startup).
- Copying propagates whatever was wrong in the source. A health check that returns 200 unconditionally has been copied into more production services than any other single mistake (Probes: Readiness, Liveness and Startup).
- "Once it is real" arrives as an incident. The first time anyone wants a dashboard is when the service is already misbehaving.
- Each service ends up with a different shape, so an operator responding to a page has to relearn where things are for every service.
- Retrofitting graceful shutdown, structured logging or config validation into a running service means changing behaviour under traffic, which is a much larger change than including it at creation (Graceful Shutdown).
- The parts that get skipped are the parts nobody notices missing until they are needed: alert routing, ownership record, rollback path.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A template is a production-readiness checklist made executable. Instead of a document asking whether the service has health endpoints, the service has them because it could not have been created without them (Production Readiness Review).
- The template encodes an opinion at a moment in time. That is its strength — one reviewed opinion instead of many unreviewed ones — and the source of its main problem: services diverge from it the instant they are created.
- Which makes the update mechanism the real design question. A template that can only be applied at creation is a rubber stamp for day one; a template whose changes can be proposed to every service it created is a fleet-wide fix mechanism.
- A template concentrates risk in exactly the same way it concentrates leverage: a defect in it is present in every service made from it, and typically discovered only when one of them fails.
- Templates are per shape, not per organisation. A request-serving service, an asynchronous worker and a scheduled job need genuinely different defaults, and forcing one template across all three produces meaningless health checks and misleading alerts.
The checklist, as a template
Read this as the production-readiness review moved to the left. Each row is something a review would otherwise ask for weeks later, when providing it means changing a service under traffic.
The rightmost column is what makes each row honest: a template can create a file, and only the evidence tells you whether the thing in the file works.
| What the template ships | Why on day one | Evidence it is real, not present |
|---|---|---|
| Health endpoints, separated | Startup, readiness and liveness answer different questions and retrofitting the distinction changes rollout behaviour (Probes: Readiness, Liveness and Startup) | Readiness fails when a required dependency is down; liveness does not |
| Metrics for the golden signals | Rate, errors, duration and saturation are the basis of every alert you will later want (The Four Golden Signals) | A dashboard shows non-zero data before the first real traffic |
| Structured logs with request context | Adding correlation later means changing every log line | One request can be followed across services by identifier |
| Tracing wired | Instrumenting a running service is a much bigger change than starting with it (Instrumentation: From Code to Signal) | A trace shows the service's spans and its downstream calls |
| Graceful shutdown | Deploys drop requests without it, on every rollout, quietly (Graceful Shutdown) | A rolling deploy under load produces no client-visible errors |
| Config validated at startup | A missing value should stop the process now, not surface at 3am on a rare path (Validate at Startup, Fail Clearly) | Removing a required value makes it fail immediately, with a clear message |
| Secrets via workload identity | Retrofitting identity means changing how the service authenticates everywhere (Workload Identity) | No static credential in the image or repository |
| Pipeline and digest-addressed artifact | Build-once-promote-many is hard to adopt later (Build Once, Deploy Many) | The same digest is what ran in staging and in production |
| Rollout with health gating and rollback | The first rollback should not be the first time anyone tries | A rollback was performed at creation and took minutes |
| One symptom-based alert, routed | A service with no alert is a service you learn about from users (Alert on Symptoms, Not on Causes) | A test page reached a human on the owning team |
| Runbook stub with real content | Written when the author remembers, not during the incident (Runbooks) | Someone other than the author used it successfully |
| Ownership record | Determines who is paged and who decides (The Ownership Record) | An operator with no context finds the owner in under a minute |
Snapshot or living reference?
A template is applied once and then the service walks away from it. Everything about whether the template keeps paying off comes down to what happens to the services already created when the template improves.
Neither shape is wrong for all content. Application code should be generated once and owned by the team; pipeline definitions and base images are better referenced, so a fix reaches everything at once.
Create service from template
-> files copied, no version recorded
-> template improves five times
-> existing services unaffected and unknown
-> defect found in the original health check
-> "which services have this?"
-> read every repository by handCreate service from template
-> files copied, template version recorded
-> shared parts referenced, not copied
(pipeline definition, base image)
-> template improves
-> shared parts update on next build
-> copied parts arrive as a pull request
per service, owner reviews and merges
-> "which services have this?" is a queryThe recorded version turns a fleet-wide question into a lookup instead of an investigation. Splitting shared-by-reference from copied-and-owned is what lets urgent fixes propagate immediately while leaving application code under the owning team's control — a silent push into a service you do not own is a change with no reviewer and no owner awake.
When the template is the defect
The failure mode worth rehearsing, because it is the price of the leverage: the template is wrong, and it is wrong everywhere at once. The timeline below is an illustration rather than a report of a specific incident, but the shape is the common one — a template defect that is harmless until a dependency has a bad day.
- T-6 monthschangeTemplate ships with a liveness endpoint that verifies the database connection. It looks thorough and passes review.
- T-0signalThe database has a brief availability event lasting under a minute.
- T+20ssignalLiveness fails across every service created from the template. The platform does what liveness means and restarts them.
- T+40ssignalRestarted instances reconnect simultaneously, and the connection surge extends the database event (The Connection Budget).
- T+2msignalDatabase recovers; services are now in a restart loop with cold caches and empty connection pools.
- T+6mactionOperators identify the common factor: every affected service reports the same template version.
- T+9mactionLiveness relaxed via configuration across the fleet, without a redeploy; restarts stop.
- T+15mrecoveryTraffic normal. The database event itself lasted under a minute; the amplification lasted fifteen.
- T+1dchangeTemplate fixed so liveness tests the process and readiness tests the dependency. Change proposed as a pull request to all consuming repositories, tracked by template version.
The lesson is not "avoid templates". A hand-written probe with the same mistake in twelve services is the same outage with no mechanism to find the twelve or fix them together. The template made the defect uniform, which made it both wider and — because the version was recorded — findable.
How to do it properly
Most important first.
- Ship the operability essentials in the template: health endpoints that mean something, metrics and traces wired, structured logging with request context, graceful shutdown on the platform's termination signal, config validation at startup (Validate at Startup, Fail Clearly).
- Ship the operational context too: a dashboard, at least one symptom-based alert routed to the owning team, a runbook stub with real content, and the ownership record.
- Ship the delivery path: a pipeline, an artifact addressed by digest, a rollout with health gating and an exercised rollback.
- Record the template version in the generated service, so you can ask which services carry a given defect.
- Propagate template changes as pull requests against consuming repositories — visible, reviewable, and never a silent push into someone else's service.
- Keep the templates few and shaped by workload type, and test them by actually creating a service from each on a schedule.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A template defect is latent in every service created from it. Contained by recording the template version per service, so a fix can be targeted, and by delivering that fix as a per-service change the owner reviews.
What can go wrong
- A defect in the template reproduced across every service made from it — the leverage running backwards.
- A health check included but meaningless, which is worse than none: it makes a broken service look ready and passes rollout gating (A Successful Deploy Is Not Evidence of a Healthy System).
- An empty runbook stub, which looks like the box is ticked and helps nobody at 3am (Runbook Anti-Patterns).
- Templates that are copied rather than referenced, so improvements never reach existing services.
- A template so opinionated that teams strip half of it out, and nobody tracks that they did.
- One template stretched over incompatible workload shapes, producing readiness semantics that mean nothing for two of the three.
- "The template made it production ready." The template gives it the *mechanisms* for readiness. Whether the alert thresholds mean anything, whether the runbook has real content and whether the dependencies are understood is still a review (Production Readiness Review).
- "We can copy the last service instead." That is a template with no version, no owner and no update path — it propagates defects with none of the mechanisms that would let you fix them.
- "More in the template is better." Everything in the template is maintained by whoever owns the service. Generated code they do not understand is a liability with good intentions.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A service created from the template appears on a dashboard and can page its owner before it carries any traffic.
- Its health endpoints fail when a genuine dependency it cannot work without is unavailable, and do not fail when an optional one is.
- A deliberate rollback has been performed once, at creation, and took less than a few minutes.
- You can list which template version every running service came from.
- A bad template version is un-defaulted for new services immediately; that part is cheap.
- Services already created from it need a propagated fix rather than a rollback, which is why template-version tracking is what makes the fix possible at all.
- Template changes delivered as pull requests are individually revertible by the owning team, which is the correct blast-radius shape for a fleet-wide change.
- Automate creation, wiring, and the proposal of template updates to existing services.
- Automate a scheduled end-to-end test that creates a service from each template and takes it to a running state, so template rot is found by a robot rather than by a team on a deadline.
- Do not automate the merge of a template update into a service the platform team does not own. Propose; let the owner decide when.
- Do not template judgement: which dependencies are critical, what constitutes user impact, and what the alert threshold should be belong to the team (Golden Paths).
- Templates encode opinions, and every opinion is wrong for someone.
- Maintaining several templates well is more work than maintaining one badly, and much more useful.
- Generated code is code somebody has to own. A template that generates a large amount of scaffolding hands each team a maintenance burden they did not write.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe checklist below is what a service needs to be operable on any runtime. What differs is who provides each item: on a managed platform, shutdown handling and health probing may be the provider's, in which case the template's job is to conform to their contract rather than to implement it.
- PLATFORM-SPECIFICShutdown and readiness semantics are set by the runtime. A container platform sends a termination signal and waits a configured grace period; a serverless platform may freeze the process between invocations, where a graceful-shutdown handler has no meaning and the equivalent concern is what happens to in-flight work at freeze. Templating the wrong one produces code that never runs.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.