The Internal Developer Platform
Turning "create a service" into one standardised workflow that produces a repository, a pipeline, a deployment, observability, secrets, infrastructure and documentation.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does "create a service" have to actually produce before a team can call it self-service?
Standing up a new production service touches eight or nine systems that know nothing about each other, and getting any one of them wrong produces a service that runs and is not operable.
Give the team access to all the systems and a checklist. They are engineers; they can create a repository, add a pipeline, request a role and wire up dashboards.
It takes days, and most of those days are spent discovering which system to ask, not doing the work.
- It takes days, and most of those days are spent discovering which system to ask, not doing the work.
- Steps at the end of the checklist are the ones that get skipped, and the end of the checklist is where alerts, runbooks and ownership records live. The service reaches production and is invisible (The Ownership Record).
- Each service ends up wired slightly differently, so an operator responding to an incident cannot rely on any of it being where it was last time.
- The checklist encodes today's correct answer. Nothing updates the services created from last quarter's version of it.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The platform's job is to accept one intent — "a service called X, owned by team Y, of shape Z" — and fan it out into every system that needs to know, in an order that respects the dependencies between them.
- What makes it a platform rather than a script is that the result is *recorded*, not just created. The platform knows that service X exists, who owns it, which template version produced it and what it was granted. That record is what makes later fleet-wide change possible at all.
- Provisioning is the easy half. The hard half is day two: the template improves, a default turns out to be wrong, a service is decommissioned and its role, secret, bucket, DNS record and alert route must go with it.
- Every step is an integration with a system that can be down, rate-limited or slow, so the workflow needs the same properties as any distributed operation: it must be resumable, and partial completion must be visible rather than silent.
One intent, many systems
The value of the platform is that the team states an intent once and the fan-out happens in an order that respects what depends on what. The repository must exist before the pipeline references it; the identity must exist before the pipeline can push; the infrastructure must exist before the deployment targets it.
Every arrow below is an integration that can fail on its own. The workflow is therefore not a script that runs to completion but a desired state that is converged towards, which is why re-running it has to be safe.
The nine outputs, and how each one fails
The test of a creation workflow is not whether it produces a running process. It is whether it produces an *operable* one — and the outputs most often missing are the ones that only matter later, at three in the morning.
- 1Repository
Code from a versioned template, with the template version recorded.
fails by Copied from another service, so its defects and its owner's assumptions come too.
evidence The repository records which template version created it.
- 2Pipeline
Build, test, produce a signed artifact addressed by digest.
fails by Hand-written per service, so no two pipelines fail the same way (Tags Versus Digests).
evidence A green run produces an artifact in the registry, referenced by digest.
- 3Deployment
A rollout with health gating and a rollback path.
fails by Deploy configured, rollback never exercised (Rollback: Only Useful If It Is Actually Safe).
evidence A deliberate rollback has been performed once, at creation.
- 4Infrastructure
Provisioned from reviewed modules with defaults that match the shape (Modules: Reuse Without Hiding).
fails by Console clicks that no plan will ever reconcile (Drift).
evidence A plan against the live account reports no changes.
- 5Identity and secrets
A workload identity scoped to this service, with secrets issued to it rather than embedded (Workload Identity).
fails by A shared long-lived key copied into CI variables.
evidence The service starts with no static credential in its image or repository.
- 6Observability
Metrics, logs and traces emitted and routed, with a dashboard that exists on day one.
fails by Instrumentation deferred until the first incident needs it.
evidence The service appears on a dashboard before it carries traffic (Dashboards an Operator Can Act On).
- 7Alerts and on-call
At least one symptom-based alert routed to the owning team.
fails by Alerts route to a default channel nobody watches (Alert on Symptoms, Not on Causes).
evidence A test page reaches a human on the owning team.
- 8Ownership record
Team, escalation path and dependencies, in a queryable place.
fails by Ownership lives in someone's memory until they change team.
evidence An operator with no context can find the owner in under a minute.
- 9Documentation and runbook
A stub with what the service does, its dependencies and its first three operations.
fails by Empty stub, which is worse than none because it looks answered (Runbook Anti-Patterns).
evidence Someone other than the author used it to do something.
The last four are the ones that get skipped when a human follows a checklist, and they are precisely the ones that decide whether the service is operable by anyone but its author.
Day two is the half that compounds
Creation is a one-off benefit per service. Everything that makes a platform worth its overhead happens afterwards: changing many services at once, knowing what exists, and removing things cleanly.
A platform that only creates is a scaffolding tool. Useful, and it does not earn a team.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A vulnerability is announced in the base image | Nobody can say which services are affected | No record of what each service was built from | Record template and base image version per service; make it queryable (Software Bill of Materials) |
| The default health check turns out to be wrong | Fix is applied to new services only | Templates are copied at creation and never revisited | Propagate template changes as pull requests to existing repositories |
| A team is disbanded | Its services keep running and paging a dead rotation | Ownership recorded once and never reconciled | Periodic ownership reconciliation with an owner-or-decommission outcome |
| A service is decommissioned | Role, secret, bucket and DNS record remain for years | Creation is automated, deletion is a manual afterthought | Decommission is a first-class workflow that removes exactly what creation made |
| Cloud spend rises with no obvious cause | Cost cannot be attributed to a team | Resources created without required ownership tags | Enforce tagging at provisioning time as policy, not convention (Policy as Code) |
| The platform workflow fails halfway | A service that exists in three systems and not in two | The workflow is a script rather than a convergent desired state | Make it idempotent and resumable; report partial state instead of hiding it |
How to do it properly
Most important first.
- Make one intent produce every artifact, including the unglamorous ones: ownership record, on-call route, runbook stub, dashboard, rollback path (Service Templates).
- Record the template version each service was created from, so "which services still have the old health check" is a query rather than an investigation.
- Make the workflow idempotent and resumable. Re-running it on an existing service should converge, not fail halfway or duplicate (Reconciliation: The Loop Under Everything).
- Provision the smallest useful default — one environment, minimum privilege, modest resources — and make growing it another self-service action (Least Privilege in Production).
- Own decommissioning as first-class. If the platform can only create, it manufactures orphans.
- Expose the same workflow through more than one surface if teams want it — a CLI, a pull request, a portal — but keep one implementation behind them.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The creation path itself is contained — a bad template affects services created after it. The platform's credentials are not: they can typically change any account it provisions into, which is contained only by scoping its permissions per target environment.
What can go wrong
- Partial provisioning: the repository and pipeline exist, the role does not, and the failure surfaces at the first deploy rather than at creation.
- The platform holds credentials broad enough to create anything in any account, making it the highest-value target in the organisation (Identity and Access Management (IAM)).
- A creation workflow with no deletion workflow, producing a slow accumulation of resources with no owner and a cost line nobody can attribute (Cost Drivers).
- The portal becomes the only interface, so anything it does not have a form for requires a ticket — a gate wearing a platform's clothes (Guardrails, Not Gates).
- Template drift: services created six months apart differ in ways nobody can enumerate.
- "An internal developer platform is a portal." The portal is a surface. The platform is the workflow, the records and the ability to change the fleet afterwards; a portal on top of manual steps is a form, not a platform.
- "If it provisions everything, teams do not need to understand what was provisioned." They will be reading it during an incident. Provisioning invisibly is fine; provisioning *unreadably* is not.
- "We can add decommissioning later." Later means after the orphans exist, when nobody is sure which of them anything depends on.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A team creates a service and reaches production the same day, without a platform engineer.
- For any running service you can answer, from a record rather than a person: who owns it, what template made it, what it was granted, and where its runbook is.
- Decommissioning a service leaves nothing behind — no role, no secret, no DNS record, no alert route pointing at a dead thing.
- Re-running creation on an existing service is safe and does nothing.
- A failed creation must be reversible as a unit. Track what was created so far and be able to remove exactly that, rather than leaving half a service and a human to work out which half.
- For the platform itself, keep the previous template version usable. A new template that produces broken services must not strand the teams currently mid-creation.
- Automate the entire fan-out: repository, pipeline, registry entry, infrastructure, secret issuance, observability wiring, ownership record, documentation stub.
- Automate the propagation of template changes as proposed pull requests against consuming repositories — visible, reviewable, per-team, and never a silent push.
- Keep the decision about what shape a service should be with the team. The platform offers shapes; it does not choose architecture.
- One workflow across many systems means the platform is coupled to all of them, and any of their API changes are now the platform team's work.
- Standardising creation makes unusual services harder, and some of them are unusual for good reasons.
- A recorded fleet is a genuinely powerful thing and also an accurate map of everything worth attacking.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICWhich systems the workflow must touch is entirely local — an organisation with one cloud account, one repository host and no separate secret store has a three-step workflow, and one with per-team accounts, an internal registry, an approval system and a CMDB has a fifteen-step one with more failure points.
- SCALE-SPECIFICWorth building at the point where services are created often enough that the workflow is exercised — roughly monthly. Below that the workflow rots between uses and fails when it is finally needed, and a written checklist that a human follows is more honest.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.