On-Call for Product Engineers
If you shipped it, you own the page. That means alerts on what customers feel, a runbook that makes the page short, and treating every page as a signal about the product, not only about the system.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
Why should the engineers who build checkout features be paged when checkout breaks at night — and what makes that bearable?
A new engineer, looking at the rotation: "Why am I on call? Isn't that what the platform team is for? I build features."
The platform or SRE team runs production. Feature teams ship code; if something breaks, the platform on-call handles it and files a ticket for the feature team in the morning.
The platform engineer paged at 02:00 sees orders failing on a promo edge case. They cannot tell whether the promo is supposed to allow that combination, cannot safely change the pricing code, and can only restart things and wait for morning.
- The platform engineer paged at 02:00 sees orders failing on a promo edge case. They cannot tell whether the promo is supposed to allow that combination, cannot safely change the pricing code, and can only restart things and wait for morning.
- The feature team finds the ticket at 09:00. Seven hours of customers hit the bug. The fix is a flag flip that anyone on the team could have done in two minutes — if they had been paged.
- Because the feature team never feels the page, the same fragile code ships again. The pain lands on people who cannot fix it, and the people who can never learn it hurts.
- The alerts were written by the platform team for the platform: CPU, memory, error counts. Nothing fires for "promo applied and total is zero", because nobody who understands the product wrote the alert.
What is really going on
- On-call closes the loop between shipping and consequences. The team that feels the page for a change is the team that adds the guard, the test, the flag. Take the page away and the incentive to write operable code weakens (The Support Loop closes the same loop for confusion rather than failure).
- Only the feature team knows what correct looks like. Whether a zero total is a bug or a legitimate full-discount promo is a product question. The person who shipped the promo can answer it at 02:00; anyone else has to guess.
- A page is as short as the runbook allows. The difference between a ten-minute page and a two-hour one is almost never skill; it is whether the first three steps are written down, with the commands and the flag names, by someone who knew them in daylight.
- Good alerts fire on what customers feel — orders not being created, payment failures by provider, checkout latency at the tail — not on the causes you happened to think of. Cause alerts are noisy and miss the failure you did not predict.
- On-call is only sustainable if pages go down over time. Each page is either fixed at its cause or turned into a better alert, or the rotation fills with the same pages every week and people leave it (Postmortems That Change Things).
If you shipped it, you own the page
The platform engineer paged at 02:00 is competent and powerless: they can see orders failing but cannot tell whether a zero total is a bug, and they will not touch pricing code they do not own. The only move they have is to wait for the feature team, which means the customers wait too.
Putting the feature team on the page is not about punishment or fairness. It puts the one person who knows what correct looks like in front of the problem, and it puts the cost of operability where the decisions about operability are made.
Alert on what customers feel
The alerts that belong to a feature team are the ones only it can write: the symptoms of its product being wrong. "Orders created per five minutes is below 50% of the same time last week" catches the promo bug, the payment outage, the bad deploy and the failure nobody predicted, because all of them show up as customers not getting orders.
Cause-based alerts — a specific exception, a queue depth — are fine as diagnostics on a dashboard. As pages they are noisy, and they miss everything you did not think of (Instrumentation First).
PricingService: NullPointerException count > 5 in 1 minute
Checkout: orders created in last 10 min < 50% of same window last week, for 10 min. Customers are paying or trying to pay and not getting orders. Runbook: checkout/orders-low.
The first fires on one predicted cause and stays silent for every other reason orders fail. The second fires on the thing that actually matters, says what it means for customers, and links to what to do.
The runbook that makes the page short
A runbook is written in daylight by the person who shipped the feature, for someone half awake who did not. It says what the alert means for customers, how to confirm it is real in one step, the first safe action, and who to call next. It does not explain the architecture.
1# Checkout: orders created below baseline2 3**What customers feel:** they reach payment and do not get an order,4or cannot reach payment at all. Money may be captured without an order.5 6## 1. Confirm (2 min)7- Dashboard "Checkout funnel", panel "orders vs last week". Real if both8 payment attempts and orders dropped, or orders dropped alone.9- If payment attempts dropped too, check the provider status page first.10 11## 2. First safe action (2 min)12- Deployed in the last 2 hours? Roll it back: deploy tool, checkout, "previous".13- Promo launched today? Turn off its flag in the flag console.14- Both are safe to do without asking anyone.15 16## 3. If still low after 10 min17- Page platform on-call if database or payments-gateway errors are high.18- Post in #incidents: "Checkout orders low since HH:MM, rolled back X, still low."19 20## 4. After it recovers21- If money was captured without orders, open the affected-orders query22 (queries/payments-without-orders.sql) and hand the result to the incident lead.23 24Owner: checkout team. Last tested: 2026-09-10 (game day).Step 4 is the product part: the runbook does not end when the graph recovers, because the customers are not whole yet (Incidents Are Product Events). "Last tested" is what keeps the flag names true.
Asking for the page
The conversation that sets this up is usually with an engineering manager who worries about the team's time, or with engineers who worry about their nights. It goes better when it starts from a specific night.
Your manager is unsure about putting the feature team on the rotation. Platform currently takes all pages.
"I think we should be on call because we should own our stuff. It is best practice for product teams."
"Last month's promo bug failed orders from 02:00 to 09:00 and the fix was a flag flip. Platform could see it but could not tell if the promo was wrong. I would like us to take pages for three alerts — orders low, payment failures, checkout errors — each with a runbook we write first. Platform stays the escalation. We review every page weekly, and if we get more than a couple a week, that goes on the roadmap."
How to do it
Most important first.
- The team that owns the feature is on the rotation for it, with the platform team as escalation for infrastructure. Write the boundary down: which alerts page whom.
- Write alerts on customer symptoms for the flows you own: order creation rate against baseline, payment failure rate by provider, checkout errors per minute. Few, specific, each with a runbook link.
- Every alert has a runbook, written by the engineer who shipped the feature, before the feature ships: what this alert means for customers, how to confirm it, the first safe action (usually a flag), and who to call (Done Means Someone Used It).
- Review every page in the weekly meeting: was it real, was the runbook enough, what would stop it happening again. Delete alerts that were not actionable.
- Protect the rotation: compensation or time off after night pages, a week at most, and a rule that a page at night does not mean a full day of work after it.
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the customer and the time: "Last month, a promo bug failed orders from 02:00 to 09:00. The fix was a flag flip. We were not paged, so customers waited seven hours for two minutes of work."
- Then the principle, in one sentence: "The team that ships checkout is the only team that knows whether a checkout is correct. That is why we hold the page."
- Then what makes it fair: "Each alert fires only on something customers feel, each has a runbook written before launch, and we review every page weekly to make the next one rarer."
- Then the deal: "Night pages get time back. If our rotation gets noisy, that is a product problem we fix on the roadmap, not something you absorb."
- "I am a frontend engineer, I cannot fix database problems." You do not need to. You need to recognise that checkout is failing, flip the flag, and escalate to the right person. The runbook tells you how.
- "This will slow down feature work." It will, a little, for a while. The alternative is slower in a way nobody measures: nights of customer failures and mornings spent on fires that were two-minute fixes.
- "Why not pay the platform team to do it?" Because they cannot tell a broken promo from a working one, and they cannot change the code that caused it. We would be paying for someone to wait until we wake up.
What can go wrong
- Alert fatigue: too many alerts, most not actionable, so the real one at 02:00 is snoozed with the others.
- Runbooks written once and never updated. The flag in step two was renamed in the spring; the on-call engineer spends twenty minutes discovering that.
- Heroics as a culture: the same senior engineer fixes everything because they know the system, the runbooks never get written, and the rotation depends on one person's sleep.
- Putting feature engineers on call without access or training: they are paged, they cannot see the logs or flip the flag, and they wake someone else anyway.
- Treating on-call load as a personal endurance matter rather than a signal that the product has operational debt.
- "On-call is an operations job." Operating a feature is part of building it. The platform team runs the platform; the team that shipped checkout runs checkout.
- "More alerts means safer." More alerts means more noise and slower response to the one that matters. The goal is few alerts, each tied to a customer symptom and a runbook.
- "A good engineer does not need a runbook." A good engineer at 02:00, half awake, in code they did not write, needs exactly the runbook a good engineer wrote at 14:00.
Knowing whether it worked
- Pages per week for the team trend down over a quarter while the feature count goes up.
- The median time from page to mitigation is short, and the runbook was the thing that made it short — the on-call engineer says so in the review.
- Engineers add the alert and runbook in the same PR as the feature without being reminded.
- Nobody dreads their on-call week — or, if they do, the team knows exactly which alert is the reason and has a ticket for it.
- On a five-person team, everyone is on call for everything and the runbook is a shared page. The principle — whoever ships it can be paged for it — holds without any structure.
- At 10x team size, ownership has to be explicit: which team owns which alert, with a routing table and an escalation path. Unowned alerts page whoever was nearest and are ignored.
- At 10x traffic, the cost of a slow page rises with every minute. Runbooks move towards automatic first actions — the flag flips itself when the order rate drops — and humans handle what automation cannot.
- It costs sleep and attention. Even a quiet rotation is a week of carrying a phone and not being fully off; that has to be paid for, in money or time.
- Feature engineers need operational skills and access they did not need before. Training and permissions take time, and access has a security cost.
- Writing runbooks and symptom alerts before launch adds a day or two to features. It is the cheapest time to do it, and it is still time.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- TEAM-SPECIFICWith a strong platform or SRE team, feature teams hold pages for their own flows and escalate infrastructure. Without one, the feature team holds everything, and the runbook has to cover the infrastructure steps too.
- STAGE-SPECIFICEarly on, "everyone is on call" is informal and the founder is the escalation. Past a few teams, it needs routing, rotations and a written ownership table, or pages land on whoever is known to answer.
- PRODUCT-SPECIFICA consumer store can often wait until morning for non-checkout issues and should page at night only for money and orders. A B2B product with contractual availability or a regulated service may have to page for much more, and defines severity by contract.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.