IncidentsORG-SPECIFICGENERAL

Telling People What Is Happening

Different audiences need different things at different cadences — and none of them should have to interrupt the person fixing it.

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

While an incident is in progress, who needs to be told what, how often, and by whom?

The problem

Everyone outside the incident wants information, the only person who has it is the one working on it, and every request for an update makes the incident longer.

What teams do first

The responder posts updates when there is something to say. If people need more, they can ask.

How it breaks

Silence is read as absence. With no update, stakeholders assume nothing is happening and escalate — usually by messaging the responder directly.

How it breaks in production
  • Silence is read as absence. With no update, stakeholders assume nothing is happening and escalate — usually by messaging the responder directly.
  • Every direct message costs a context switch during focused debugging, and the cost is far higher than the sender imagines.
  • "When there is something to say" means updates stop exactly during the hard part, which is when anxiety outside is highest.
  • Support answers customers with guesses, which later have to be corrected — and a wrong reassurance is worse than no statement.
  • Internal engineering detail leaks into customer-facing messages, or the reverse: internal responders get a vague public statement and cannot tell what is known.
CodeBuildTestArtifactReleaseDeployRunObserveOperateIncidentRecoverLearnImprove

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • Communication during an incident serves three distinct audiences with genuinely different needs. Responders need technical state: what is known, tried, ruled out, in flight. Internal stakeholders — support, leadership, adjacent teams — need impact, expectation and what is being asked of them. Customers need whether they are affected, whether their data is safe, and when to check back.
  • Mixing them fails everyone. A technical channel is unreadable to leadership; a leadership summary is useless to responders.
  • The most important property is cadence over content. A scheduled update saying "no change, still investigating, next update in 20 minutes" is far more valuable than an unpredictable stream of detail, because it removes the reason to interrupt.
  • Communication should come from someone other than the person investigating, whenever there is more than one responder (Roles During an Incident).
  • Impact statements are the unit of value: what is broken, for whom, since when, what to do meanwhile. Cause is interesting and almost never what the audience needs.
  • Over-promising is the standard failure. "Should be fixed in ten minutes" creates an obligation you rarely control, and missing it costs more trust than the outage did.

Three audiences, three messages

The same incident produces three different documents, and the commonest failure is writing one and sending it to everyone.

The last column is the test: if a message does not let its audience do that thing, it is the wrong message for them.

AudienceNeedsDoes not needCadenceShould be able to
RespondersTechnical state: known, tried, ruled out, in flightReassuranceContinuous, in the channelPick up work without asking what has been done
SupportWhat to tell customers, and any workaroundCauseEvery update, promptlyAnswer a customer without contacting engineering
LeadershipImpact, trend, what is being done, what is neededDebugging detailOn a stated cadenceDecide whether to intervene, and otherwise stay out
Adjacent teamsWhether they are affected, and what is asked of themEverything elseOn declaration and on material changeCheck their own dependency and act
CustomersAm I affected, is my data safe, when to check backInternal architectureOn a stated cadence, ending with a closeDecide whether to wait or work around it

Two status updates

ORG-SPECIFICHow much detail is appropriate externally varies with your customers and sector: a developer platform's users often expect and benefit from more technical specificity than a consumer product's. The constant is not naming an unconfirmed cause, and not promising a resolution time.

Both were posted twenty minutes into the same incident. The first is what an engineer writes; the second is what the audience needs.

The right-hand version contains less information and is far more useful, because everything in it is something the reader can act on.

Twenty minutes in, to a customer-facing status page
Written for engineers
Investigating elevated 503s from the
payment gateway. Suspect connection
pool exhaustion following the v482
rollout. Rolling back and monitoring.
Will update when we know more.
Written for the audience
Some customers are unable to complete
checkout. This began at 14:04 UTC and
is ongoing.

Browsing, accounts and existing orders
are unaffected. No data has been lost.
Carts are preserved — please retry in
a few minutes.

We have identified a likely cause and
applied a change. Next update: 15:00 UTC.

The left version names a suspect that may be wrong — and if it is, the correction is now public. It gives no impact scope, no reassurance about data, no workaround, and no time to check back, so every reader has to ask. The right version commits to nothing it does not control, tells the reader what is safe, and removes the reason to contact support.

The cadence, and what each update contains

Updates have a shape that changes as the incident progresses. Committing to the shape in advance means the communicator is filling in a template rather than composing under pressure.

The rule underneath all of it: every update states when the next one comes, and that promise is kept even when there is nothing new.

Update sequence for a customer-visible incident
  1. 1
    First statement

    Acknowledges impact in user terms, with a start time.

    fails by Waiting until the cause is known, so the first news the customer gets is from their own users.

    evidence Posted within minutes of declaration, before diagnosis.

  2. 2
    Scope update

    Says who is affected and who is not, and what is safe.

    fails by Claiming a scope the dashboard cannot actually support (Dashboards an Operator Can Act On).

    evidence Scope claim matches what the postmortem later establishes.

  3. 3
    Regular updates

    Keeps the cadence, even with nothing new to report.

    fails by Going quiet during the hard part.

    evidence Each update names the time of the next one, and it arrives.

  4. 4
    Mitigation update

    States that impact should be reducing, without declaring victory.

    fails by Announcing resolution on the first improvement in the graph.

    evidence Language is "we believe" until user-facing confirmation exists.

  5. 5
    Resolution

    Confirms impact has ended, with the actual window.

    fails by Declaring on error rate while a backlog is still failing users (Operating Queues and Scheduled Work).

    evidence Probe passing, support queue quiet, backlog drained.

  6. 6
    Follow-up

    Says what happened and what is changing, once known.

    fails by Never appearing, which is what most status pages do.

    evidence A published summary consistent with the internal postmortem (Postmortems).

How to do it properly

Most important first.

  • Separate the channels: a technical incident channel, a stakeholder update thread, and a customer-facing status page. Do not merge them.
  • Post the first impact statement within minutes, before you understand anything. "We are aware that checkout is failing for some users since 14:04 and are investigating" is complete and sufficient.
  • Commit to a cadence and state it in every update: "next update at 14:40". Post at 14:40 even with nothing new.
  • Describe impact in user terms, not system terms. "Orders are failing at payment" rather than "the gateway is returning 503s".
  • Say what is unknown explicitly. "We do not yet know whether saved carts are affected" is trusted; silence on the point is not.
  • Give a next-update time rather than a resolution time. You control the first and not the second.
  • Close the loop when it ends: what happened, actual impact window, what you are doing next. Include the backlog and reconciliation, not just the error rate recovery (Reconstructing What Actually Happened).
  • Where data or security may be involved, route through the path your organisation has defined for it before saying anything externally — disclosure obligations are owned by Security Engineering and are frequently statutory.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

Communication does not contain technical blast radius; it contains the organisational one — and a mishandled statement can outlast the outage.

What can go wrong

Failure modes, including of the mitigation
  • The responder becomes the communicator by default and stops responding.
  • Cadence lapses during the hardest part, which is when it matters most.
  • A resolution time is promised and missed, converting a technical incident into a trust incident.
  • Recovery is announced early, then retracted — the single most damaging communication error available.
  • Contradictory messages from different sources because nobody owns the statement.
  • A status page that requires manual updating and is therefore updated last, after customers have already noticed.
  • Speculation about cause published externally and later corrected, which is remembered longer than the outage.
Misreads this invites
  • "No news is good news." Externally it reads as nobody being on it.
  • "Wait until we know the cause." Impact is what the audience needs, and you have that immediately.
  • "Technical detail builds trust." Impact and honesty build trust; detail mostly confuses and occasionally commits you to a story you have to retract.
  • "The status page is marketing's job." It is part of the incident response, and if it updates last it is worse than not having one.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • Stakeholders stopped asking the responder for updates, because the updates arrive on their own.
  • Support handled customer contacts using the published statement rather than asking engineering.
  • The published impact window matched the one the postmortem later established.
  • No update during the incident had to be retracted.
How you get back
  • Correcting a statement is normal and should be done promptly and plainly: what was said, what is now known, and why it changed. A quiet edit destroys more trust than the original error.
  • Never retract a recovery announcement casually. If recovery is uncertain, say "we believe impact has ended and are monitoring" rather than declaring it, and keep the incident open until confirmed (A Successful Deploy Is Not Evidence of a Healthy System).
What to automate, and what stays human
  • Automate the scaffolding: create the channel, post the template, notify the stakeholder list, and start the update timer.
  • Automate cadence reminders to the communicator — the update that gets missed is missed because everyone is busy, not because nobody cares.
  • Automate the mechanical impact numbers into the draft: affected endpoints, error rate, region and tenant scope, start time.
  • Do not automate the customer-facing statement. Wording matters, it carries commercial and sometimes legal weight, and an automated message during a confusing incident will say something wrong (The Automation Trap).
What this costs
  • A dedicated communicator is a responder who is not investigating. At low severity that is a bad trade; at high severity it is one of the best available.
  • Frequent updates cost the communicator's attention and can pressure the team to have something to report.
  • Transparency builds trust and gives competitors, and occasionally attackers, information. Most organisations under-share rather than over-share, but the trade is real and it is a business decision.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • ORG-SPECIFICWho may speak externally, what must be disclosed and within what deadline, and whether legal or communications review is required before a customer statement are all local policy — and in regulated sectors, statutory. What generalises is only the audience separation and the cadence discipline; the approval path does not, and finding out what it is during an incident is too late.
  • GENERALThe three audiences and their different needs hold anywhere software has users and colleagues. What changes is how formal each channel is.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Observability & Performanceslosli
API Designdeprecation
Domains that do not exist yet
  • System Design — how status pages and customer-facing health signals are built so they do not share a failure domain with the system they report on.