Incidents and postmortems
- Status
- Review
- Version
- 0.1.2
- Authority
- UASK standard
- Applies to
- Core
Summary
An incident is an unplanned event that stops or degrades a system people rely on, loses data or exposes it. This item says how a small team handles one: a single incident lead, service restored before the cause is understood, a written log, the owner told in time, and no promise to a client that the owner has not made. It then says when to write a blameless postmortem, what it holds, and the rule that an escaped defect is never only fixed: it becomes a new check, so the same failure is caught earlier next time.
Rules
Declaring and running an incident
- R1. A project in Support MUST state what it counts as an incident and grade it in at most three severities, each with an example: the system down or data lost or exposed; a key path degraded; a minor fault with a
workaround. It MUST name the person to whom anyone reports one (
ops.production-readiness-and-handover#R6). - R2. Whoever notices a possible incident MUST report it at once; the report MUST NOT wait until the cause is known. The lead, or the supporter the hand-over names, MUST declare the incident and MUST name one incident lead. Where one person does everything, that person still writes down who they are acting as.
- R3. The incident lead MUST keep a log in one place that others can read: the time, what was seen, what was done and who decided. The log MUST be kept as the incident goes on, not rebuilt afterwards.
- R4. The first aim MUST be to restore service. The usual first step MUST be the rollback written and approved with the last release (
quality.quality-release#R13), which is the standing authority for it (process.project-lifecycle#R8), or a restore from a backup proved by restore (ops.backups#R6), not a search for the cause on the live system. The cause is looked for after service is back, or on a copy. - R5. An action that this practice reserves to the owner (
mgmt.roles-and-decision-rights#R9) MUST NOT be taken in an incident without the owner’s yes or a standing rule that names it (mgmt.roles-and-decision-rights#R17). Restoring over live data that the approved rollback does not cover, changing a domain, spending money and telling a client are such actions; the approved rollback itself does not wait for the owner. The owner MUST be told of a severity-one incident within the time the project states in its service levels, and MUST be asked for a decision with the options and a default (mgmt.roles-and-decision-rights#R12). - R6. A client MUST be told through the owner (
mgmt.roles-and-decision-rights#R7). The statement MUST say what is known and not known, MUST NOT name a cause that is not established, and MUST NOT promise a time that nobody has agreed. - R7. A suspected loss or exposure of personal data MUST be put to the owner at once, because the duty to tell the regulator within 72 hours of becoming aware, where a risk to people is likely, runs from the moment of awareness (UK GDPR Article 33; ICO guidance). The owner decides whether to report; the lead prepares the facts. For a client’s system, where the client is usually the controller of the data, the processor MUST tell the client without undue delay (UK GDPR Article 33(2)); that notice goes through the owner.
The postmortem
- R8. A project MUST set before any incident the criteria for a postmortem. They MUST include: a severity-one or severity-two incident, any data loss, a manual intervention such as a rollback, an incident that monitoring did not
find, and a time to restore longer than the project’s service levels allow (
ops.service-levels#R6). - R9. A postmortem MUST be blameless: it MUST describe what happened and what the people involved knew, and MUST NOT name a person or a team as the cause. It SHOULD be written within five working days of the end of the incident.
- R10. A postmortem MUST hold: a summary, the impact in measured terms (minutes, orders, records, measured against the service levels), a timeline from the log, the contributing causes, what went well, where it was luck, and the actions. It MUST NOT stop at a single root cause where the facts show several.
- R11. Each action MUST have an owner and a date and MUST be made a work package or an issue (
process.work-packages#R1). An action nobody owns MUST be dropped explicitly, with a reason, not left open. - R12. A defect that reached production MUST NOT be handled as a quick fix: it proves that a check was missing. The change that fixes it, whatever its class (
process.project-lifecycle#R10), MUST add a check that would have caught it (process.work-packages#R13): a regression test, a monitor or an alert. Where the defect shows that the Definition of Ready or of Done, or the readiness checklist, missed something, those MUST be changed too. - R13. The postmortem MUST be shown to the owner and kept with the project’s records. A lesson that is general MAY be proposed to the Hub with every client, name and figure removed. The lesson takes the route of
process.knowledge-and-memory#R10and#R11.
Guidance
Roles in a small team
The large-team roles shrink to hats. The incident lead decides and keeps the log; a second person, if there is one, works the fix; the owner or a person the owner names speaks to the client. With one person, write the log as you go; it is what lets you hand over at the end of a long day and what makes the postmortem possible. Hand the lead role over explicitly, in the log, when someone takes over.
Restore first
The temptation is to understand the failure while the site is down. Roll back or restore first, then look. A rollback that was not written before the release is a plan made under stress. If service cannot be restored by either, say so to the owner early and ask for the decision (R5), rather than at the end.
Severity and the clock
Grade by what the user feels and by data, not by how hard it was to fix. A mail queue stuck for an hour is severity two; a shop that cannot take payments is severity one. The clock for telling the owner starts at declaration, not at diagnosis.
What a good postmortem reads like
A postmortem is a short document with a timeline and a table of actions. Its tone is the tone of the facts: “the export job stopped at 02:10 because the disk was full; the alert went to an address nobody reads”. The blameless rule is not about being kind: if people are punished for causes, they hide them, and the next postmortem has less in it. The SRE guidance also says to set the criteria beforehand, so that the decision to write one is not made while people are tired and defensive.
An escaped defect becomes a check
A defect found in production already says something about the checks: they did not catch it. The fix is therefore always accompanied by a new check, at the lowest level that reproduces it, and the question “which line of the Definition of Done or of the readiness checklist should have asked for this?” is put on the same page. Without that second step, the same class of failure returns.
Common mistakes
- Declaring an incident only after the cause is found.
- Investigating on the live system while it is down.
- A postmortem whose root cause is a person (“the engineer forgot”).
- Actions without owners, “be more careful” among them.
- Telling the client a cause the next day proves wrong.
- Fixing the escaped defect as a typo-level change with no new check.
Explanation
Why one lead and a log. The Google SRE book describes a few clear roles and a living incident document as how a team keeps hold of the state of an incident, with an explicit hand-off when the day ends. A single incident lead, a place where the state is written and an explicit hand-off are the smallest version of that, and cost almost nothing in a small team. Why blameless. The same book describes blameless postmortems as a way to learn from failure without hiding it, and recommends setting criteria for writing one in advance. Why a new check. A project that fixes only the symptom keeps the gap that let it through. The Hub’s own method asks for the check as part of the fix, and treats a defect from a live system as the proof of a missing check, because the quick fix is the one that leaves no trace.
Examples
Good. A small UK shop on a Saturday: the payment provider’s confirmation stops reaching the shop, so paid orders stay “pending”. The supporter declares a severity-one incident at 10:05, names herself incident lead and starts a log. She rolls the callback address back to the previous release’s (R4), orders confirm again at 10:40, and the owner is told at 10:20 with two options and a default (R5). The client hears from the owner that “some orders were delayed by about 35 minutes; the cause is being established” (R6). The postmortem on Tuesday records 14 orders affected, the causes (a provider change not announced, no check that a test payment completes, the alert going to an unread mailbox) and three actions with owners; the second action adds a scripted test payment to monitoring (R12), and the readiness checklist’s monitoring line gets a sentence.
Bad. The same failure: nobody declares anything for three hours, the developer investigates on the live site and edits a file there, the owner learns from the client, and the client is told the provider is to blame. The postmortem reads “the developer did not test the callback”. The same failure returns in March.
Sources
- Site Reliability Engineering, chapter 14, Managing Incidents — Google, CC BY-NC-ND 4.0; linked and summarised only, not copied: incident roles and a live incident document. Accessed 2026-11-21.
- Site Reliability Engineering, chapter 15, Postmortem Culture — Google, CC BY-NC-ND 4.0; linked and summarised only: blameless postmortems, criteria set beforehand, actions with owners. Accessed 2026-11-21.
- ICO, UK GDPR data breach reporting — Information Commissioner’s Office, Open Government Licence v3.0; assess the risk, and notify within 72 hours where a risk is likely. Accessed 2026-11-21.
- The method of the owner’s operating-system project (a defect found on a live machine is never only fixed: the fix brings a new check), described here as a pattern without paths.