Skip to content

Draft — not approved. A working draft by UASK. It may change without notice.

Service levels in small form

Status
Review
Version
0.1.3
Authority
UASK standard
Applies to
Core
Markdown source

Summary

A supported system needs a short, written answer to “is it working, and how fast do we react?”. This item keeps the idea of service level objectives from site reliability engineering and cuts it to what a small system can run: two to four indicators users would notice, a target for each over a window, a check from outside that measures it, the hours and response times of the support, and a monthly look at the numbers. An objective is an internal target. An agreement with a client that has a price or a penalty is a decision of the owner.

Rules

What is stated

  • R1. A system in Support MUST have a written statement of its service levels, and it MUST be one of the lines that the hand-over checks (ops.production-readiness-and-handover#R7). The statement MAY be one page, and at the lightweight tier it MAY be a single indicator with its target.
  • R2. A statement MUST name between two and four indicators (an SLI), or a single one at the lightweight tier: measures of what a user of the system feels, such as whether the main page and the key path work, how long the key path takes, and, where the system moves data between places, whether the data arrived in time. It MUST NOT use internal measures such as processor load as an indicator.
  • R3. Each indicator MUST be defined so that two people compute the same number: what counts as a good event, where it is measured, and how often.
  • R4. Each indicator MUST have a target (an SLO) over a stated window, for example “99.5% of the outside checks of the order page succeed over 30 days”. A target MUST NOT be 100%, and MUST NOT be set only from what the system happens to do today; it MUST come from what the owner or the client needs and what the system can bear.
  • R5. The statement MUST give the allowed gap, the error budget (100% minus the target, over the window), in time or in events, so that everyone can see how much failure the target tolerates.

Support and agreements

  • R6. The statement MUST give the support hours, the time within which a supporter reacts to a report, and the time within which service is to be restored, by severity. These times MUST be kept apart from the indicators: they say how fast people act, not how well the system works.
  • R7. An agreement with a client (an SLA) that gives the client a remedy, a refund or a penalty for a missed target MUST be decided by the owner in writing; it is a decision about money and commitments (mgmt.roles-and-decision-rights#R9). No one MUST promise a client a level that is not measured.

Measuring and acting

  • R8. Each indicator MUST be measured by a check that runs outside the system, from a place that does not share its failure, and its results MUST be kept for the window. A target that nobody measures MUST NOT be reported as met.
  • R9. Alerts SHOULD fire on what users feel, and SHOULD be tied to the target: a failing outside check that repeats, or an error budget being spent faster than the window allows. An alert MUST reach a named person (ops.production-readiness-and-handover#R6).
  • R10. The statement MUST be reviewed with the numbers at least monthly at the standard tier and above, at each hand-over, and when traffic, the tier or the owner’s needs change. A missed target MUST lead to a record: an incident record (ops.incidents-and-postmortems), a change, or a decision by the owner to accept the level.
  • R11. When the error budget of the window is spent, the lead MUST tell the owner and put reliability work ahead of new features until the budget recovers or the owner decides otherwise. At the high-assurance tier this MUST be written down as a policy that the owner has agreed.

Guidance

A statement in small form

Two indicators and one line of support are enough for a small shop.

Indicator Good event Target (30 days)
The order page works an outside check each minute gets a page with status 200 and the basket button 99.5% of checks
Orders reach the back office an order placed is in the accounting export within 15 minutes 99% of orders

Support (R6, kept apart from the indicators): hours 9:00 to 17:30 UK time, Monday to Friday; a report is answered by a person within 4 working hours.

99.5% over 30 days leaves about 3.6 hours of failed checks. Write that number next to the target: it is the error budget (R5), and it is what the monthly review looks at.

How to choose

Start with one indicator for the thing that would make the owner phone you, and add a second only when the first misses a failure that mattered. Keep the aggregation simple: a count of good checks over all checks is easier to explain than a percentile of a derived measure. Avoid absolutes: “always available” is not a target, and a target that is met only by heroic effort will soon be ignored.

What to measure with

A free outside uptime monitor from another provider is enough to start; for the key path a scripted check that places a test order is better. Keep the results; a chart of a month is the evidence at the review.

Objectives and agreements

An objective (SLO) is what the team aims for. An agreement (SLA) is what is promised to a client with consequences. Teams often write an agreement and call it an objective, or promise a client the objective. Keep the objective tighter than anything promised, and write the promise only when the owner has chosen to make it.

Common mistakes

  • Indicators taken from what is easy to read on the server rather than what the user feels.
  • A target of 100%, or of “five nines” for a system with one supporter.
  • Reaction times written as if they were availability.
  • Targets reported as met from memory; no check, no claim (R8).
  • A statement written at hand-over and never looked at again.

Explanation

Why indicators, objectives and agreements are kept apart. The SRE book separates the measure (indicator), the target (objective) and the contract (agreement) because people use “SLA” for all three, and the confusion makes a team promise what it has not measured. Why few and from outside. Indicators that users do not feel produce alerts that nobody believes, and a check from inside the failing system goes dark with it. The book advises as few objectives as possible and simple aggregations for the same reason. Why an error budget. A target below 100% says how much failure is acceptable; spending it on purpose, for a release, is fine, and finding it spent is a signal to change what is built next. The chapter on risk and the workbook describe a budget agreed between those who build and those who run; in a small project the lead and the owner are those people (R11).

Examples

Good. A small UK online shop in Support has the statement of the table above. An outside monitor checks the order page every minute and a nightly script compares placed orders with the accounting export. At the monthly review on 1 December the page met 99.7% (target 99.5%, budget 3.6 hours, used 2.2) and one order missed the 15 minutes because a mail queue stuck; the supporter records it as a change (a retry of the export). The owner sees one line of numbers and approves the next month’s plan.

Bad. The statement says “the site will be available 24/7 and any problem fixed immediately”. There is no check, no number and no window; the client is told it is guaranteed; when the host restarts the database for forty minutes, nobody can say whether the promise was kept, and the owner learns of it from the client.

Sources