Skip to content

Draft — not approved. A working draft by UASK. It may change without notice.

Status: review · Version 0.1.0

Quality and release

Summary

This playbook takes work from “ready to start” to “released, verified and recorded”. It sets up a project’s quality baseline once — Definition of Ready, Definition of Done, test strategy, an automated gate, a changelog — and then runs every release through the same path: version, changelog entry, build, staging check, approval, backup, deploy, post-release verification, and rollback or hotfix when something goes wrong.

It does not restate rules that have a home elsewhere: versioning is config.semantic-versioning, backups are ops.backups, commit messages are config.conventional-commits. For operating systems the System Construction Framework (SCF) applies in full; this playbook names the SCF rules it lines up with.

Rules

Rules are cited as quality.quality-release#R<n>.

Set up (once per project)

  • R1. A project MUST keep a Definition of Done in its repository, started from templates/definition-of-done.md, that every change merged to the main branch meets; the Definition of Done MUST be changed only through a reviewed change.
  • R2. A project SHOULD keep a Definition of Ready, started from templates/definition-of-ready.md; work on an item that does not meet it SHOULD NOT start.
  • R3. A project MUST keep a test strategy, started from templates/test-strategy.md, that states for each test level it uses what the level covers, where it runs and which gate it blocks, and that lists the critical user paths checked after every release.
  • R4. A project that ships releases MUST keep a changelog file in its repository, started from templates/changelog.md.

Every change

  • R5. Every change to the main branch MUST pass the automated gate defined in the test strategy before it is merged.
  • R6. The gate MUST be a script kept in the repository that runs the same checks locally and in CI, so that any CI failure can be reproduced outside CI.
  • R7. A failing gate MUST NOT be bypassed by merging anyway; a check MAY be disabled or quarantined only in its own reviewed change, with an issue that says why and when it returns.
  • R8. A defect fixed after a release MUST come with a regression test at the lowest test level that reproduces it.

Every release

  • R9. A release MUST be versioned and tagged as config.semantic-versioning requires, and its tag MUST point to a commit that passed the gate.
  • R10. A release MUST have a changelog entry written for its users that marks breaking changes, deprecations and security fixes and, for a MAJOR release, states the upgrade steps; a list of commits alone does not meet this rule.
  • R11. Release artefacts MUST be produced by a scripted build from the tagged commit and SHOULD be produced by CI rather than on a personal machine; their checksums MUST be recorded.
  • R12. Each release MUST have a release checklist, filled in from templates/release-checklist.md, whose deploy plan and rollback plan are written before deployment starts.
  • R13. The rollback plan MUST state the trigger that means “roll back”, the steps, the expected time and who decides; when the release changes stored data, it MUST state whether that change can be reversed and, if not, how the data is restored from the pre-change backup.
  • R14. A release MUST be approved by a named person against its checklist before it reaches production; a green pipeline alone is not an approval.
  • R15. A release SHOULD be verified on a staging or test environment that matches production before it is deployed there; where no such environment exists, the checklist MUST record that as an accepted risk.
  • R16. A release MUST NOT knowingly ship a dependency with a vulnerability listed as known to be exploited (for example in the CISA KEV catalogue) unless a named person accepts it as a recorded exception.

Deploy and verify

  • R17. Before a deployment to a live system that holds data, a pre-change backup MUST be taken and recorded as ops.backups#R4 and ops.backups#R5 require, and the deployment MUST NOT start until the backup is confirmed.
  • R18. A deployment SHOULD run in a window chosen from traffic or load data, with someone able to roll back available until verification ends.
  • R19. Right after deployment, post-release verification MUST check the critical user paths from the test strategy, the error logs and the monitoring signals, and record the result in the checklist.
  • R20. When a rollback trigger fires, the release MUST be rolled back as planned; the approver MAY instead decide to fix forward, and the fix then follows the hotfix rules R22 and R23.
  • R21. The release record — version, commit, artefact checksums, changelog entry, completed checklist with verification results, backup location and approver — MUST be kept for as long as the release is supported.

Hotfix

  • R22. A hotfix MUST be the smallest change that removes the defect, released as a PATCH of the affected release line, and MUST pass the same gate, backup and post-release verification as any release; only the checklist steps marked as shortened for hotfixes MAY be shortened.
  • R23. A hotfix MUST also land on the main branch, with its regression test (R8), so that the next release does not bring the defect back.
  • R24. Changes SHOULD NOT be made directly on a production system; when an emergency requires one, it MUST be recorded and brought into the repository through this playbook within one working day.

Stages

# Stage When Artefact Template Drafts → approves
1 Set up once; review at each MAJOR DEFINITION-OF-READY.md, DEFINITION-OF-DONE.md, TEST-STRATEGY.md, CHANGELOG.md, gate script templates/definition-of-ready.md, templates/definition-of-done.md, templates/test-strategy.md, templates/changelog.md lead developer → owner
2 Build a change every backlog item pull request with evidence — lead developer → reviewer
3 Prepare the release main holds what should ship changelog entry, release candidate, filled checklist templates/release-checklist.md lead developer → release approver
4 Back up and deploy approved release backup record, deploy log templates/release-checklist.md lead developer
5 Verify right after deployment verification results templates/release-checklist.md lead developer → release approver
6 Close end of the observation period release record, follow-up issues templates/release-checklist.md lead developer
7 Hotfix defect that cannot wait PATCH release, regression test templates/release-checklist.md (hotfix lane) lead developer → release approver

1. Set up

  • Entry. The project has a repository and a main branch (Project Foundation is done), or an inherited project has no written quality baseline — an Architecture Review often finds this.
  • Do. Copy the templates and cut them down to what the project really runs: a Definition of Done that lists checks nobody performs is worse than a short honest one. Choose the test levels (see Guidance) and write each one into the test strategy with the command that runs it; record missing levels as issues instead of inventing tests. Write one gate script — for example tests/run.sh or a make check target — and call it from CI. Make the gate block merging: a required status check where the hosting allows it; where it does not, a written rule that nobody merges on red plus a local pre-push hook. Start CHANGELOG.md with an Unreleased section.
  • Outputs. DEFINITION-OF-READY.md, DEFINITION-OF-DONE.md, TEST-STRATEGY.md, CHANGELOG.md, the gate script and its CI job, issues for the gaps.
  • Exit. R1–R6 are met: the files are merged, the gate runs locally and in CI with the same command, and a failing gate stops a merge.

2. Build a change

  • Entry. The issue meets the Definition of Ready.
  • Do. Write the test first at the lowest level that can catch the failure. Make the change. Run the gate locally before pushing. Add a line under Unreleased in the changelog when users will notice the change. Open a pull request that shows the evidence the Definition of Done asks for.
  • Outputs. A pull request with evidence for every Definition of Done line, or “n/a” with a reason.
  • Exit. Definition of Done met, gate green in CI, reviewed, merged (R5–R8).

3. Prepare the release

  • Entry. Main is green and holds what should ship; the scope of the release is agreed.
  • Do. Decide the version from the changes since the last release (config.semantic-versioning); when in doubt, it is MAJOR. Turn the Unreleased section into the release entry — a draft generated from Conventional Commits is a starting point, not the entry. Tag a release candidate (vX.Y.Z-rc.1) and let CI build the artefacts and record checksums. Deploy the candidate to staging and run the release-candidate gate there; for a website, staging is a copy with production-like data and the same software versions. Run the ecosystem’s dependency audit (for example npm audit, pip-audit, composer audit) and check findings against the KEV catalogue. Fill in the release checklist: scope, deploy plan, backup plan, rollback plan with triggers. Ask the approver to approve against the checklist.
  • Outputs. Changelog entry, release-candidate artefacts with checksums, staging results, the filled checklist (sections 1–6), the recorded approval.
  • Exit. R9–R16 are met; the approval is recorded in the checklist.

4. Back up and deploy

  • Entry. Approved release; deploy window reached; someone able to roll back is available.
  • Do. Take the backup the checklist names — files, database, configuration — and confirm its completion and size before anything changes. Tag the release commit if only a candidate was tagged. Deploy in the planned window, follow the steps, write down any deviation. For services with many users, roll out gradually (a canary or a share of traffic) when the platform allows it.
  • Outputs. Backup record (where, when, size, how to restore), deploy log.
  • Exit. The release runs in production as planned (R17, R18).

5. Verify

  • Entry. Deployment finished.
  • Do. Run the critical-path checks from the test strategy, read the error logs, and watch monitoring for the observation period set in the checklist. If a trigger fires, roll back first and diagnose afterwards, or fix forward only on the approver’s decision.
  • Outputs. Verification results in section 8 of the checklist.
  • Exit. Verified, or rolled back as planned (R19, R20).

6. Close

  • Entry. Observation period over.
  • Do. Record the outcome and keep the release record: in the repository (for example releases/vX.Y.Z.md), in the release pull request, or attached to the release. Turn every incident into an issue and every escaped defect into a regression test and, if it shows a gap, a change to the Definition of Done or the test strategy.
  • Outputs. The release record; follow-up issues.
  • Exit. R21 is met; the Unreleased section is empty and ready for the next release.

7. Hotfix

  • Entry. A defect in a released version is confirmed, and the approver agrees it cannot wait for the next planned release.
  • Do. Fix on main first with a regression test and cherry-pick to the release branch where the project keeps release branches; otherwise branch from the released tag and merge the hotfix back to main straight away. Release it as a PATCH through the hotfix lane of the checklist. Gate, backup, approval and verification are never shortened. If production was changed by hand, bring that change into the repository first.
  • Outputs. PATCH release with its checklist; regression test; the fix on main.
  • Exit. R22–R24 are met; the hotfix is verified in production and present on main.

Guidance

When to use

  • Set-up — right after Project Foundation, or when an inherited project has no written quality baseline.
  • Every change — the Definition of Done and the gate apply to each pull request; there is nothing to “start”.
  • Every release — anything that reaches users or production: a package, a theme or plugin, a website deployment, an operating-system image, a documentation release.
  • Hotfix — a defect in a released version that cannot wait for the next planned release.

Roles

Role Does Typical holder
Owner sets quality goals; accepts recorded risks and exceptions (R15, R16); approves releases or names who does product owner
Lead developer keeps the Definition of Done, test strategy and gate; prepares the release, writes the checklist, deploys, verifies developer or coding agent
Reviewer reviews changes against the Definition of Done and the checklist before approval another person or a second agent
Release approver the named person of R14; decides on rollback or fix-forward (R20) owner, or a person the owner names

Tailoring

A throw-away prototype that nobody depends on needs only R5–R6. Everything else uses the whole playbook; small projects keep the documents short rather than skipping them. By product type:

  • Library, theme, plugin, package. The public interface (hooks, settings, template variables, CSS classes others override) decides the version. Test against the oldest and newest supported host versions. Releases are artefacts with checksums; “deploy” is publishing them.
  • Website or online shop. Back up files and database before every deploy; staging is a copy of the live site; verification covers search, product page, cart and checkout, forms and e-mail. The deploy window comes from traffic data.
  • Operating system. SCF applies in full: rollback is designed and tested for every release (SCF-UPD-006), automated checks can fail every build (SCF-EVD-002), a named person approves against the gate (SCF-EVD-005), releases are immutable (SCF-ART-005), known-exploited vulnerabilities block release (SCF-VUL-003), the evidence is kept with the release (SCF-EVD-004) and channels have promotion and rollback rules (SCF-LCM-002). Use scf.release-record-template as the release record and this playbook’s checklist for the deploy steps.
  • Documentation or knowledge set. The gate is validation, link checks and the site build; the release is a tag and a bundle; rollback is redeploying the previous tag.

Working with agents

An agent can run stages 1–3 alone and stages 4–7 when it has been given access to the target system. Approval stays with a named person (R14): an agent that merges its own green pull requests is still not approving a release. Agents report gate results as command output, not as “tests pass”. The skill quality-release gives agents the order of work; it does not repeat these rules.

Test levels

The five test levels of the ISTQB Foundation syllabus, with the names developers usually use:

Level Also called Checks Typical gate
Component unit one function, class or module in isolation every change
Component integration integration how components work together: database, file system, framework hooks every change
System end-to-end the whole product through its real interface: a browser, a CLI, a booted image every change if fast, else release candidate
System integration contract, external the product with external services: payment, delivery, APIs release candidate
Acceptance user acceptance the owner or users confirm the product does what they need release candidate

Regression testing is not a separate level: it is re-running the existing tests after a change. Static checks (lint, type checks, schema validation) and non-functional checks (performance, accessibility, security) are attached to the level where they run. Most tests sit at the lower levels; a suite that relies mainly on end-to-end tests is slow and flaky (the test pyramid).

Quality gates

Gate When Typical content
G1 Change every pull request static checks, component and component-integration tests, fast system tests, build
G2 Release candidate before approval full suite, system integration and acceptance on staging, dependency audit
G3 Post-release after deployment critical paths, logs, monitoring signals

The gate is what can fail the build, not what is merely reported. A check that produces only warnings nobody reads is not part of the gate.

Measuring

Track the DORA delivery measures from the release records: change lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate. They show whether the process makes releases both faster and safer; numbers that get worse are a reason to revisit the gate or the batch size.

Anti-patterns

  • “CI is green, ship it.” A pipeline checks what it was told to check; approval is a person reading the checklist (R14).
  • Release from a laptop. Artefacts that nobody can rebuild from a tag cannot be trusted or rolled back (R11).
  • Backup after the fact, or a backup nobody has ever restored. Restore tests are ops.backups#R6.
  • “Rollback: restore the backup.” Without triggers, steps, time and a decision maker it is a hope, not a plan (R13).
  • Changelog as a commit dump. Users need what changed for them and what to do (R10).
  • Moving a tag to “fix” a release. Release a new PATCH instead (config.semantic-versioning#R3).
  • Hotfix by editing production, then forgetting it — the next deploy brings the bug back (R23, R24).
  • Skipping the gate because it is urgent. Urgent changes are the ones most likely to break something (R22).
  • Ice-cream cone tests. Mostly slow end-to-end tests and few component tests: slow feedback, flaky gate.
  • Ignoring flaky tests. A gate that fails at random trains everybody to re-run it until it passes (R7).
  • Big-bang releases. Months of changes in one release make every failure hard to find and every rollback expensive.
  • Friday-evening deploy with nobody on call (R18).

Explanation

Quality is built in while the change is made; the release only proves it and makes it reversible. The Definition of Done makes “finished” a shared, checkable statement instead of a feeling, and the Definition of Ready stops work that cannot be finished from starting. The test strategy decides where each kind of failure is caught — as early and as cheaply as possible — and the gate makes that decision binding for every change.

Releases fail for reasons tests do not cover: data, configuration, load, the environment. That is why the playbook insists on things that look like paperwork — a backup confirmed before the change, a rollback plan with a trigger, a named approver, a record kept afterwards. Each of them is cheap before the deploy and very expensive to improvise during an incident. DORA’s research reports that teams that release in small batches with strong automated checks are both faster and more stable, so speed and safety are not a trade-off.

The SSDF asks producers to test executable code before release (PW.8) and to archive and protect each release with the data needed to verify it (PS.3); SLSA describes how build provenance makes artefacts traceable to their source. R11 and R21 are the small-project form of those practices.

Examples

Good — a theme release 2.4.0:

  1. Unreleased lists “Added: optional currency setting” and “Fixed: tax rounding”; the version is MINOR.
  2. Tag v2.4.0-rc.1; CI builds the package and records its SHA-256.
  3. Staging shop updated; checkout, cart and the new setting checked; no KEV findings in the audit.
  4. Checklist: deploy at 07:00 (lowest traffic in the last 30 days), files and database backed up and size checked, rollback = reinstall 2.3.2 package and restore the settings table, trigger = checkout error rate above 1 % or any PHP fatal in the log, decided by the owner.
  5. Owner approves; tag v2.4.0; deploy; critical paths pass; record kept in releases/v2.4.0.md.

Good — hotfix 2.4.1: a rounding error found on day two. Fix with a component test on main, cherry-pick to the 2.4 line, gate green, backup, owner approves the hotfix lane, deploy, verify checkout, changelog “Fixed: rounding of discounts”, record kept.

Bad — “small fix” edited directly in the live template on Friday night, no backup, no record. On Monday the 2.5.0 deploy overwrites it, the bug returns, and nobody knows which version the shop was really running.

Sources