Skip to content

Draft — not approved. A working draft by UASK. It may change without notice.

Fitness functions - architecture rules as self-testing checks

Status
Review
Version
0.1.2
Authority
UASK standard
Applies to
Core
Markdown source

Summary

A rule that only a document states is a wish: the first deadline breaks it and nobody sees. A fitness function is an automatic check that gives an objective verdict on one architectural characteristic (a dependency rule, a contract, a number from a quality scenario) and runs with the project’s other checks. This item says how to make one, why each must prove itself on a deliberately broken sample, and how to keep the document, the rule and the check from drifting apart: they change in one edit. It builds on the dependency rules of arch.building-blocks-and-dependencies, the contracts of arch.context-and-interfaces and the measures of arch.quality-attributes.

Rules

What gets a check

  • R1. Every architecture rule that a Blueprint states MUST have a fitness function, or a dated plan to have one, from the standard tier: each dependency rule, each interface contract that others depend on, and each quality scenario whose measure can be read by a tool. Rules for which the plan is “manual until” a date SHOULD name the person who checks.
  • R2. Each fitness function MUST be listed in a register that is kept with the Blueprint, with an id (FF-1, FF-2, …) that is never reused, the rule, contract or scenario it protects, what it measures, its threshold, where it runs, who is told when it fails, and how to run it on a developer’s machine.
  • R3. Where it runs MUST be one of: the merge gate (it blocks a change that breaks the rule); a schedule (it runs nightly or weekly and raises an alert); or a monitor in production (it reads a measure and raises an alert). A rule that a pull request can break MUST be on the merge gate.

A check that proves itself

  • R4. Every fitness function MUST have a self-test: a deliberately broken sample that the check has to reject, and a clean sample that it has to accept. The self-test MUST run with the check, and a change to the check MUST NOT merge while its self-test fails.
  • R5. A fitness function MUST fail when it finds nothing to check: an empty set of files, a missing report, a monitor that has sent no reading. A check that cannot fail MUST NOT count as a check.
  • R6. A failure message MUST name the rule (its id), the place that broke it, and the way out: fix the code, or record an exception (R9). It SHOULD be short enough to read in a terminal and plain enough for an agent to act on.

One edit

  • R7. The document that states a rule, the rule’s id, the fitness function and its self-test MUST change in the same pull request. A pull request that changes one of them without the others MUST NOT be merged until the reviewer has seen why.
  • R8. The ids of the rules in the document and the rules named by the checks MUST be compared by a script that fails when a rule has no check and no dated plan, or a check names no rule. At the high-assurance tier this script MUST run on every change; at the standard tier it SHOULD.

Exceptions and thresholds

  • R9. A rule that is broken on purpose MUST have a recorded exception: the rule id, the place, the reason, the owner and an end date. An exception without an end date MUST be recorded as an ADR (arch.adr). Violations that exist when a check is introduced MAY be listed as a baseline, and the check MUST then fail on any violation that is not in it.
  • R10. A threshold MUST come from a quality scenario, an ADR or a constraint, and MUST be changed by the edit of R7. A check MUST NOT be disabled, loosened or skipped to get a change through; the rule is changed by an ADR or an exception, and then the check changes with it.

Tiers

  • R11. At the lightweight tier, a project MUST have at least one fitness function, for the rule whose breaking would cost most (a cycle, a secret in the repository, a table read outside its block), and R4 and R5 apply to it. At the standard tier, all the rules apply. At the high-assurance tier, every top quality scenario MUST have a fitness function or, where no tool can read its measure, the rehearsal that arch.quality-attributes#R11 allows, with its date, and the results of the checks MUST be kept as evidence of each release.

Guidance

Four kinds of check

Kind Protects Example Runs
Structure a dependency rule, a layout, a naming rule no block but the catalogue names the shop’s tables; no cycles between blocks merge gate
Contract an interface that others use the export’s columns equal the contract; a recorded sample still parses merge gate
Measure a number of a quality scenario the 95th percentile of the product page; the age of the newest dump schedule or monitor
Process a rule of the method every ADR that is owner-level has a confirmation; every rule id has a check merge gate

Tools exist for the structural kind in most languages, such as ArchUnit for Java, and a script of a few lines does the job where none fits.

Writing one

  1. Start from a rule with an id (DR-1) and a reason.
  2. Write the smallest check that fails on a break. Count what it looks at.
  3. Write the broken sample first: a file, a data set or a copy of the real tree with one deliberate fault. Watch the check fail on it, and with the rule’s id in the message. This is the red step.
  4. Write the clean sample and watch the check pass.
  5. Put both in the check’s self-test and run the self-test with the check.
  6. Register the check (R2), and put its id next to the rule in the document.

Do the same when a check is changed. A change that makes a check accept something that the self-test says it must reject is a change to the rule, and goes through R7 and R10.

Keeping the document and the rule in one edit

The simplest way to keep R7 is to write the rule id in both places: in the Blueprint (DR-1) and in the check’s message and name. A short script reads the ids from the Blueprint and from the register and fails on any that appear in one and not the other (R8). The Hub provides one, tools/check_fitness_register.py: it takes the documents that define the rules (--rules, repeatable) and the document with the register (--register, a table with the columns Id and Protects), fails on a dependency rule that no check protects and that has no dated plan (manual until YYYY-MM-DD), on a plan whose date has passed, on a check that names no rule, and on a run that finds nothing to compare. Then a change to a rule that skips the check, or a new check for a rule nobody wrote down, fails the build.

Exceptions and the baseline

A project that adopts a rule late has violations. Do not weaken the rule; list the violations that exist, once, as a baseline, and let the check fail on any new one. The baseline shrinks as the violations are fixed, and a shrinking list is also a measure of the technical-debt register. An exception that lasts has the same shape as a decision: a reason and a date.

Explanation

Why a check and not a review. A reviewer can see a violation in the code they read, and no reviewer can see all of the code on every change. Architecture decays by many small shortcuts, each of which looks reasonable alone. A fitness function looks at everything each time. The term comes from evolutionary architecture, in which a fitness function gives an objective assessment of an architectural characteristic, and the aim is that the characteristic is preserved in an automated, continual manner.

Why it must prove itself. A check that never fails gives false comfort, and the most common cause is a mistake in what it looks at: a path that matches no files, a pattern that matches nothing. A broken sample that the check must reject, and a refusal to pass when it sees nothing, turn “it ran and was green” into “it can see the fault”. That is the same idea as the failing test before the code, applied to the check.

Why one edit. There are three things to keep true: the sentence in the document, the rule the check enforces and the sample that tests it. If they can change separately, they will, and the first to be left behind is the document. Moving them together costs one pull request and removes a kind of drift.

Why a baseline, not a loosening. A rule that is relaxed to let old code pass stops protecting the new code. A baseline keeps the rule strict where it can be and honest where it cannot.

Why this is where the Hub’s decisions meet the code. The confirmation of an owner-level ADR (arch.adr#R12) is, where it can be, a fitness function. The dependency rule of the building blocks becomes one. The measure of a quality scenario becomes one. Together they are what makes the Blueprint something the build can contradict.

Examples

Good. A synthetic example: the shop of the earlier items has the dependency rule DR-1, “no block except the catalogue (B-2) names the shop’s tables”, with the reason “the catalogue may change its tables”. Its check is a short script:

#!/usr/bin/env python3
"""FF-1, rule DR-1: no block except the catalogue (B-2) names the shop's tables. Usage: ff1.py <source-root>"""
import datetime, pathlib, re, sys
TABLES = re.compile(r"\b(shop_products|shop_orders|shop_stock)\b")
EXCEPTIONS = {"export/orders.php": "2027-03-01"} # DR-1, TD-2, with its end date (R9)
root = pathlib.Path(sys.argv[1])
active = {f for f, end in EXCEPTIONS.items() if end >= datetime.date.today().isoformat()} # an ended exception no longer excuses
files = [p for p in root.rglob("*.php") if "catalogue" not in p.parts and p.relative_to(root).as_posix() not in active]
if not files:
sys.exit("FF-1: no files found to check - the path is wrong, or the check cannot fail")
bad = [f"{p}:{n}: DR-1: table named outside B-2; fix the code or record an exception (R9)" for p in files
for n, line in enumerate(p.read_text().splitlines(), 1) if TABLES.search(line)]
print("\n".join(bad))
sys.exit(1 if bad else 0)

Its self-test runs it on two small trees kept in the repository. The clean one has a page that names no table: the check must exit 0. The broken one has pages/basket.php with SELECT * FROM shop_orders: the check must exit 1 and print DR-1 and the way out. A third run on an empty folder must exit 1 (R5). One file, export/orders.php, is on the list of exceptions with its end date, because the nightly export reads the order tables directly for now (R9); the script reads the date, so after 2027-03-01 the file is checked like any other and the build fails until the debt is paid or the exception is renewed. The register of the shop’s checks:

Id Protects Measures Threshold Runs Told Run locally
FF-1 DR-1 files outside B-2 that name a table none allowed, but the exception below merge gate the author, by the failing build python3 ff1.py src
FF-2 DR-2 files outside B-3 that name the payment provider none allowed merge gate the author python3 ff2.py src
FF-3 DR-4 a cycle in the graph of the blocks none allowed merge gate the author python3 ff3.py
FF-4 I-3 the columns of the nightly export against its contract equal merge gate the author python3 ff4.py
FF-5 NFR-4 the page’s largest content on a mid-range phone, 95th percentile under 2.5 s weekly schedule the lead, by mail the synthetic check’s own command
FF-6 NFR-6, the operability block the age of the newest database dump under 26 hours monitor the owner, by mail ls -l on the dump folder
FF-7 DR-3 the rights of the export’s database account read only weekly schedule the lead, by mail python3 ff7.py

The exception register of the same shop: DR-1, export/orders.php, “no time for an interface before the sale”, owner: the lead, until 2027-03-01, recorded as the technical-debt entry TD-2 (arch.risks-and-technical-debt). When the shop moves the stock into its own table shop_stock_levels, the pull request changes the pattern in ff1.py and the broken sample. When it also adds a block B-6 that may name the table, the sentence of DR-1 in the Blueprint changes too, and the pull request carries all of it together (R7): the sentence, the pattern, the broken sample and the register row. The script of R8 compares DR-1 in the Blueprint with DR-1 in the register and finds both.

Bad. The Blueprint says “pages must never read the database directly”, and a check was written for it with the pattern src/pages/*.php. The folder was renamed in the second month, so the pattern matches nothing and the check prints “0 violations” on every change. The rule is broken in the third month, and nobody learns until a table is renamed and the pages stop working. The check had no broken sample and did not fail on an empty set (R4, R5), and the folder rename changed the code without the check (R7).

Sources