Evaluation and conformance
Summary
Version: 0.2
1. Principle
A system is not evaluated by whether it boots, nor by the number of features it contains. Evaluation answers:
- can the system be reconstructed from its own record;
- is everything in it accounted for and authenticated;
- does it hold its security properties in its delivered configuration, not in a hardened variant;
- does it protect the data of the people who run it;
- can it be updated, rolled back and eventually retired safely;
- are its claims bounded by evidence.
A pipeline that is green proves that the pipeline ran. It does not prove any of the above.
Rules
2. Evidence levels
| Level | Name | What it establishes |
|---|---|---|
| E0 | Declaration | someone states the property holds. Not evidence. |
| E1 | Specification review | the specification, lock and policy have been read and are consistent with the requirement. |
| E2 | Offline artefact verification | the produced artefacts were inspected mechanically without booting: configuration, kernel options, signatures, bill of materials, scan results, licence and trademark checks. |
| E3 | Emulated dynamic verification | the system was booted in a virtual environment with firmware and a virtual security module, and the behaviour was observed. |
| E4 | Reference-hardware verification | the property was demonstrated on the declared reference hardware, including firmware interaction, storage encryption unlock, suspend, update and rollback. |
| E5 | Field evidence | the property is confirmed by operational data the producer is lawfully entitled to hold, support records, incident history or independent audit. |
A property may only be claimed at the level at which it has been observed. A kernel option present in a configuration file is E2 evidence about the configuration, not E4 evidence about the running system.
The artefact is not the medium. A property verified on a mounted image is evidence about the image, not about what a recipient will boot. An image may rely on filesystem features the delivery medium does not have — symbolic links resolve inside ISO9660 and do not exist on FAT32 — so a defect can be invisible to every check that looks inside the image and still reach the user. Where the product is delivered on a medium, E2 MUST be established on the medium as written, not only on the image as mounted.
Required minimum levels:
| Requirement class | Minimum |
|---|---|
| specification, provenance, licensing, bill of materials | E2 |
| isolation, hardening, defaults, credentials | E2, with E3 for anything that must be observed at run time |
| boot integrity, storage encryption, measurement | E4 |
| update, rollback, anti-rollback, interruption safety | E4 |
| recovery path | E4 |
| privacy behaviour of the delivered system | E3 minimum, E4 for devices |
| accessibility of installer, first run and recovery | per the applicable HIF profile |
3. The verification corpus
Every product maintains a verification corpus. Each entry has the form:
CHECK-IDrequirement SCF-*property what must be truemethod the tool, command or procedureenvironment offline | emulated | reference hardwareexpected resultevidence levelownerThe baseline corpus below is mandatory for every profile unless a profile records an exception. It is intentionally expressed as properties and methods, not as tool invocations, because tools change faster than obligations.
The baseline is a floor, not a complete mapping of the Constitution. Where a requirement has no entry here, the product MUST add one to its own corpus before claiming that requirement met; a requirement claimed without an entry is an E0 declaration.
3.1 Specification and provenance
| Property | Method | Environment |
|---|---|---|
| The lock resolves every input to a version and digest | inspect the lock; fail if any input is unpinned | offline |
| Every origin has an authenticated trust anchor under producer control | inspect source configuration in the built image and in the build record | offline |
| The build consumed only the lock | compare build log inputs against the lock | offline |
| The release is rebuildable | rebuild from lock in a clean environment and compare | offline |
| No trust-disabling flags anywhere in the delivered configuration | search the image for options that disable signature or certificate verification | offline |
3.2 Artefacts and attestation
| Property | Method | Environment |
|---|---|---|
| Manifest complete and consistent with artefacts | validate manifest against produced files and digests | offline |
| Bill of materials present, valid and covering top-level components | schema validation plus coverage comparison against the closure | offline |
| Provenance attestation present and verifiable | verify the signature and the referenced lock | offline |
| Every distributed artefact is signed and its digest published | verify signatures with the published key | offline |
| Verification instructions are published with the artefact | inspect the delivery surface | offline |
| Corresponding source is available from the same place | fetch it as a recipient would | offline |
3.3 Boot integrity
| Property | Method | Environment |
|---|---|---|
| Boots with firmware verification enabled | boot with verification on | emulated, then hardware |
| Boot components carry current revocation generation | compare the shipped generation against current revocation policy | offline |
| Boot chain components are signed by keys the chain trusts | inspect signatures of loader and kernel | offline |
| Measurement values are as documented, where the profile requires measurement | read measurements after boot | emulated, then hardware |
| System volume integrity enforced where the profile requires it | verify at run time and attempt a modification | emulated |
| Anti-rollback refuses an older version | attempt to install and boot an older release | hardware |
3.4 Cryptography and credentials
| Property | Method | Environment |
|---|---|---|
| Storage encryption uses parameters compliant with the algorithm policy | inspect the header of an encrypted volume created by the installer | emulated |
| Key binding to platform state is bound to a meaningful state | inspect the enrolment and change the state deliberately | hardware |
| No deprecated primitives in any trust decision | inspect signature and protocol configuration | offline |
| No shipped credentials, no empty passwords, no unintended privileged accounts | inspect account and key material in the image | offline |
| Automatic login and passwordless administration are off | inspect display-manager and privilege-escalation configuration | offline |
| Long-lived secrets are generated at first boot | launch two instances and compare | emulated |
3.5 Isolation and attack surface
| Property | Method | Environment |
|---|---|---|
| Mandatory access control enforcing, no unresolved denials on first boot | read the enforcement state and the audit record after a clean boot | emulated |
| No unexpected listening services | enumerate listening sockets after first boot and scan from outside | emulated |
| Firewall active with default-deny ingress | read the effective ruleset | emulated |
| Kernel hardening baseline met | check the shipped kernel configuration and boot parameters against the baseline | offline |
| Unsigned kernel code cannot load | attempt to load an unsigned module | emulated |
| Network-facing services are confined | inspect the confinement of each such service | offline |
| Installed surface matches the profile | compare the installed set against the declared selection | offline |
3.6 Data and privacy
| Property | Method | Environment |
|---|---|---|
| No network contact after first boot other than declared endpoints | observe traffic on a clean instance for a defined period | emulated |
| Telemetry off until an explicit choice | inspect defaults and the first-run surface | offline plus emulated |
| No stable device identifier transmitted without necessity | inspect observed traffic | emulated |
| Erasure and reset return the system to its as-delivered state | perform the documented procedure and inspect the result | hardware |
| Logging can be disabled and its consequences are documented | perform the procedure | emulated |
3.7 Updates and vulnerabilities
| Property | Method | Environment |
|---|---|---|
| Security updates apply automatically by default | inspect configuration, then observe on a running instance | emulated |
| Updates are verified before application | attempt an update signed by an untrusted key | emulated |
| An interrupted update leaves a bootable system | interrupt at several points | hardware |
| Rollback returns to the previous known-good state | perform a rollback after a deliberately broken update | hardware |
| Offline update path works with the same guarantees | perform an offline update | emulated |
| No known exploited vulnerability in delivered components | scan the bill of materials against current vulnerability data | offline |
| Exploitability statements exist for accepted findings | inspect the published statements | offline |
3.8 Legal and lifecycle
| Property | Method | Environment |
|---|---|---|
| Every component has an identified licence and required notices are delivered | inspect the licence inventory and the image | offline |
| No upstream mark subject to removal remains in the image | automated search of filesystem, boot media, artwork, metadata and strings | offline |
| Support period and end-of-life date are declared and visible | inspect the product record and the delivery surface | offline |
| Baseline conformance measured where the profile requires it | run the conformance scan against the named baseline revision | offline |
3.9 Human surfaces
Verified per the applicable HIF profile, with at least: installation without data loss on a machine with existing data; first run completed with a keyboard only; first run completed with a screen reader; every destructive action correctly describing its scope; recovery entered and exited without a network.
4. Severity
S0 Blocker— the system cannot be trusted or used: it does not boot, an update bricks it, data is lost, a secret is shared across installations, an artefact is unsigned or unaccounted for, or an active exploitation report cannot be handled.S1 Critical— a security or privacy property claimed by the product does not hold in the delivered configuration.S2 Major— a required property holds only after manual configuration, or rollback and recovery work only in some cases.S3 Moderate— evidence is weaker than the required level, or documentation contradicts behaviour.S4 Minor— a local inconsistency with no effect on the properties claimed.Observation— a hypothesis, not a confirmed defect.
Severity is determined by consequence for the people and organisations running the system, not by how visible the problem is.
5. Conformance levels
SCF-Declared
- profiles chosen;
- the traceability matrix exists;
- known exceptions are registered.
This marks the beginning of a process. It is not a quality signal and MUST NOT be presented as one.
SCF-Conformant
- every applicable MUST is met;
- every
SHOULDexception is justified and registered; - the whole baseline verification corpus has been executed at its required evidence level, including reference-hardware checks;
- the release is rebuildable from its lock;
- the bill of materials, provenance and signatures are complete;
- the support period, update contract and disclosure policy are published.
SCF-Validated
- SCF-Conformant, plus:
- the release has been rebuilt independently and compared;
- high-risk paths — boot integrity, update, rollback, recovery, erasure — have been demonstrated on reference hardware;
- the vulnerability process has been exercised, not only described;
- operational monitoring for the support period is defined and staffed.
A conformance statement MUST name the SCF version, the profiles, the release, the architectures and the reference hardware to which it applies. A statement without those bounds is void.
6. Traceability matrix
obligation (law, threat model, customer requirement)→ SCF requirement→ product requirement→ specification field, component or policy→ implementation→ automated check→ manual or hardware scenario→ current evidence and its level→ exceptionsA requirement with no path to evidence is incomplete and MUST NOT be counted as met.
7. Release gate
Before a release is signed:
- the scope of change is defined and the affected profiles listed;
- the lock is complete, and every input is authenticated and pinned;
- the build ran in a compliant boundary and its log is retained;
- all offline checks pass, including licence, trademark and vulnerability checks;
- the bill of materials, manifest and provenance are complete and valid;
- the boot chain generation is current against revocation policy;
- emulated dynamic verification passes;
- reference-hardware verification passes for boot, update, rollback, recovery and storage encryption;
- accessibility checks required by the applicable HIF profiles pass;
- the support period, channel and end-of-life date are set and published;
- every open exception has an owner and an unexpired review date;
- a named person approves, and the approval is recorded with the evidence.
The rollback plan and the criteria for withdrawing the release are part of the gate, not a later decision.
8. Exceptions
IDSCF version and requirementproduct and profilereasonaffected users, devices and datarisk and likelihoodcompensating controlownerexpiry or review dateevidenceAn expired exception blocks a new conformance statement until it is reviewed. An exception MUST NOT be used to carry a permanent architectural decision; if a requirement is wrong, propose a change to the standard.
9. After release
The support period is part of the product, so evaluation does not stop at release.
- monitor upstream for changes to the inputs of every supported release, and be able to state how far behind each release is;
- monitor vulnerability data against the bill of materials of every supported release;
- monitor revocation and signing-authority policy;
- record time from report to fix to published advisory, and review it;
- re-run the verification corpus for every update that touches boot, kernel, cryptography, update logic or defaults;
- record field failures against the corpus, and add a check for every failure the corpus did not catch.
10. Changing the standard
If a product cannot meet SCF:
- check whether the problem is local architecture rather than the requirement;
- check the alternatives, including a different platform family;
- register an exception with evidence;
- gather evidence over at least one release cycle;
- only then propose a change to the standard.
A widespread industry practice is not evidence that a requirement is wrong. The absence of an upstream facility is a constraint to be recorded, not a justification for a weaker claim.