Skip to main content

Evidence records

A maturity rung in manifests/adsb.json is a claim. A conformance report is a measurement taken somewhere. Neither is evidence on its own: a reader cannot tell whether maturity: L4 was earned or typed, and cannot tell which tree, which fixtures or which versions a report describes.

An evidence record is the missing document. One JSON file per adapter, carrying the manifest it measured, the commit it measured at, every version axis in force, the Conformance Suite's own report with one field made portable — conformance.adapter.fixtures reads <packaged>/<directory> and never the absolute path the run read, since 2026-09-16, because the path made every record machine-specific — the loss report, and a SHA-256 of every fixture file the run read.

pip install synapse-cdm

synapse evidence generate --all --out evidence/
synapse evidence verify evidence/adsb/1.0.0/evidence.json
synapse evidence provenance
synapse badges --from evidence/

What is in a record​

evidence/<adapter>/<adapter-version>/evidence.json, written in the canonical serialisation ARCHITECTURE.md §6.2 fixes for everything this framework compares — sort_keys=True, two-space indent, UTF-8, trailing newline.

FieldWhat it is
adapterthe adapter's published manifest, embedded whole. Not its id: a record that named its subject would be a record whose subject could change under it
source_committhe commit the run measured, with a -dirty suffix when the working tree had uncommitted changes, or unknown outside a checkout
package_version, cdm_version, sc_oes_version, adapter_api_version, manifest_versionthe five version axes in force, as data — this is what makes a record readable years later
evidence_schema_versionthis record's own shape, on its own axis
test_runcommand, exit, passed / failed / skipped, duration_s, python, platform — the envelope around the run
conformancethe Conformance Suite's own JSON report, with one field made portable: conformance.adapter.fixtures reads <packaged>/<directory> — the label the --all sweep writes — and never the absolute path the run read
loss_reportthe six categories, below, and under loss_report.ledger the path-bound preservation ledger's reading — basis (ledger where the adapter declares MAPPINGS, heuristic where it does not), counts per category and per loss kind, and diagnostics that name paths, rules and types and never a value
fixture_hashes{path, sha256, bytes} for every file the suite read, including malformed/ and the goldens. Paths are relative to the packaged fixture root, never absolute
fixture_provenancethe summary the Synthetic Fixtures badge is grounded in
snapshotwhich tree the run measured, exactly: the commit, whether it was dirty, a digest of the tracked diff, every untracked file with its digest and size, and one digest over the two — null when clean. source_commit is read off it. Added by the audit's F07 (2026-09-19)
evidence_categoriesone reading per kind of evidence — internal_fixture, self_round_trip, independent_expected, normative_schema, independent_endpoint — each PRESENT, ABSENT or NOT_APPLICABLE with a basis sentence and the hashed exercise reports it rests on. Derived, never typed; the three external kinds are PRESENT only from a validated exercise report by an independent implementation, and none exists today
maturity_supportthe DECLARED rung held to the categories it needs, local_complete reported apart from external_outstanding, so a rung the internal evidence supports and a rung only an outside party can supply are never one word
artifact_hashesempty, for a reason of order: the release pipeline generates the records before it builds the wheel and sdist, and verify re-derives a record from the tree, so a digest of an artefact that is not in the tree could never reproduce. The artefact digests are published in SHA256SUMS and in the witness record instead
generated_at, generatorthe instant, and what produced the record

test_run records the conformance suite run — the thing verify re-runs and the thing conformance holds the report of. passed / failed / skipped are therefore counts of the fifteen checks by verdict.

Verifying one​

verify does not read the record back, which would prove nothing. It produces a new record from the tree in front of it — re-hashing every fixture, re-running every check — and compares the two field by field.

$ synapse evidence verify evidence/pntmap/1.0.0/evidence.json
REPRODUCED: evidence/pntmap/1.0.0/evidence.json (masked: generated_at, test_run.duration_s, environment: test_run.python, test_run.platform (recorded, not compared))
  environment differs, not compared — test_run.platform: record says 'linux-x86_64', this tree gives 'macosx-26.0-arm64'

Three fields are masked and nothing else is. The rule is that a measured duration is not a measurement of the tree, and the list is an enumeration of where one appears rather than a pattern — a pattern would mask a future field nobody had looked at.

  • generated_at — two runs are never at one instant;
  • test_run.duration_s — wall time for the whole run, which differs by scheduling;
  • source_commit, only when either side is dirty, and verify says so in its own output. A dirty tree has no commit that describes it, so comparing the field would be comparing a label to a thing it does not name.

Two more fields are recorded and not compared, which is a different class from masked: test_run.python and test_run.platform describe the host the record was made on rather than the tree it measured. They stay in the record — a reader matching a record against a wheel tag needs both — and verify prints them where they differ, after its verdict and never affecting it. Two reproductions of one tree on two machines are the same evidence; until 2026-09-16 they were not, because a CI record verified from any other checkout differed on exactly these two fields and on the absolute fixtures path, none of which says anything about the tree. That is what makes a record produced by CI verifiable by a third party on their own machine.

There were four until 2026-09-08. The fourth was conformance.checks.H.details.refusals[].seconds — how long one malformed payload took to be refused — and it was found by running the reproducibility proof rather than by reasoning about it: two generations from one fresh clone differed in exactly one field. Masking it was the wrong repair, and the reason is worth keeping. The same value is written into conformance-<version>.json, whose SHA-256 the release pipeline records; two tests compare THOSE BYTES, unmasked, so the mask made verify reproducible while leaving the published artefact a function of machine load. On M's ruling of 2026-09-08 the field was removed from the report instead: check H still measures the elapsed time and still enforces §21's five-second bound, and what it publishes is over_time_bound, a boolean, beside the declared timeout_s. Deterministic output should not depend on a consumer knowing which fields to ignore. Since the audit remediation's F03 (2026-09-19) the bound is enforced by a kill rather than a comparison: checks H and N run every adversarial payload in a spawned worker process, a case unanswered within timeout_s is killed and recorded as PARSER_TIMEOUT (a FAIL, never a refusal), and the volatile record — pids, exit codes, durations — goes to synapse conformance run --diagnostics PATH, not into the report.

A test sweeps every leaf of a record for a key that names a duration and fails on any that is neither masked nor a declared bound, so a timing field added by a later round cannot quietly make verify flaky.

A difference anywhere else is a difference in what was measured, and it is printed as field: record says X, this tree gives Y.

The loss report — §34's six categories​

Every scalar in a source payload lands in exactly one of six categories. The categories are a partition, and the precedence is declaration-first: a declared transform outranks an observed presence, because a knots-to-metres conversion whose source figure happens to appear elsewhere in the output is not a value that survived verbatim.

CategoryWhat it means
PRESERVEDthe value is in the output, outside the parked subtrees
NORMALIZEDthe path is declared in the adapter's TRANSFORMS, with its reason
DERIVEDthe same map, with a reason beginning derived: — the value is reachable only through a declared derivation
RESIDUALthe value arrived under attributes, payload, source_extras or §28's residual container
UNSUPPORTEDthe path is named in a structured Limitation's unsupported_paths — §34's "explicit documented exception"
DROPPEDnone of the above. The value is gone and nobody wrote down that it would be

The report is in the suite's JSON under loss_report and in --format text as a six-line summary. DROPPED is exactly the harness's own check-D set minus the paths a limitation declares, and a test asserts that identity per adapter per fixture rather than leaving two answers to one question.

--strict makes a non-empty DROPPED exit non-zero. It is implied by --require D and can be turned off with --no-strict. It is a third reason to exit non-zero and not a fourth verdict: check D's verdict stays the harness's. The two differ in one live case — an adapter that declares a source path unsupported still fails check D, because the value really did vanish, while its DROPPED count is 0 because the exception was documented in a form a machine can read.

Declaring an exception​

limitations accepts a sentence or a structured object, and both stay valid:

limitations = [
    "no document in this tree states which edition this adapter targets",
    Limitation(id="no-vendor-block",
               summary="the vendor block is not mapped",
               unsupported_paths=["vendor"]),
]

Only unsupported_paths is read by the classifier. A prose limitation contributes nothing to it — deliberately, because a documented exception a machine cannot read is not one.

Fixture provenance — §33​

Every fixture directory a tracked test reads carries a PROVENANCE.json:

{
  "schema_id": "synapse.fixture-provenance/v1",
  "directory": "pntmap",
  "fixtures": [
    {
      "file": "jamming_gulf_of_riga.json",
      "synthetic": true,
      "classification": "PUBLIC",
      "operational_data": false,
      "personal_data": false,
      "origin": "…one sentence saying where this came from…",
      "standard_ref": "klv/spec/klv_pin.json"
    }
  ]
}

synapse evidence provenance enforces it, unconditionally and with no migration flag:

  • every covered directory has a record;
  • every file on disk is listed, and every listed file is on disk;
  • synthetic is true and classification is PUBLIC for every fixture in this repository — a false is a finding, not a value to accept;
  • an origin is a sentence, and a standard_ref names a pin record that exists.

The covered set is derived from the tree, never enumerated: every directory the harness selects fixtures from, every malformed/ set, and every auxiliary family a tracked test replays (egress/, local/, refusals/, framing/, imapb/). golden/ is not covered — a golden is an adapter's output over a fixture that is already covered — and neither is spec/, which holds pinned specification records with a gate of their own. fixtures/fft has no record because the harness selects nothing there; the absence is what says it is not a fixture family.

The record is excluded from fixture selection by name and not by extension, in the harness and in the suite alike: two of these directories ship nothing but .json payloads, so an extension rule would select the record and skip the fixtures.

Badges — §35​

synapse badges --from evidence/ --out evidence/badges/

Seven shields.io endpoint files per adapter (six until the audit's F07 added the seventh on 2026-09-19), and one repository-level set that takes the worst reading of each — a repository badge reporting its best adapter would be a badge that says nothing.

BadgeGrounded in
SC-CDM Compatibleconformance.result
Conformance Lnconformance.maturity_eligible
Roundtrip Verifiedconformance.checks.E.verdict
Deterministicconformance.checks.G.verdict
Lossless Verifiedconformance.checks.D.verdict, loss_report.counts.DROPPED and loss_report.ledger — yes only when the ledger ran with nothing LOST; heuristic (amber) when check D rests on value presence alone; no on a measured failure
Synthetic Fixturesfixture_provenance.fixtures and .synthetic
Independent Evidenceevidence_categories — the external categories that are PRESENT, by name (green), or none (amber); absent from the output when the field is absent from the record

No badge is hand-assigned, and no badge outlives its evidence. badges() takes one argument — the record — so there is nothing a caller can say to influence what it renders. And a badge whose grounding field is absent from the record is absent from the output: not red, which would be a claim that the thing was measured and failed, and not grey, which would be a claim that the tool is fine. A test removes each grounding field in turn and requires the corresponding badge to disappear.

Where the records live​

evidence/ is not tracked. A record names the commit it measured and hashes every fixture the run read, so a tracked record is either stale the moment it is committed or a mechanical diff on every fixture round.

CI generates the set on every run and uploads it as the artefact evidence-<sha>; the release pipeline attaches it to the GitHub Release. That attachment is the point at which evidence.available becomes true in an adapter's manifest — synapse evidence generate means evidence is producible, not that it is available — and every adapter shipped here declared available: false until the 2.1.2 Release carried the records; since then every manifest declares it true and names that attachment.

SHA-256 here is identification, not signing​

synapse_cdm/evidence.py is the only module in the package that may import hashlib, and the allowance is enforced by module name in tests/test_cdm_boundary.py. The digest answers "are these the same bytes?" — which fixture a run read, which artefact a release published. It carries no key, authenticates nobody and asserts nothing about who produced the bytes it names.

models.Integrity is still designed and unpopulated. Signing belongs to the ledger, which holds the keys and is audited, and cryptography, hmac, secrets, ssl, nacl and oqs remain forbidden in every module of this package — including in the one exempt module.