FitSD validation harness

You are orchestrating a run, not participating in one. PROTOCOL.md states the rules of a run; this tells you how to drive one end to end and leave behind a record a third party can review — and that fitsd-web can publish with no edit anywhere else.

Read PROTOCOL.md before you start. It is the authority; where this and it disagree, it wins.

What you are producing

A run: one scenario, at one organisation, conducted per the method, against one version of the standard. It lives at scenarios/<scenario>/runs/<id>/ and contains

RUN.md          machine-readable frontmatter + the prose record of what was tested
artifacts/      the in-world records the seats produced — a real team's paperwork
findings/       one out-of-character findings file per seat, plus the synthesis

The rule that outranks everything

A run produces evidence, never rulings. You do not change the standard. Not a typo, not a requirement, not a form — no matter how obvious the fix looks from inside a seat. README.md’s “Status of results” is explicit: what changes the standard is the maintainer’s decision through the normal errata/release stream. Your output is a findings register with normative candidates drafted for that decision.

If you find yourself editing the standard, stop: you have left the harness.

Before the run

1. Pick or write a scenario. Reuse scenarios/certhub/ or author a new one. A scenario must be wholly fictional and organisation-neutral — FitSD carries no organisation’s fingerprints, and that includes its test material. Never use a real employer, customer, incident or person, even lightly disguised.
   • The organisation (organisations/<id>.md) is the company and its people. It is version-neutral and reusable: many scenarios can be set there.
   • The scenario (scenarios/<id>/scenario.md) is the work pushed through the standard: casting, trigger, proposed solution, and the stage hooks the orchestrator feeds each stage.
   • README.md’s “Adding scenarios” says what a useful new scenario stresses. Keep the shape: real internal customers, at least one third-party dependency, and a succession event.
2. Fix the standard under test. Record the exact tag, branch or commit — never “latest”. The point of a run is that its findings are checkable against a specific text.
3. Create the run directory with RUN.md (frontmatter first — see the contract below) and empty artifacts/ and findings/.

Driving the stages

Work the stage table in PROTOCOL.md in order. For each stage:

• Start a fresh seat with only its brief from agents/, the scenario, findings-template.md, the standard at the recorded ref, and the artefacts its brief names as Reads:. Nothing else.
• Substitute the placeholders the briefs use: {STANDARD}, {SCENARIO}, {RUN}.
• Collect its artefact(s) into artifacts/ and its findings file into findings/.
• Within a stage, independent seats (S5a/b/c) may run in parallel; across stages, never.

Give each seat its own context. A single conversation playing every seat in turn defeats the exercise: the successor cannot “read the records as a new owner would” if it remembers writing them. One seat, one context, every time.

The firewall — the run’s central test

Later seats see artefacts only, never another seat’s reasoning or findings. Findings files are out-of-world and no brief may grant access to them, the retro included.

It is strictest at S7: the successor gets the artefact set, the scenario and the standard — nothing else. PROTOCOL.md is blunt about the stakes: if the successor’s brief leaks history, the run’s central result is void. If you break it, say so in RUN.md rather than quietly continuing; a disclosed break is evidence, a hidden one is a corrupt run.

After S8 — synthesis

Follow PROTOCOL.md’s synthesis steps: consolidate findings by theme counting independent seats per theme (convergence is the signal — a gap three seats hit unprompted outranks one strong opinion), classify every theme, and write findings/CONSOLIDATED.md plus findings/NORMATIVE-CANDIDATES.md.

Classification is the part most likely to go wrong. Reference-tier fix and normative candidate are proposals to the maintainer, not actions — see the rule above. Draft the precise shall, its owning capability, a candidate ID, and the ripple through cards, forms and alignment tables, so the decision is cheap to make.

Log worked-wells too. A register with no praise column is as suspect as one with no gaps.

Finishing: make the run contributable

Fill in RUN.md’s frontmatter — this is what lets a contributed run render on the website with no edit anywhere else:

---
scenario: certhub          # must match a scenarios/<id>/ directory
organisation: fernbrook    # must match an organisations/<id>.md
spec_version: "0.3"        # the version of the standard tested
spec_ref: v0.3.0           # the exact tag, branch or commit
seats: ai-simulation       # ai-simulation | human | mixed — disclose it honestly
date: 2026-08              # YYYY-MM
findings: 110              # how many findings the run logged
---

Then, before opening a PR:

bash scripts/check-run.sh scenarios/<scenario>/runs/<id>

It checks the frontmatter, that the scenario and organisation resolve, that artifacts/ and findings/ are non-empty, that every file has an H1 (the website uses it as the page’s nav label), and that no two files collapse to the same page slug. Fix what it reports; see CONTRIBUTING.md.

Honesty obligations

• Disclose the seats. seats: says whether people, models or both played the roles. An all-AI run shares blind spots — RUN.md’s “Notes for the next run” on the CertHub run says as much, and recommends a human control seat.
• Never fabricate evidence in-world. If a seat would not have something, the record says so. An honest gap is the standard working, and it is the finding.
• Stay in character in artefacts, out of character in findings. Artefacts read like a real team’s paperwork; findings read like a reviewer’s notes.