Rust workspace · finished architecture
The complete evaluation harness the course builds — all four stages. How the structs, enums, traits, and functions across four crates connect, from a scenario spec through generated vignettes and logged model responses to blinded coding and typed results.
It starts with a FamilySpec — a scenario family loaded from a TOML file: its title, the doctrine it cites, the menu of actions an operator may take, and a prompt template with holes to fill. The holes are filled by Params: the four axes that make one scenario differ from another — how confident you are in attribution (who caused the event), whether you have hours or days, whether the action is reversible, and whether more information was requested. all_params enumerates every combination — a 3×2×2×2 grid of 24 — and ScenarioFamily::is_valid discards the incoherent ones. For each surviving Params, generate renders the template into a concrete prompt and hashes it. The result is a Vignette: one fully-specified scenario instance — an id, its params, the exact prompt text, and a SHA-256 of that text so the same scenario always carries the same fingerprint. A vignette is the atom of the whole harness — a single decision to put to a model.
A vignette is only a question until something answers it. ModelClient is the trait every provider implements: give it a prompt, get back a ModelResponse (the text plus token Usage); AnthropicClient is the first concrete implementation, its base URL injectable so tests point it at a mock rather than the live API. dispatch is the loop that turns questions into data — every vignette, across every model, across every epoch (a repeat, to measure consistency) — producing one model call and one ResponseRecord: which vignette, which model, the params carried along for analysis, the prompt, the raw response, the usage, a timestamp. append_record writes each one to responses.jsonl, an append-only log that is never rewritten — the record of what the models actually said is sacred.
Raw responses have to be coded — graded against a codebook — and the grading has to be blind to be trustworthy. make_sheet turns the response records into BlankSheetRows that show a coder only the response text, stripped of any hint of which model produced it or under what parameters; the mapping back to that metadata lives in a separate KeyRow file the coder never sees. When the coded sheets return, load_coded parses each into a CodedRow — and this is the payoff of the type-first design: CodedRow's fields are enums (ActionType, StrategicLogic), so a value outside the codebook is not flagged, it fails to parse. The validation is not a step you run afterward; it is the parse. check_anchors enforces the last rule — a latent code must quote the text it is grounded in — and what remains is a clean, typed judgment that Python reads downstream for the statistics.
The full four-stage path, orchestrated by the unified panoptes CLI. A TOML spec and a 24-cell parameter grid become vignettes; the dispatch loop runs them across models and epochs into an append-only log; blinded sheets go out for coding and come back as typed CodedRows that Python reads downstream.
Composition (◆ owns-a), trait implementation (▷ dashed), and the enums that make invalid states unrepresentable. Params is the hub — it is the grid element, it rides inside every Vignette, and it is stamped onto every ResponseRecord. Two traits define the extension points: ScenarioFamily (new scenario) and ModelClient (new provider).
What each crate defines, and the one stage it owns. The three stage binaries (panoptes-gen, panoptes-run, panoptes-code) are unified behind the single panoptes command.
The data model — no I/O, no network. Every type that crosses a crate boundary lives here, defined once.
Stage 1 — generation. Spec + parameter grid → vignettes + manifest.
Stage 2 — execution. Dispatches prompts across models × epochs; logs every response.
Stage 3 — coding. Blinded sheets out; typed rows back (parse = validation).
Option<String>.