Introduction

This is a course, not a manual. By the end you will have built Panoptes — a typed, reproducible evaluation harness in Rust — and, more importantly, you will understand why every piece is shaped the way it is. We are going to move the way a good pair-programming session moves: I frame a concept, you ask questions until it is solid, then we build the piece together and the compiler grades the work.

What we are building

Panoptes is the execution engine behind an SDA decision benchmark. It does four things, and each maps to one arc of this course:

1. Generate parameterized scenario vignettes from templates and a parameter grid.
2. Execute those vignettes against version-pinned model APIs, logging every raw response.
3. Code the responses against a codebook, with human judgment recorded in types that reject invalid values.
4. Validate coder agreement (this last stage hands off to Python).

Stages 5 and 6 — statistical analysis and reporting — stay in Python and read the files this harness produces. We will not build them here; we will build the thing that produces their inputs.

The end state, held in view

Everything we build serves one payoff you should keep in mind from the very first chapter: a coder will be structurally unable to record a value that is not in the codebook. Not "discouraged from," not "warned against" — unable, because the value will fail to parse. That property is what makes this instrument defensible, and it falls out almost for free once you understand Rust's type system and the serde library. When we reach it in Part II, it should feel inevitable rather than clever.

How the arcs are sequenced

The order is chosen for understanding, not for delivery speed. Each arc teaches one core idea before the next arc depends on it:

• Part I, Foundations — the test loop, then ownership, borrowing, and moves. The mental model everything else assumes. New to Rust starts here.
• Part II, Data Model — serde, derive macros, enums-as-validation. Go slow here.
• Part III, Generation — traits, the config boundary, iterators. Modeling the problem in types.
• Part IV, Async — futures, await, tokio, mock-server testing. The genuinely hard arc; budget extra time.
• Part V, Coding — types for correctness properties you personally care about (blinding, parse-is-validation).
• Part VI, Hardening — workspace hygiene, lints, the language boundary.
• Part VII, CLI — unifying the binaries into one panoptes command with proper exit codes. Where it becomes a tool.

A note on where you are starting from

You come to this with real strength in one area and newness in another, and it helps to be honest about both. You have built LLM-powered systems and agents, so the domain of this harness — models, prompts, evaluation, the shape of the problem — is familiar ground. What is new is Rust itself, and specifically its ownership model and its async story. That is exactly why this course front-loads concepts and adds a dedicated foundations arc before we write types in anger: we are not going to assume you already think in ownership and borrowing. We are going to build that mental model deliberately, because everything downstream — and especially the async arc — rests on it.

If a chapter feels hard, that is usually not a signal about your ability; it is the compiler teaching you a rule you have not yet internalized. The whole method is designed around making those lessons fast and legible rather than frustrating.

Turn the page to see how the course is structured and how to run the build loop, then we start with Phase 0.

How to Use This Course

The two chapter types

Every arc alternates between two kinds of chapter, and they ask different things of you.

Concept chapters are the lecture. Read them without touching the keyboard. They end with a short list of questions you should be able to answer for yourself — if any are fuzzy, that is the signal to stop and ask before moving to the build. Do not proceed to a build chapter on a shaky concept; the whole point of frontloading concepts is that the build then feels obvious.

Build chapters are the pair-programming session. Here you write the code. Each one follows the same test-driven loop:

1. Write the failing test. You write it, from the behavior we described — not by copying an answer.
2. Predict the failure. Say out loud (or note down) what the compiler or test runner will report.
3. Run it and check your prediction. The gap between prediction and reality is the lesson.
4. Write the minimal implementation. Just enough to make the test pass. No more.
5. Run again, watch it pass.
6. Commit.

The concept-check quizzes

Each part ends with a graded quiz. These are not busywork — they target the exact misconceptions that cause bugs three chapters later. Answer honestly before revealing; a wrong answer with a good explanation teaches more than a lucky guess. The score is for you, not for anyone else.

The answer key

There is a companion implementation plan (linked in the appendix) that contains the full, worked version of every task. Use it as an answer key, not a script. Try each build yourself first. Check the plan after. The distance between your version and the plan's version is precisely where the learning lives — if they match, you understood it; if they differ, the difference is worth investigating.

Working setup

You will want two things open side by side: this book, and a terminal in your project directory. A split screen or two monitors. The loop is fast — write test, run, read error, fix — and it only works if the feedback is immediate.

When you get stuck

Bring the exact error message, verbatim. Rust's compiler errors are unusually good; most of the time the fix is in the error itself, and learning to read them is half of learning the language. When we work through a stuck point together, we dig into what the compiler is actually objecting to rather than papering over it.

Phase 0: The Toolchain and the Loop

Before any Panoptes code, we verify one thing: that you can write a failing test, see it fail legibly, fix it, and see it pass. The entire course runs on that loop being fast and trustworthy. This phase is deliberately trivial — that is the point. We are testing the machinery, not your ability.

Objective

• Confirm a working Rust toolchain (rustc, cargo).
• Create a throwaway crate.
• Write one deliberately failing test, watch it fail, fix it, watch it pass.
• Internalize what cargo test output looks like when things break.

Install the toolchain

Rust is installed through rustup, which manages compiler versions:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# accept the defaults, then:
source "$HOME/.cargo/env"
rustc --version
cargo --version

You want a recent stable toolchain. If you already have one, rustup update brings it current.

Create a throwaway crate

cargo new hello-loop
cd hello-loop

cargo new scaffolds a tiny package: a Cargo.toml manifest and a src/main.rs. Open src/main.rs. Replace its contents with a single function and a test:

fn add(a: i32, b: i32) -> i32 {
    a + b
}

fn main() {
    println!("{}", add(2, 2));
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn two_plus_two_is_five() {
        assert_eq!(add(2, 2), 5); // deliberately wrong
    }
}

The failing run

cargo test

You will see something close to this:

running 1 test
test tests::two_plus_two_is_five ... FAILED

failures:

---- tests::two_plus_two_is_five stdout ----
thread 'tests::two_plus_two_is_five' panicked at src/main.rs:15:9:
assertion `left == right` failed
  left: 4
 right: 5

Read that carefully, because you will read a hundred of these. It tells you the exact test, the exact file and line, and — critically — left: 4, right: 5, the actual value against the expected value. assert_eq! always reports both sides. This is why we write assertions with the computed value on the left and the expected on the right: the output reads naturally.

The fix

Change the 5 to a 4:

#![allow(unused)]
fn main() {
        assert_eq!(add(2, 2), 4);
}

Run again:

test tests::two_plus_two_is_five ... ok

test result: ok. 1 passed; 0 failed; 0 ignored

That is the loop. Every build chapter is this, with more interesting types in the middle.

What just happened, named

• #[cfg(test)] means "only compile this module when running tests" — your test code is not in your shipped binary.
• mod tests { } is a module, a namespace. Tests conventionally live in a nested tests module.
• use super::*; pulls in everything from the parent module so the test can see add.
• #[test] marks a function as a test case for cargo test to discover and run.

Every one of these appears in the very first Panoptes task, so you have now seen the whole skeleton.

Done when

You can run cargo test, read a failure, and fix it without thinking about the mechanics. When that loop is automatic, turn the page: Part II begins with the concept that makes the whole harness worth building in Rust.

Concept: Ownership and Moves

Kind: Concept (read, do not code). New to Rust: this is the chapter everything else assumes.

Rust's one genuinely unfamiliar idea, the thing that has no direct equivalent in Python or most languages you have used, is ownership. Every other Rust concept in this course is ordinary once ownership is solid. So we go slow here, and we build a mental model you can actually reason with, not a rule you memorize.

The one rule

Here is the whole thing in a sentence, from Rust for Rustaceans: Rust's memory model centers on the idea that all values have a single owner — exactly one location, usually a scope, is responsible for ultimately deallocating each value.

Read that twice. Every value has exactly one owner. When the owner goes away (its scope ends), the value is cleaned up — "dropped" — automatically. No garbage collector deciding later, no manual free. The owner's scope ending is the cleanup signal.

What "move" means

Now the consequence that trips up everyone new to Rust. What happens when you assign a value to a new variable, or pass it to a function? For most types, the value moves. The book again: if the value is moved — by assigning it to a new variable, pushing it to a vector, or placing it on the heap — ownership moves from the old location to the new one, and you can no longer access the value through variables that flow from the original owner, even though the bits are technically still there.

This is the shock. In Python, b = a gives you two names for the same object. In Rust, for an owned type, let b = a; moves the value into b, and a is now unusable. Not copied — moved. The compiler will reject any later use of a with a clear error. Ownership transferred.

#![allow(unused)]
fn main() {
let s1 = String::from("hello");
let s2 = s1;          // the String moves from s1 into s2
// println!("{}", s1); // COMPILE ERROR: s1's value was moved into s2
println!("{}", s2);    // fine — s2 owns it now
}

The escape hatch: Copy

Some small types do not move — they copy. The book calls them "rebels": if a value's type implements the special Copy trait, the value is not considered to have moved even if reassigned; instead it is copied, and both locations remain accessible. Integers, floats, booleans, and small types made only of those are Copy. That is why this works fine:

#![allow(unused)]
fn main() {
let x = 42;
let y = x;             // i32 is Copy — x is COPIED into y
println!("{} {}", x, y); // both fine, both hold 42
}

The rule for what can be Copy is precise and worth internalizing: to be Copy, it must be possible to duplicate the value simply by copying its bits — which excludes any type that owns a resource it must deallocate when dropped. A String owns heap memory, so it can never be Copy. An i32 is just bits, so it is. This is exactly why, back in Part II, the small Params struct — made only of a u8, two simple enums, and a bool — can derive Copy, while a Vignette containing a String cannot.

The flows mental model

The single most useful way to think about all of this, and the model the borrow checker actually uses internally, is flows. Picture each value as having a line — a flow — that starts when the value is created and traces through your program to its last use. A move ends one flow and starts another. Using a variable after its flow has ended (because the value moved away) is the error the compiler catches. Hold this picture; in the next chapter it makes borrowing click immediately.

Questions to lock

Genuinely stop on each. This is the foundation.

1. In let b = a; where a is a String, what happens to a? Why? What would happen instead if a were an i32?
2. Why can a type that owns heap memory (like String) never be Copy?
3. What is the connection between "every value has one owner" and "Rust needs no garbage collector"?

Next chapter: borrowing — how you let code use a value without taking ownership of it, which is what most function calls actually do.

Concept: Borrowing and References

Kind: Concept. Builds directly on the previous chapter — read that first.

If every value has one owner and moving transfers it, how does a function use your value without stealing it? The answer is borrowing, and it is what the overwhelming majority of Rust function signatures actually do.

Lending without giving up ownership

From Rust for Rustaceans: Rust allows the owner of a value to lend out that value to others, without giving up ownership, through references. References are pointers that come with an additional contract for how they can be used.

A reference, written &value, lets code look at (or modify) a value it does not own. When the reference goes away, nothing is dropped — the reference never owned the value, so there is nothing to clean up. The owner keeps ownership the whole time. This is why you will see & everywhere: passing &params to a function lets it read your Params without moving it, so you can keep using params afterward.

Contrast this with the previous chapter, where passing an owned String moved it. Borrow instead, and the value stays yours:

fn describe(vignettes: &[String]) -> String {
    format!("{} vignettes ready", vignettes.len())
}

fn main() {
    let vignettes = vec!["ca_geo-030-H-REV-INFO".to_string(), "ca_geo-060-D-IRREV-NOINFO".to_string()];
    let msg = describe(&vignettes);          // lend it out...
    println!("{msg}");                        // 2 vignettes ready
    println!("still mine: {} entries", vignettes.len()); // ...and it is still ours
}

If describe took vignettes: Vec<String> instead, the second println! would be a compile error — value used after move. The & is the difference between lending and giving.

Two kinds of reference, one iron rule

There are exactly two kinds, and the distinction is the whole game.

Shared references (&T) — read-only, and you can have as many as you like at once. The book: a shared reference is a pointer that may be shared; any number of other references may exist to the same value, and values behind shared references are not mutable. Many readers, no writers.

fn main() {
    let prompt = String::from("You are the duty officer.");
    let a = &prompt;
    let b = &prompt;
    println!("{a} | {b} | {prompt}"); // three readers at once — no conflict
}

Mutable references (&mut T) — read-write, but exclusive. While a &mut exists, nothing else may touch the value. The book: the compiler assumes that the mutable reference is exclusive — no other reference, shared or mutable, may coexist with it.

Put together, this is the rule the borrow checker enforces everywhere: either any number of shared (&) references, or exactly one mutable (&mut) reference — never both at once. Many readers or one writer. Not both.

Watch the rule fire

Here is the violation, in the smallest form you will actually meet — holding a reference into a Vec while pushing to it (a push may reallocate and move every element, which would leave first pointing at freed memory; the rule exists precisely to forbid this):

fn main() {
    let mut log = vec!["line 1".to_string()];
    let first = &log[0];          // shared borrow begins
    log.push("line 2".into());    // mutable borrow while shared is alive: ERROR
    println!("{first}");          // shared borrow still in use here
}

error[E0502]: cannot borrow `log` as mutable because it is also borrowed as immutable

The fix is usually not clone() — it is reordering, so the shared borrow's flow ends before the mutable one begins:

fn main() {
    let mut log = vec!["line 1".to_string()];
    let first = &log[0];
    println!("{first}");          // last use — the shared borrow's flow ends here
    log.push("line 2".into());    // exclusive access is now fine
    println!("log has {} lines", log.len());
}

Same statements, different order, compiles clean. The compiler tracks flows by last use, not by scope braces — once first is used for the last time, its borrow is over.

Back to the flows model

Remember flows from the last chapter — each value's line from creation to last use. Borrowing adds flows too: a shared borrow starts a flow that must not overlap a mutable one. The borrow checker's job, in the book's words, is to check that there cannot be two parallel flows with mutable access to a value, nor a flow that borrows a value while there is no flow that owns the value. When you see "cannot borrow as mutable because it is also borrowed as immutable," that is two flows illegally overlapping. The fix is almost always to let one flow end (stop using the first reference) before the other begins.

Where this lands in Panoptes

You do not need to fight the borrow checker much in this harness — the code is mostly straightforward — but you will read & and &mut constantly and need to know why each is there:

• ModelClient::generate(&self, prompt: &str) takes a shared reference to the client and a shared reference to the prompt string: it reads both, owns neither, so the caller keeps them.
• The append-only log writer opens a file and writes — the exclusivity of &mut on the file handle is what makes "one writer at a time" a compile-time fact.
• Passing &vignettes to the dispatch loop lets it read every vignette without moving the vector, so it is still yours afterward.

When an async error in Part IV mentions lifetimes or borrows, this is the model to reach for: which flow is this, and does it overlap another?

Questions to lock

1. What does passing &params to a function let the function do, and what can you still do with params afterward?
2. State the borrow rule in one sentence. Why does forbidding "shared and mutable at once" prevent data races?
3. In the flows model, what illegal situation does the error "cannot borrow as mutable because it is also borrowed as immutable" describe?

That is the foundation. With ownership, moves, and borrowing solid, the rest of the course is mostly applying them. Part II begins — the data model, and the serde concept that makes the whole harness worth building in Rust.

Concept: serde and the derive Macro

This is the data-model arc, and everything in it exists to earn one payoff. This chapter builds the foundation for that payoff: understanding what serde is, what a derive macro actually does, and why generating conversion code at compile time changes when your bugs surface.

The problem serde solves

The harness lives or dies on moving structured data across boundaries. A scenario becomes a prompt file. A model response becomes a line in a log. A coder's judgment becomes a row in a sheet. Every one of those is the same underlying operation: a Rust value in memory turning into text on disk, or text on disk turning back into a Rust value.

In Python you would reach for json.dumps and json.loads and mostly not think about it, because Python does not care what shape the thing is until it breaks at runtime. If a field is missing or the wrong type, you find out when the code hits that line — often in production, often at the worst time.

serde is Rust's answer. The name is just "serialize / deserialize." It is a library, not a language feature, but it is so universal it might as well be built in.

The part that looks like magic

Here is the thing that feels like magic at first, and that is worth demystifying now so it does not stay magic: you almost never write the conversion code yourself. You annotate a struct like this —

use serde::{Deserialize, Serialize};

#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Params {
    attribution_confidence: u8,
    info_request: bool,
}

fn main() {
    let p = Params { attribution_confidence: 60, info_request: true };

    let json = serde_json::to_string(&p).unwrap();
    println!("{json}"); // {"attribution_confidence":60,"info_request":true}

    let back: Params = serde_json::from_str(&json).unwrap();
    assert_eq!(p, back);
    println!("round-tripped: {back:?}");
}

— and serde generates the conversion for you, tailored to that exact type. Run it: field names become JSON keys, types map to JSON values, and the trip back reproduces an equal struct — without you writing a line of conversion code. The code that turns a Params into JSON is written, by the macro, before your program ever starts. There is no reflection, no runtime inspection, no dictionary of fields being walked while your program runs, the way Python does it.

What "derive" actually is

This is the piece most people treat as incantation. Let us not.

A derive macro is a code generator that runs during compilation. When you write #[derive(Serialize)] above a struct, you are telling the compiler: look at the fields of this struct, and write the serialization function for me. The macro reads your struct's shape — its field names and their types — and emits real Rust source code: an implementation of the Serialize trait. That generated code then gets compiled right alongside everything you wrote by hand.

You have already met this mechanism without thinking about it. #[derive(Debug)], which lets you print a value with {:?} for inspection, is the same thing: it generates the code that formats your type for debugging. #[derive(Clone)] generates the code to duplicate it. serde's macros are more elaborate, but mechanically identical: read the type, emit the code.

Why this matters: bugs move to compile time

Here is the consequence that makes serde the right tool for a reproducibility-critical instrument. Because the generation happens at compile time against your specific type, the compiler can see the whole thing. If a field's type cannot be serialized, you find out when you build — not when a log write fails at two in the morning during your evaluation run. Watch it happen:

use serde::Serialize;

#[derive(Serialize)]
struct Bad {
    name: String,
    handle: std::fs::File, // what would serializing an open file even mean?
}

fn main() {}

error[E0277]: the trait bound `File: serde::Serialize` is not satisfied

The program never existed. In Python, the equivalent (json.dumps on an object holding a file handle) is a TypeError — at runtime, on the code path that happened to hit it.

This is the first appearance of a theme that runs through the entire course:

Why this arc is the foundation

Hold the end state in view, because it is why we start here. In a few chapters, your codebook criteria become Rust enums — StrategicLogic with variants Control, Maritime, Political, and so on. The reason that works — the reason a coder literally cannot record a value outside the codebook — is that serde's deserialization of an enum will reject any string that does not match a variant.

That rejection is generated by the same derive mechanism we just discussed, applied to an enum instead of a struct. So the "parse is validation" property that makes this whole harness worth building in Rust is not a separate trick you will learn later. It is this concept — serde derive — pointed at an enum. Get this arc solid and that payoff is almost free. Rush it and the payoff will feel like luck.

Questions to lock before the build

Genuinely stop and make sure each of these is crisp. If one is fuzzy, that is the signal to re-read or ask.

1. Why does generating conversion code at compile time, rather than inspecting the value at runtime like Python does, let the compiler catch a whole class of bugs before the program runs?
2. What is a derive macro actually doing to your struct, mechanically? (If your answer is "it adds serialization," push further: how?)
3. Can you already see, at least in outline, why an enum plus serde would refuse an out-of-codebook value — even though we have not written that code yet?

Next chapter is the first build: we create the workspace and the parameter types, and you write the first failing test.

Build: Workspace + Parameter Types

Maps to: Task 1 in the task plan. Kind: Build (you write the code).

Objective

Stand up the Cargo workspace and create panoptes-core, the crate that will hold every type crossing a boundary. Define the scenario parameter enums (TimePressure, Reversibility) and the Params struct, and prove with tests that they round-trip through strings and JSON.

Scaffold

Create (this is a new project directory, separate from this book's repo):

• Cargo.toml — the workspace manifest. Declare every shared dependency for the whole course under [workspace.dependencies] now (full annotated list in the Workspace Scaffold appendix); later crates then just write dep = { workspace = true }.
• crates/panoptes-core/Cargo.toml — with [dependencies]: serde, serde_with, strum, chrono (all { workspace = true }).
• crates/panoptes-core/src/lib.rs — module declarations + re-exports.
• crates/panoptes-core/src/params.rs — the types and their tests.

Dependencies this chapter actually exercises: serde (derives) and strum (Display, EnumString). Add serde_json under [dev-dependencies] for the JSON round-trip test.

Expected result: cargo test -p panoptes-core params → 2 tests pass (time_pressure_string_roundtrip, params_json_roundtrip).

The spec (design decisions — givens, not puzzles)

• TimePressure has variants Hours, Days; Reversibility has Reversible, Irreversible.
• Both enums carry #[strum(serialize_all = "UPPERCASE")] — the string form is "HOURS", "IRREVERSIBLE". This convention is a design decision (it becomes the manifest CSV's wire format), not something to infer.
• Derives on both enums: Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Display, EnumString.
• Params has exactly four fields: attribution_confidence: u8, time_pressure: TimePressure, reversibility: Reversibility, info_request: bool. Derives: Debug, Clone, Copy, PartialEq, Serialize, Deserialize.

Full code: Answer Key, Task 1.

Concepts exercised

• Cargo workspaces and [workspace.dependencies] for shared version pinning.
• #[derive(...)] stacking: Serialize, Deserialize, Debug, Clone, Copy, PartialEq.
• strum's Display + EnumString for enum↔string conversion.

What "round-trip" means, concretely

Encode a value, decode the result, and assert you end up with exactly the value you started with — while also pinning what the encoded form looks like in between. A round-trip test proves the codec is lossless and freezes the wire format, so a refactor that silently changes "HOURS" to "Hours" fails a test instead of corrupting every downstream join. These strings become the manifest CSV and the JSONL log — the file contract.

The two tests exercise two independent codecs, and they encode the same enum differently:

The build loop (you drive)

1. Write the failing test for TimePressure string round-tripping (Hours ↔ "HOURS") and for Params JSON round-tripping. Derive them from the behavior described, not from the answer key.
2. Predict the failure — will it fail to compile or fail at runtime? (Hint: the type does not exist yet. What does the compiler say about that?)
3. Run, check the prediction.
4. Implement the enums and struct with the minimal derives to pass.
5. Run green, then commit.

Done when

cargo test -p panoptes-core params shows two passing tests, and you can explain why Copy is safe to derive on these types (all fields are themselves Copy).

Check yourself

Compare against Task 1 in the appendix task plan. If your derives differ, work out whether the difference matters — some are load-bearing (Deserialize), some are ergonomic (Copy).

Concept: Enums as Validation

Kind: Concept (read, do not code).

This is the payoff chapter the whole arc has been building toward. The idea is small and the consequence is large.

The core idea

A Rust enum is a closed set of named alternatives. StrategicLogic is Control or Maritime or Political or Procedural or None or Mixed — and nothing else can exist. There is no seventh value. The compiler enforces this everywhere the type is used.

Now combine that with what you learned about serde: when serde deserializes a string into an enum, it matches the string against the known variants. A string that matches no variant is a deserialization error. Not a warning. Not a silently-accepted fallback. An error that stops the parse.

Here is the whole idea, runnable — the stringly-typed world and the typed world, side by side:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
enum StrategicLogic { Control, Maritime, Political, Procedural, None, Mixed }

fn main() {
    // Stringly-typed: anything parses, garbage included.
    let loose: String = serde_json::from_str("\"AGGRESSIVE\"").unwrap();
    println!("String accepts: {loose:?}");

    // Typed: the codebook IS the type.
    let ok: Result<StrategicLogic, _> = serde_json::from_str("\"CONTROL\"");
    println!("enum + CONTROL:    {ok:?}");

    let bad: Result<StrategicLogic, _> = serde_json::from_str("\"AGGRESSIVE\"");
    println!("enum + AGGRESSIVE: {}", bad.unwrap_err());
}

String accepts: "AGGRESSIVE"
enum + CONTROL:    Ok(Control)
enum + AGGRESSIVE: unknown variant `AGGRESSIVE`, expected one of `CONTROL`, `MARITIME`, `POLITICAL`, `PROCEDURAL`, `NONE`, `MIXED` at line 1 column 12

Read that error closely: it names the offending value, lists the entire legal codebook, and gives the position in the input. That is your validation report, generated for free by the derive — and note the #[serde(rename_all = "SCREAMING_SNAKE_CASE")] attribute, which is what makes serde's wire format match the codebook's "CONTROL" convention (the build chapter's spec pairs it with the equivalent strum attribute for Display).

Why this is the instrument's spine

In the Python sketch of this harness, validating a coded value meant a separate lint.py script that checked each value against an allowed list — a step you had to remember to run, that ran after the data already existed in a possibly-invalid state.

In Rust, the allowed list is the type. A coded row whose c2_logic field is "AGGRESSIVE" does not become an invalid CodedRow that you later catch — it fails to become a CodedRow at all. The validation is not a step in the pipeline; it is a property of the boundary. Parsing the file is validating it.

Connecting back to the thesis

Recall why rubric validity is the primary intellectual risk: if the codebook reads as one person's opinion, the thesis fails at defense. The enum does not make the criteria defensible — that is the doctrinal grounding and the inter-rater reliability. But it does guarantee that the recorded data conforms exactly to the codebook you defined, with zero drift, which is one less thing a committee can poke. The instrument's categories are frozen in the type.

Questions to lock

1. What is the difference between "validate the value after parsing" and "parse in a way that rejects invalid values"? Why does the second eliminate a class of bugs the first cannot?
2. If StrategicLogic has six variants, what happens — mechanically — when serde tries to deserialize "CONTROL"? What happens with "AGGRESSIVE"?
3. Why does putting the codebook's allowed values in an enum mean no downstream code ever needs to re-check them?

Next: we turn StrategicLogic and friends into real code, and write the test that proves an invalid value is rejected.

Build: The Codebook Types

Maps to: Task 2. Kind: Build.

Objective

Create codes.rs in panoptes-core: the ActionType and StrategicLogic enums and the CodedRow struct. Write the test that is the whole reason we chose Rust — an invalid codebook value must fail to deserialize.

Scaffold

Create: crates/panoptes-core/src/codes.rs. Modify: crates/panoptes-core/src/lib.rs (declare and re-export the module).

Dependencies: no manifest changes — strum (for SCREAMING_SNAKE_CASE serialization) and serde_with (for NoneAsEmptyString on coder_notes) are already in core's Cargo.toml from the previous chapter.

Expected result: cargo test -p panoptes-core codes → 3 tests pass (valid_logic_parses, invalid_logic_is_rejected, coded_row_json_roundtrip).

The spec (givens)

• ActionType variants: SensorRetask, Maneuver, Monitor, EscalateToCommand, RequestData, NoAction. StrategicLogic variants: Control, Maritime, Political, Procedural, None, Mixed.
• Both carry two attributes: #[strum(serialize_all = "SCREAMING_SNAKE_CASE")] and #[serde(rename_all = "SCREAMING_SNAKE_CASE")] — wire form "SENSOR_RETASK", "CONTROL" in both codecs — plus the same derive stack as the params enums. Unlike params.rs (where the strum and serde codecs never cross paths), the coded files are hand-filled with codebook strings and parsed by serde — so here the two codecs must agree, which means saying it to each codec separately. Omit the serde attribute and serde_json::from_str::<StrategicLogic>("\"CONTROL\"") fails with unknown variant CONTROL, expected one of Control, ….
• CodedRow fields, in order: response_id: String, c1_action: ActionType, c2_logic: StrategicLogic, c2_anchor_quote: String, c3_escalation: u8, codebook_version: String, coder_notes: Option<String> — the last annotated #[serde_as(as = "NoneAsEmptyString")] under a #[serde_as] struct attribute.

Full code: Answer Key, Task 2.

Concepts exercised

• #[strum(serialize_all = "SCREAMING_SNAKE_CASE")] to match codebook string conventions.
• serde_with::NoneAsEmptyString for optional free-text fields.
• The negative test: asserting that a parse fails.

The build loop (you drive)

1. Write three failing tests: a valid value parses; an invalid value ("AGGRESSIVE") is rejected; a full CodedRow round-trips through JSON.
2. Predict: the invalid_logic_is_rejected test asserts result.is_err(). Before implementing, what makes it currently fail — the assertion, or the fact that the type does not compile yet?
3. Run, check.
4. Implement the enums and struct.
5. Run green, commit.

Done when

cargo test -p panoptes-core codes is green, and the invalid-value test proves the codebook constraint lives in the type rather than in a separate check.

Build: Records, Vignettes, the File Contract

Maps to: Task 3. Kind: Build.

Objective

Create vignette.rs and record.rs in panoptes-core. Define Vignette, the deterministic vignette_id function, Usage, and ResponseRecord — the struct that becomes one line in responses.jsonl, the dataset of record. These types are the file contract: the interface every later stage and the Python analysis tail depend on.

Scaffold

Create: crates/panoptes-core/src/vignette.rs and crates/panoptes-core/src/record.rs. Modify: crates/panoptes-core/src/lib.rs (re-export both).

Dependencies: no manifest changes — chrono (with its serde feature, for DateTime<Utc>) is already declared.

Expected result: cargo test -p panoptes-core → 7 tests pass across the crate (2 params + 3 codes + 1 vignette + 1 record). The shared data model is complete.

The spec (givens)

• Vignette fields: id: String, family: String, params: Params, prompt: String, prompt_sha256: String.
• vignette_id(family: &str, p: &Params) -> String produces family-<conf:03>-<H|D>-<REV|IRREV>-<INFO|NOINFO> — e.g. ca_geo-030-H-REV-INFO (confidence zero-padded to 3 digits).
• Usage fields: input_tokens: u32, output_tokens: u32.
• ResponseRecord fields, in order: response_id: String, vignette_id: String, model: String, epoch: u32, params: Params, prompt: String, response: String, usage: Usage, run_at: chrono::DateTime<Utc>.

Full code: Answer Key, Task 3.

Concepts exercised

• chrono::DateTime<Utc> with serde support for timestamps.
• Deterministic ID construction from typed fields (no stringly-typed field access).
• Why the JSONL schema is the contract that makes the underlying tool fungible.

The build loop (you drive)

1. Write failing tests: vignette_id produces the exact formatted string (ca_geo-030-H-REV-INFO); ResponseRecord round-trips through a JSON line.
2. Predict what vignette_id returns for a given Params, character by character, before running.
3. Run, check.
4. Implement.
5. Run green, commit.

Done when

All of cargo test -p panoptes-core passes (params + codes + vignette + record). You now have the complete shared data model — every type the other three crates will import.

Concept-Check: Data Model

You have built the shared data model. Before moving to generation, confirm the ideas that the next arcs assume.

Concept: Traits and the Config Boundary

Kind: Concept.

Traits, briefly

A trait is a set of behaviors a type promises to provide — Rust's version of an interface. ScenarioFamily will require any family type to answer name() and is_valid(&Params). Code can then work with any family through the trait, without knowing which concrete family it is.

You have used traits already (Serialize, Display are traits). Here you define one for the first time in this project, which is a small but real step up.

Here is the complete mechanism on a toy domain (the same coffee shop from the iterators chapter): a trait, two types implementing it differently, and one generic function that works with both — and with any implementation written later:

#[derive(Debug, Clone, Copy)]
enum Size { Small, Large }
#[derive(Debug, Clone, Copy)]
enum Roast { Light, Dark }

struct Order { size: Size, roast: Roast, iced: bool }

/// The behavior contract: any house can say what it is called
/// and which orders it will actually make.
trait House {
    fn name(&self) -> &str;
    fn allows(&self, order: &Order) -> bool;
}

struct Downtown;
impl House for Downtown {
    fn name(&self) -> &str { "downtown" }
    fn allows(&self, o: &Order) -> bool {
        !(matches!(o.size, Size::Small) && o.iced) // house rule: no small iced drinks
    }
}

struct Airport;
impl House for Airport {
    fn name(&self) -> &str { "airport" }
    fn allows(&self, o: &Order) -> bool {
        !matches!(o.roast, Roast::Light) // house rule: dark roast only
    }
}

/// Generic over the trait: works with ANY house, present or future.
fn count_allowed(house: &impl House, orders: &[Order]) -> usize {
    orders.iter().filter(|o| house.allows(o)).count()
}

fn main() {
    let orders = [
        Order { size: Size::Small, roast: Roast::Light, iced: true },
        Order { size: Size::Large, roast: Roast::Light, iced: false },
        Order { size: Size::Large, roast: Roast::Dark, iced: true },
    ];
    println!("{}: {} of {} allowed", Downtown.name(), count_allowed(&Downtown, &orders), orders.len());
    println!("{}: {} of {} allowed", Airport.name(), count_allowed(&Airport, &orders), orders.len());
    // downtown: 2 of 3 allowed
    // airport: 1 of 3 allowed
}

The mapping is one-for-one: House ↔ ScenarioFamily, allows ↔ is_valid, count_allowed(&impl House, …) ↔ generate(family: &impl ScenarioFamily, …). Each house's rule is ordinary Rust in its impl — typed, testable, compiled — which is exactly where the next section says validity logic belongs. Adding a second scenario family later is adding an impl, not touching generate.

The design decision that matters

The consequential idea in this chapter is not the trait mechanics — it is where the validity logic lives. In the Python sketch, families carried a list of string "valid_rules" that were evaluated at runtime. That means: logic expressed as data, interpreted while the program runs, with errors surfacing only when a bad rule is hit.

We are making the opposite choice. TOML holds content and metadata only — the prompt template, the doctrine references, the action menu. The logic of which parameter combinations are meaningful lives in Rust code, in each family's is_valid implementation. That means the compiler checks it, and a malformed rule is a compile error, not a runtime surprise.

Questions to lock

1. What does a trait let you do that you could not do by writing a function per concrete type?
2. Why is "which parameter combinations are valid" logic, and therefore code, rather than data that belongs in TOML?
3. What do you lose by moving that logic out of config, and why is the trade worth it for this project specifically?

Build: Family Spec + Validity Trait

Maps to: Task 4. Kind: Build.

Objective

Create the panoptes-gen crate. Define FamilySpec (the TOML shape, loaded with the toml crate), the ScenarioFamily trait, and a first concrete family CaGeo whose is_valid encodes a real exclusion rule. Write the example ca_geo.toml.

Why TOML for the spec: it is the dialect you already speak (Cargo.toml), the toml crate is what Cargo itself builds on, and TOML has no implicit typing — a level named "NO" can never silently become false the way it can in YAML. For an instrument whose configs are the experimental conditions, unambiguous scalars beat terseness.

Scaffold

Create (a whole new crate — remember to list it in the workspace members):

• crates/panoptes-gen/Cargo.toml — [dependencies]: panoptes-core = { path = "../panoptes-core" }, plus serde, toml, tera, sha2, itertools, clap, anyhow (all { workspace = true }); [dev-dependencies]: serde_json.
• crates/panoptes-gen/src/lib.rs, src/family.rs (the FamilySpec + from_toml), src/validity.rs (the ScenarioFamily trait + CaGeo).
• scenarios/families/ca_geo.toml — the first family spec (content + metadata only).

Dependencies this chapter exercises: toml (spec parsing). tera/sha2/itertools sit unused until the next two chapters — declaring them now just saves manifest edits later.

Expected result: cargo test -p panoptes-gen family and cargo test -p panoptes-gen validity → 5 tests pass between them.

The spec (givens)

• FamilySpec fields: name: String, version: u32, title: String, doctrine_refs: Vec<String>, action_menu: Vec<String>, template: String. Derives: Debug, Clone, Deserialize. Constructor: from_toml(s: &str) -> anyhow::Result<Self>.
• The ScenarioFamily trait has exactly two methods: fn name(&self) -> &str and fn is_valid(&self, p: &Params) -> bool.
• CaGeo's exclusion rule: a combination is invalid when time_pressure is Hours and info_request is false (no time to act and no way to gather data).
• The example ca_geo.toml content (family name, doctrine refs, action menu, template text) is data, not an exercise — copy it from the Answer Key.

Full code: Answer Key, Task 4.

Concepts exercised

• toml deserialization into a struct (same derive, different format).
• Defining and implementing a trait.
• Keeping content (TOML) and logic (Rust) on opposite sides of the boundary.

The build loop (you drive)

1. Write failing tests: FamilySpec::from_toml parses the fields; CaGeo::is_valid rejects the excluded combo and accepts valid ones.
2. Predict: the TOML parse test — what happens if a required field is missing from the TOML? Compile error or runtime Err? Why?
3. Run, check.
4. Implement.
5. Run green, commit.

Done when

cargo test -p panoptes-gen family and cargo test -p panoptes-gen validity pass. You can articulate why the same #[derive(Deserialize)] works for TOML and JSON alike (serde is format-agnostic; the derive describes the shape, the format crate handles the encoding).

Concept: Iterators and the Cartesian Product

Kind: Concept. New crate: itertools — this chapter shows it working before you build with it.

The idea

Generating vignettes means taking every combination of parameter values — 3 confidences × 2 time pressures × 2 reversibilities × 2 info-request states — and producing one vignette per valid combination. That "every combination" is a cartesian product, and Rust's iterator ecosystem expresses it cleanly: product → filter → map → collect, as a single chain.

That one sentence assumes a lot if iterators are new to you, so this chapter builds up to it with runnable code. Every example below runs — click the play button, or paste it into a scratch project.

First, iterators themselves

An iterator is any value implementing the Iterator trait, which has one essential method: next(), returning Some(item) until the sequence is exhausted, then None. A for loop is sugar over exactly that. What makes iterators powerful is the adapters — methods like map and filter that wrap an iterator and return a new one — and the consumers like collect that drive the chain and produce a final value.

fn main() {
    let confidences = [30u8, 60, 95];

    let doubled: Vec<u8> = confidences.iter().map(|c| c * 2).collect();
    println!("{doubled:?}"); // [60, 120, 190]

    let high: Vec<u8> = confidences.iter().copied().filter(|&c| c >= 60).collect();
    println!("{high:?}"); // [60, 95]
}

Two things worth pausing on, because they will show up in your build:

• .iter() yields references (&u8), because iterating must not consume the array. .copied() converts &u8 → u8 for Copy types, which keeps the closures clean. Your parameter enums derive Copy for exactly this kind of ergonomic win.
• The closure |&c| c >= 60 uses a pattern in its argument: &c matches a reference and binds the value. You will meet this again with tuples below.

Laziness, demonstrated

Adapters do no work when you attach them. The chain is a description of a computation; a consumer like collect() is what actually pulls values through it. Watch the print order:

fn main() {
    let chain = [1, 2, 3].iter().map(|n| {
        println!("computing {n}");
        n * 10
    });
    println!("chain built — nothing has printed yet");

    let out: Vec<i32> = chain.collect();
    println!("{out:?}");
}

chain built — nothing has printed yet
computing 1
computing 2
computing 3
[10, 20, 30]

The map closure runs during collect, not before. For generation this means the 24-combination grid never exists as a whole until you collect it — and if you only wanted the first valid vignette, .find(...) would stop the whole chain early, computing nothing else.

What we are replacing: nested loops

Here is a cartesian product over a deliberately toy domain — coffee orders, so it cannot be mistaken for the answer key — written the obvious way:

#[derive(Debug, Clone, Copy)]
enum Size { Small, Large }
#[derive(Debug, Clone, Copy)]
enum Roast { Light, Dark }

fn main() {
    let mut combos = Vec::new();
    for size in [Size::Small, Size::Large] {
        for roast in [Roast::Light, Roast::Dark] {
            for iced in [true, false] {
                combos.push((size, roast, iced));
            }
        }
    }
    println!("{} combos, first = {:?}", combos.len(), combos[0]);
    // 8 combos, first = (Small, Light, true)
}

This works. Its weaknesses are structural: the intent ("every combination") is implicit in the indentation; the whole product is built eagerly into a Vec whether or not you need it all; and anything you do to each combination — filtering, transforming — has to happen inside the loop body, where it cannot be tested as a separate piece.

itertools and iproduct!

itertools is a widely-used community crate that extends the standard iterator vocabulary with extra adapters and macros — it is to Iterator roughly what serde_with is to serde. It is already declared in panoptes-gen's manifest. The piece you need is the iproduct! macro: give it N iterables and it yields tuples of every combination, lazily.

use itertools::iproduct;

#[derive(Debug, Clone, Copy)]
enum Size { Small, Large }
#[derive(Debug, Clone, Copy)]
enum Roast { Light, Dark }

fn main() {
    let combos: Vec<(Size, Roast, bool)> = iproduct!(
        [Size::Small, Size::Large],
        [Roast::Light, Roast::Dark],
        [true, false]
    ).collect();

    println!("{} combos, first = {:?}", combos.len(), combos[0]);
    // 8 combos, first = (Small, Light, true)
}

Same eight combinations, same order (rightmost dimension varies fastest, like an odometer) — but now the product is an iterator: a value you can pass around, chain adapters onto, and consume once at the end.

The full shape: product → filter → map → collect

Now the pattern the build chapter asks of you, complete on the toy domain. One combination is nonsense for this shop — no small iced drinks — and each surviving combination becomes a typed struct:

use itertools::iproduct;

#[derive(Debug, Clone, Copy)]
enum Size { Small, Large }
#[derive(Debug, Clone, Copy)]
enum Roast { Light, Dark }

#[derive(Debug)]
struct Order { size: Size, roast: Roast, iced: bool }

fn valid(size: Size, iced: bool) -> bool {
    // house rule: no small iced drinks
    !(matches!(size, Size::Small) && iced)
}

fn main() {
    let orders: Vec<Order> = iproduct!(
        [Size::Small, Size::Large],
        [Roast::Light, Roast::Dark],
        [true, false]
    )
    .filter(|&(size, _roast, iced)| valid(size, iced))
    .map(|(size, roast, iced)| Order { size, roast, iced })
    .collect();

    println!("{} valid of 8 raw", orders.len()); // 6 valid of 8 raw
    println!("first = {:?}", orders[0]);
    // first = Order { size: Small, roast: Light, iced: false }
}

This maps one-for-one onto what you will write: the arrays become CONFIDENCES × the two parameter enums × the info-request bools (3 × 2 × 2 × 2 = 24); valid becomes the family's is_valid (CaGeo's rule cuts 24 to 18); Order becomes Vignette (with template rendering and hashing inside the map). The domain is different; the shape is identical.

When the map can fail: collecting Results

In the real generate(), the map step renders a template — which can fail — so the closure returns anyhow::Result<Vignette>, not Vignette. That would seem to leave you with an awkward Vec<Result<…>>, but collect() has a trick: it can target Result<Vec<T>, E> directly, succeeding only if every element succeeded and short-circuiting on the first error.

fn main() {
    let ok: Result<Vec<i32>, String> = ["1", "2", "3"].iter()
        .map(|s| s.parse::<i32>().map_err(|e| e.to_string()))
        .collect();
    println!("{ok:?}"); // Ok([1, 2, 3])

    let bad: Result<Vec<i32>, String> = ["1", "oops", "3"].iter()
        .map(|s| s.parse::<i32>().map_err(|e| e.to_string()))
        .collect();
    println!("{bad:?}"); // Err("invalid digit found in string")
}

This is why generate() can return anyhow::Result<Vec<Vignette>> from a single collect() call — one bad template render fails the whole generation loudly, instead of producing a partial scenario set silently.

Why iterators, not loops

Now that you have seen both versions: the chain is lazy (nothing computes until consumed), composable (the validity predicate is a free function you can unit-test without running any product), and legible — "the cartesian product, minus invalid combos, each turned into a vignette" reads directly off the code, while the loop version buries that shape in indentation.

Questions to lock

1. What is a cartesian product, and why is it the right description of "all parameter combinations"?
2. What does "lazy" mean for an iterator chain, and why is collect() the moment the work actually happens? (Point to the line of output that proves it.)
3. In the coffee example, why does filter's closure take |&(size, _roast, iced)| but map's take |(size, roast, iced)|?
4. Why is a filter → map chain easier to test than the equivalent nested loops? (Where does valid live in each version?)
5. What type does .collect::<Result<Vec<_>, _>>() produce when one element fails, and why is that the right behavior for scenario generation?

Build: Vignette Generation

Maps to: Task 5. Kind: Build.

Objective

Write generate.rs: the all_params() iterator over the parameter grid, and generate() which filters by validity, renders each prompt through tera, hashes it, and produces Vignettes. Prove the counts and the uniqueness of IDs with tests.

Scaffold

Create: crates/panoptes-gen/src/generate.rs. Modify: crates/panoptes-gen/src/lib.rs (re-export all_params, generate).

Dependencies: no manifest changes — itertools (iproduct!), tera, and sha2 were declared when you created the crate.

Expected result: cargo test -p panoptes-gen generate → 4 tests pass (grid_has_24_raw_combinations, validity_filter_reduces_count → 18, ids_are_unique, template_renders_params).

The spec (givens)

• The confidence levels are const CONFIDENCES: [u8; 3] = [30, 60, 95]; — a design decision from the study, not derivable.
• all_params() -> impl Iterator<Item = Params>: the cartesian product of confidences × both TimePressure variants × both Reversibility variants × [true, false] for info_request (3 × 2 × 2 × 2 = 24).
• generate(family: &impl ScenarioFamily, spec: &FamilySpec) -> anyhow::Result<Vec<Vignette>>.
• The tera context gets exactly four keys: attribution_confidence (number), time_pressure (its Display string), info_request (bool), action_menu (the vec joined with ", ").
• prompt_sha256 is the lowercase-hex SHA-256 of the rendered prompt bytes.

Full code: Answer Key, Task 5.

Concepts exercised

• itertools::iproduct! over typed enum arrays.
• tera templating: injecting parameters into the prompt template.
• sha2 hashing for the prompt fingerprint.

The build loop (you drive)

1. Write failing tests: the raw grid has exactly 24 combinations; the validity filter reduces it to the expected number; all generated IDs are unique; a rendered prompt contains the injected parameter.
2. Predict: how many vignettes remain after CaGeo's filter removes the HOURS + NOINFO combos? Work it out from the grid dimensions before running.
3. Run, check the arithmetic against reality.
4. Implement.
5. Run green, commit.

Done when

cargo test -p panoptes-gen generate passes, including the uniqueness guard.

Concept: clap and Writing Files

Kind: Concept. New crate: clap — plus the std::fs calls the build chapter needs. This chapter shows both working before you build with them.

The idea

The next build turns your generation library into a program: a binary someone runs from a shell, pointing at a family spec, that leaves a manifest and prompt files on disk. That takes two skills the course has not taught yet — parsing command-line arguments and writing files — and this chapter covers exactly the slice of each you need.

clap: your CLI is a struct

clap's derive style inverts how you might expect argument parsing to work. You do not write parsing code and extract values from it — you declare a struct that is your binary's interface, and #[derive(Parser)] generates the parser from its shape at compile time. Fields become flags, field types become argument types, doc comments become help text:

use clap::Parser;
use std::path::PathBuf;

/// Generate vignettes from a family spec.
#[derive(Parser)]
#[command(name = "panoptes-gen")]
struct Cli {
    /// Path to the family TOML
    #[arg(long)]
    family: PathBuf,
    /// Output directory
    #[arg(long, default_value = "scenarios/generated")]
    out: PathBuf,
}

fn main() {
    // parse_from simulates: panoptes-gen --family scenarios/families/ca_geo.toml
    let cli = Cli::parse_from(["panoptes-gen", "--family", "scenarios/families/ca_geo.toml"]);
    println!("family = {}", cli.family.display());
    println!("out    = {} (defaulted)", cli.out.display());
}

family = scenarios/families/ca_geo.toml
out    = scenarios/generated (defaulted)

Reading the attributes: #[arg(long)] makes a --family <value> flag; no default means it is required. default_value makes --out optional. The fields are PathBuf — a typed, owned filesystem path — so by the time your main body runs, the arguments already have the right types. (In the real binary you call Cli::parse(), which reads the actual command line; parse_from is the same machinery fed from code, handy in examples and tests.)

This is the same move the whole course keeps making: the interface is a type, and the compiler holds it. It is also serde's move — declare the shape, derive the machinery — pointed at argv instead of JSON.

What the derive gives you for free

Run your compiled binary with --help and clap has already written the manual, straight from your doc comments:

Generate vignettes from a family spec

Usage: panoptes-gen [OPTIONS] --family <FAMILY>

Options:
      --family <FAMILY>  Path to the family TOML
      --out <OUT>        Output directory [default: scenarios/generated]
  -h, --help             Print help

Forget a required flag and you get a real error, a usage reminder, and — note, ahead of Part VII — a non-zero exit code (clap uses 2):

error: the following required arguments were not provided:
  --family <FAMILY>

Usage: panoptes-gen --family <FAMILY>

For more information, try '--help'.

You wrote none of that. The struct declaration bought all of it.

Writing files: three functions

Everything the build chapter does on disk is three std::fs calls — create_dir_all (make a directory, parents included, fine if it already exists), fs::write (create-or-truncate a file with the given bytes), and read_to_string (the whole file back as a String). Here they are producing exactly the build chapter's artifact — a manifest CSV built in memory, written, and read back:

use std::fs;

struct Row { id: String, confidence: u8 }

fn main() -> std::io::Result<()> {
    let rows = vec![
        Row { id: "ca_geo-030-H-REV-INFO".into(), confidence: 30 },
        Row { id: "ca_geo-060-D-IRREV-NOINFO".into(), confidence: 60 },
    ];

    // 1. Build the CSV in memory: header first, then one line per row.
    let mut csv = String::from("vignette_id,attribution_confidence\n");
    for r in &rows {
        csv.push_str(&format!("{},{}\n", r.id, r.confidence));
    }

    // 2. Write it. Create the directory first; fs::write creates or truncates the file.
    let dir = std::env::temp_dir().join("panoptes-demo");
    fs::create_dir_all(&dir)?;
    let path = dir.join("manifest.csv");
    fs::write(&path, &csv)?;

    // 3. Read it back to prove the round trip.
    let back = fs::read_to_string(&path)?;
    print!("{back}");
    println!("({} lines: 1 header + {} rows)", back.lines().count(), rows.len());
    Ok(())
}

vignette_id,attribution_confidence
ca_geo-030-H-REV-INFO,30
ca_geo-060-D-IRREV-NOINFO,60
(3 lines: 1 header + 2 rows)

That is the entire write_manifest pattern: a String accumulated with push_str, one format! line per record, one fs::write at the end. Note main returning std::io::Result<()> so the ? operator can propagate any I/O failure — and, per the previous chapters' theme, a failed write is then an error, not a silent absence. Prompt files are even simpler: one fs::write(dir.join(format!("{id}.txt")), &prompt) per vignette.

Mapping onto the build

The build chapter's Cli struct is the first example with the spec's names on it; write_manifest is the CSV example with the full seven-column header; the binary's last line prints the summary to stderr with eprintln! — stdout is kept clean, a habit that pays off when tools are chained in Part VII.

Questions to lock

1. What does #[derive(Parser)] generate, and from what information? Where does the --help text come from?
2. What happens — message and exit code — when a required argument is missing, and who wrote that behavior?
3. Why is hand-built CSV acceptable for the manifest but not for the prompt text, and what is the signal to switch to the csv crate?
4. Why does the file-writing example's main return std::io::Result<()>, and what does ? do with a failed write?

Build: The Generation CLI + Manifest

Maps to: Task 6. Kind: Build.

Objective

Add the panoptes-gen binary: a clap CLI that loads a family TOML, generates vignettes, writes each prompt to prompts/, and writes manifest.csv — the join spine every later stage reads. Test the manifest writer, then run the binary end to end.

Everything new here — the clap derive, PathBuf arguments, create_dir_all/fs::write, and the hand-built CSV pattern — was introduced with worked examples in the previous chapter. If any line of this build feels unfamiliar, that is the page to reread; this chapter should only be assembly.

Scaffold

Create: crates/panoptes-gen/src/main.rs. Modify: crates/panoptes-gen/Cargo.toml — add a [[bin]] name = "panoptes-gen" path = "src/main.rs" section and tempfile = "3" under [dev-dependencies] (the manifest test writes to a temp dir).

Expected result: cargo test -p panoptes-gen manifest → 1 test passes; then cargo run -p panoptes-gen -- --family scenarios/families/ca_geo.toml prints generated 18 vignettes → scenarios/generated and leaves manifest.csv plus 18 files under scenarios/generated/prompts/.

The spec (givens)

• CLI args: --family <PathBuf> (required) and --out <PathBuf> defaulting to "scenarios/generated".
• The manifest header is exactly vignette_id,family,attribution_confidence,time_pressure,reversibility,info_request,prompt_sha256, then one row per vignette using each enum's Display string. No prompt body in the manifest.
• Each prompt is written to <out>/prompts/<vignette_id>.txt; the closing message goes to stderr: generated {n} vignettes → {out}.

Full code: Answer Key, Task 6.

Concepts exercised

• clap derive for command-line argument parsing.
• Filesystem writing and directory creation.
• The manifest as the single place parameters live for the join.

The build loop (you drive)

1. Write a failing test for write_manifest: header present, one row per vignette, IDs appear.
2. Predict the line count of the manifest for N vignettes (remember the header).
3. Run, check.
4. Implement, then run the actual binary: cargo run -p panoptes-gen -- --family scenarios/families/ca_geo.toml.
5. Verify the output files exist, commit.

Done when

The binary emits manifest.csv plus one prompt file per vignette, and you can point to the manifest as the reason Stage 5 analysis will be a simple join rather than forensic reconstruction.

Concept-Check: Generation

Concept: async, await, and the Runtime

Kind: Concept. This is the hard arc — read it twice.

The problem async solves

Your harness makes hundreds of API calls. Each one spends almost all its time waiting — for the network, for the model to generate. If you made them one at a time, synchronously, the program would sit idle during every wait. Async lets a single thread start a call, and while it waits, do other useful work (like starting the next call).

What async and await actually mean

An async fn does not run when you call it. It returns a future — a value representing a computation that is not finished yet. The future does nothing until it is driven. You drive it with .await, which means "pause here until this future is ready, and let other work proceed meanwhile."

This is the mental shift: in synchronous code, calling a function runs it. In async code, calling an async fn gives you a description of work; .await is what actually advances it. Prove it with print order:

async fn fetch(label: &str) -> String {
    println!("  fetch({label}) is actually running");
    format!("{label}: done")
}

#[tokio::main]
async fn main() {
    let future = fetch("alpha");
    println!("future created — nothing has run yet");

    let result = future.await;
    println!("{result}");
}

future created — nothing has run yet
  fetch(alpha) is actually running
alpha: done

The body of fetch runs after the "future created" line — calling the function did not run it; .await did. (This is the same laziness you saw with iterators: adapters describe, collect drives. Futures describe, .await drives.)

The runtime

Futures need something to drive them — a scheduler that polls them, parks the ones that are waiting, and wakes them when their I/O is ready. That scheduler is the async runtime. Rust does not ship one in the standard library; the ecosystem standard is tokio. The #[tokio::main] attribute on main sets up the runtime so your top-level async code has something to run on.

The lifecycle, precisely

It is worth having the exact lifecycle in mind rather than a vague sense of "it runs later." Async Rust describes a future's life this way: when a future is created, it is idle — it has yet to be executed. Once executed, it can yield a value, resolve, or go to sleep because it is pending, and each subsequent poll returns either Pending or Ready until the future is resolved or cancelled. That polling loop — idle, then polled repeatedly, each poll answering "ready yet?" — is what the runtime is doing under every .await. You will rarely implement poll by hand, but knowing that .await is sugar over "poll this until it is Ready" demystifies the whole model.

The same book makes the concurrency payoff concrete with a kitchen analogy: several tasks each spending time waiting can overlap, because the executor sets a task to idle when it hits an await and works on the next task in the queue while polling the idle ones. That single sentence is why your harness benefits from async at all — while one API call waits on the network, another can be in flight.

You can measure the payoff directly. Two 200ms "API calls," awaited one after the other versus driven concurrently with tokio::join!:

use std::time::{Duration, Instant};
use tokio::time::sleep;

async fn api_call(label: &str, ms: u64) -> String {
    sleep(Duration::from_millis(ms)).await; // stand-in for waiting on the network
    format!("{label} answered")
}

#[tokio::main]
async fn main() {
    let t = Instant::now();
    let a = api_call("first", 200).await;
    let b = api_call("second", 200).await;
    println!("sequential: {a} / {b} in ~{}ms", t.elapsed().as_millis());

    let t = Instant::now();
    let (a, b) = tokio::join!(api_call("first", 200), api_call("second", 200));
    println!("concurrent: {a} / {b} in ~{}ms", t.elapsed().as_millis());
}

sequential: first answered / second answered in ~404ms
concurrent: first answered / second answered in ~202ms

One thread, two waits overlapped, half the wall-clock. The dispatch loop you build in this arc awaits calls sequentially first — correctness before concurrency — but this is the capacity the async foundation gives you when the epoch counts grow.

Why this arc is genuinely harder

Async stacks several new things at once: futures, await, the runtime, and — next chapter — traits that contain async methods, plus the Send + Sync bounds that make futures safe to move between threads. Each is a real concept. Expect the compiler to be least forgiving here, and expect the errors to be longer.

The good news is that this arc rests directly on the ownership foundation from Part I. Most async borrow-checker fights are ownership and lifetime problems in disguise — a value moved into a future, a reference that does not live long enough — which is precisely the flows-and-moves model you already built. When an async error looks intimidating, the first move is to read it as an ownership question: what owns this value, and how long does this borrow need to live? The answer is usually there.

Questions to lock

1. What does an async fn return when you call it, and why does calling it not run the work?
2. What does .await do, and why does it let one thread stay busy during I/O waits?
3. What is the runtime's job, and why does #[tokio::main] exist?

Concept: Async Traits, Send + Sync

Kind: Concept.

Why this needs its own chapter

You want a ModelClient trait with an async fn generate. But plain Rust traits could not, for a long time, contain async methods directly, and even now the ergonomic path for a trait used as Box<dyn ModelClient> is the async_trait macro. It rewrites your async trait methods into a form that works with dynamic dispatch. You annotate the trait and its impls with #[async_trait] and write async methods as if it just worked.

Here is the whole pattern, working — a trait with an async method, two implementations, and a Vec<Box<dyn …>> calling them uniformly:

use async_trait::async_trait;

#[async_trait]
trait Oracle: Send + Sync {
    fn name(&self) -> &str;
    async fn answer(&self, question: &str) -> String;
}

struct Cheerful;
#[async_trait]
impl Oracle for Cheerful {
    fn name(&self) -> &str { "cheerful" }
    async fn answer(&self, q: &str) -> String { format!("{q}? Absolutely!") }
}

struct Grumpy;
#[async_trait]
impl Oracle for Grumpy {
    fn name(&self) -> &str { "grumpy" }
    async fn answer(&self, q: &str) -> String { format!("{q}? No.") }
}

#[tokio::main]
async fn main() {
    let oracles: Vec<Box<dyn Oracle>> = vec![Box::new(Cheerful), Box::new(Grumpy)];
    for o in &oracles {
        println!("{}: {}", o.name(), o.answer("Will it compile").await);
    }
}

cheerful: Will it compile? Absolutely!
grumpy: Will it compile? No.

Substitute Oracle → ModelClient, answer → generate, and the two oracles → Anthropic and whatever provider comes second, and this is the dispatch loop's skeleton: heterogeneous clients in one Vec, one uniform async call. Note the three pieces you must not forget, because each produces a different confusing error when missing: #[async_trait] on the trait and on every impl, and the Send + Sync supertrait bound — which exists for the reason the next section explains.

Send + Sync, briefly

Because the runtime may move futures between threads, the things inside them must be safe to send across threads. Two marker traits express this: Send (safe to move to another thread) and Sync (safe to share by reference across threads). When you write Box<dyn ModelClient>, you will often need ModelClient: Send + Sync so the boxed clients can be used by the multi-threaded runtime.

You will not usually implement these — they are automatic for most types. But you will require them in bounds, and the compiler will tell you when a bound is missing, sometimes with an error that points several layers away from the real cause. Learning to read those is part of this arc.

Questions to lock

1. Why do we reach for the async_trait macro instead of just writing async fn in the trait?
2. What do Send and Sync guarantee, and why does the multi-threaded runtime need them for boxed clients?
3. What does Box<dyn ModelClient> buy us that a concrete type would not, for the dispatch loop?

Build: ModelClient + the Append-Only Log

Maps to: Task 7. Kind: Build.

Objective

Create the panoptes-harness crate. Define the ModelClient trait (with #[async_trait]) and ModelResponse. Write the append-only JSONL writer and the test that guards its most important property: a second append must not overwrite the first.

Scaffold

Create (new crate — add it to workspace members):

• crates/panoptes-harness/Cargo.toml — [dependencies]: panoptes-core = { path = "../panoptes-core" }, plus serde, serde_json, reqwest, tokio, async-trait, chrono, sha2, clap, anyhow (all { workspace = true }); [dev-dependencies]: wiremock = "0.6", tempfile = "3". (sha2 is for the dispatch chapter's response-id hashing — the original task plan omits it there, so declare it now.)
• crates/panoptes-harness/src/lib.rs, src/client.rs (the trait), src/jsonl.rs (the writer).

Dependencies this chapter exercises: async-trait, serde_json (one record per line), tempfile (dev).

Expected result: cargo test -p panoptes-harness jsonl → 1 test passes (appends_without_truncating).

The spec (givens)

• ModelResponse fields: text: String, usage: Usage (from core). Derives: Debug, Clone.
• The trait: #[async_trait] pub trait ModelClient: Send + Sync with exactly two methods — fn model_name(&self) -> &str (the pinned model string) and async fn generate(&self, prompt: &str) -> anyhow::Result<ModelResponse> (single-turn; the prompt is the entire input).
• append_record(path: &Path, rec: &ResponseRecord) -> anyhow::Result<()> — opens with OpenOptions::new().create(true).append(true), writes one serde_json line.

Full code: Answer Key, Task 7.

Concepts exercised

• Defining an async trait with async_trait.
• std::fs::OpenOptions with .append(true) for append-only semantics.
• Testing filesystem behavior with tempfile.

The build loop (you drive)

1. Write the failing test appends_without_truncating: write two records, assert the file has two lines and each is valid JSON.
2. Predict: what would the test show if you opened the file with .create(true).write(true) instead of .append(true)? (This is the bug the test exists to catch.)
3. Run, check.
4. Implement the writer.
5. Run green, commit.

Done when

cargo test -p panoptes-harness jsonl passes and the append-only property is proven.

Concept: Testing Against a Mock Server

Kind: Concept. New crate: wiremock — this chapter shows it working before you build with it.

The idea

You need to test that your Anthropic client correctly builds a request and parses the response. You do not want those tests to hit the real API — that costs money, needs a key, is slow, and is non-deterministic. The solution is a mock server: a fake HTTP server, started inside the test, that returns a canned response you control.

The crucial thing to internalize — and the first quiz question — is that wiremock is not a mocked object or a stubbed function. It starts a real HTTP server on a random local port. Your client makes a genuine network request to it; the server just happens to be under your control. The client cannot tell the difference, so the entire request-building and response-parsing path is exercised for real.

These examples use wiremock, which is not available on the Rust playground, so there is no play button. To run them, make a scratch crate with anyhow, reqwest (features ["json"]), serde/serde_json, tokio (features ["full"]) as dependencies and wiremock = "0.6" under [dev-dependencies] — the same set the harness crate declares.

A mock server is a real server

Start one, teach it one behavior, and hit it with an ordinary HTTP client:

use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. A real HTTP server starts on a random local port.
    let server = MockServer::start().await;
    println!("mock listening at {}", server.uri());

    // 2. Teach it exactly one behavior.
    Mock::given(method("POST")).and(path("/ping"))
        .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "ok": true })))
        .mount(&server).await;

    // 3. Any HTTP client can hit it — it is a socket, not a fake object.
    let body: serde_json::Value = reqwest::Client::new()
        .post(format!("{}/ping", server.uri()))
        .send().await?
        .json().await?;
    println!("matched:   {body}");

    // 4. A request no mock matches gets 404 — silently.
    let miss = reqwest::Client::new()
        .post(format!("{}/pong", server.uri()))
        .send().await?;
    println!("unmatched: {}", miss.status());
    Ok(())
}

mock listening at http://127.0.0.1:52866
matched:   {"ok":true}
unmatched: 404 Not Found

The vocabulary, line by line:

• MockServer::start().await — binds a fresh server to a random free port. Every test gets its own; tests can run in parallel without colliding.
• Mock::given(matcher).and(matcher)… — a chain of matchers describing which incoming requests this mock applies to: HTTP method, path, headers, even body contents.
• .respond_with(ResponseTemplate::new(200).set_body_json(…)) — what to send back when the matchers match: status code plus a canned JSON body you write out by hand.
• .mount(&server).await — registers the mock on the server. Until mounted, it does nothing.
• server.uri() — the mock's base URL (http://127.0.0.1:<port>). This is the value you hand to your client instead of the real API's URL.

And note line 4: a request that matches no mounted mock is answered with a plain 404. Keep that in mind — it is the classic wiremock trap, and we will come back to it.

#[tokio::test]

HTTP is async, so these tests are async fn — but the plain #[test] attribute cannot run an async function. #[tokio::test] replaces it: it spins up a tokio runtime for that one test and runs your async body to completion, exactly like #[tokio::main] does for a binary. This is why the harness crate's manifest needs tokio even though the dispatch binary is the only "real" async entry point.

The full pattern: a client with an injectable base URL

Here is the complete shape the build chapter asks of you, on a toy service so it cannot be mistaken for the answer key: a fortune-cookie API. POST {base}/v1/fortunes with an x-api-key header and a JSON body; the response nests the payload two levels deep, and the client parses it into a clean public type:

use serde::Deserialize;

pub struct FortuneClient {
    pub api_key: String,
    pub base_url: String, // injectable so tests can point at a mock
    http: reqwest::Client,
}

impl FortuneClient {
    pub fn new(api_key: String, base_url: String) -> Self {
        Self { api_key, base_url, http: reqwest::Client::new() }
    }

    pub async fn tell(&self, topic: &str) -> anyhow::Result<Fortune> {
        let raw: RawResp = self.http
            .post(format!("{}/v1/fortunes", self.base_url))
            .header("x-api-key", &self.api_key)
            .json(&serde_json::json!({ "topic": topic }))
            .send().await?
            .error_for_status()?
            .json().await?;
        Ok(Fortune { text: raw.fortune.text, credits_used: raw.credits.used })
    }
}

/// What callers get: flat, typed, ours.
#[derive(Debug, PartialEq)]
pub struct Fortune {
    pub text: String,
    pub credits_used: u32,
}

/// What the wire carries: nested, shaped by someone else's API.
#[derive(Deserialize)]
struct RawFortune { text: String }
#[derive(Deserialize)]
struct RawCredits { used: u32 } // "remaining" exists in the JSON; serde ignores undeclared fields
#[derive(Deserialize)]
struct RawResp { fortune: RawFortune, credits: RawCredits }

And the test — a mock that both responds and asserts:

#[cfg(test)]
mod tests {
    use super::*;
    use wiremock::matchers::{header, method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    #[tokio::test]
    async fn parses_a_mocked_fortune() {
        let server = MockServer::start().await;
        Mock::given(method("POST"))
            .and(path("/v1/fortunes"))
            .and(header("x-api-key", "test-key"))
            .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
                "fortune": { "text": "You will refactor fearlessly." },
                "credits": { "used": 1, "remaining": 41 }
            })))
            .expect(1)
            .mount(&server)
            .await;

        let client = FortuneClient::new("test-key".into(), server.uri());
        let f = client.tell("rust").await.unwrap();
        assert_eq!(f.text, "You will refactor fearlessly.");
        assert_eq!(f.credits_used, 1);
    }
}

This runs green — one real HTTP round trip, zero network, zero spend. The mapping onto your build is one-for-one: FortuneClient ↔ AnthropicClient, /v1/fortunes ↔ /v1/messages, the x-api-key header appears in both, tell() ↔ generate(), the Raw* structs ↔ RawResp/RawBlock/RawUsage, and Fortune ↔ ModelResponse. The domain is different; the shape is identical.

Notice what the matcher chain is doing in the test: header("x-api-key", "test-key") means the mock only matches if your client actually sent that header. The mock is not just a canned response — it is also an assertion about the request your code built. If you forget the header in tell(), this test fails.

One more detail worth noticing: RawCredits declares only used, while the JSON also carries remaining. serde ignores fields you did not declare — which is exactly why your RawResp for the real API can model just content and usage and skip the dozen other fields the response carries.

Why this is the right design pressure

To make any of this work, your client must accept its base URL as a parameter rather than hard-coding https://api.anthropic.com — that is the base_url field and the server.uri() handoff above. That injectability is good design independent of testing: it is also how you would point at a proxy or a rented-inference gateway. The test forces a seam that turns out to be useful in production. Testable code and flexible code are frequently the same code.

Questions to lock

1. Why is hitting the real API in a unit test a bad idea, on at least three counts?
2. What does wiremock actually start, and why can the client not tell it is talking to a mock?
3. In the fortune test, what happens if tell() forgets to set the x-api-key header — which line of the test catches it, and what status does the client see?
4. What does .expect(1) verify, and when does that verification run?
5. Why does mock-testing force the client to take its base URL as a parameter, and why is that good beyond testing?

Build: The Anthropic Client

Maps to: Task 8. Kind: Build.

Objective

Implement AnthropicClient — a concrete ModelClient that builds the request, sends it, and parses the completion into a ModelResponse. Test it entirely against a wiremock mock server, spending no API credits.

Scaffold

Create: crates/panoptes-harness/src/anthropic.rs. Modify: crates/panoptes-harness/src/lib.rs (re-export AnthropicClient).

Dependencies: no manifest changes — reqwest and the dev-only wiremock were declared when you created the crate. The test needs #[tokio::test], which tokio's full features already cover.

Expected result: cargo test -p panoptes-harness anthropic → 1 test passes (parses_a_mocked_completion).

The spec (givens)

• AnthropicClient fields: api_key, model, base_url (all public String — base_url injectable so tests can point at the mock) plus a private http: reqwest::Client. Constructor new(api_key, model, base_url).
• Request: POST {base_url}/v1/messages with headers x-api-key: <key> and anthropic-version: 2023-06-01; JSON body {"model": …, "max_tokens": 1024, "messages": [{"role": "user", "content": <prompt>}]}.
• Response shape to deserialize: {"content": [{"text": …}, …], "usage": {"input_tokens": …, "output_tokens": …}} — concatenate the block texts in order.

Full code: Answer Key, Task 8.

Concepts exercised

• reqwest JSON POST with headers.
• Deserializing a nested API response into typed structs.
• wiremock mock definitions and #[tokio::test].

The build loop (you drive)

1. Write the failing test parses_a_mocked_completion: mount a mock returning a known completion + usage, call generate, assert the parsed text and token counts.
2. Predict: the response JSON has a content array of blocks. What does your parsing do if the array is empty? Decide the behavior before implementing.
3. Run, check.
4. Implement the client with an injectable base_url.
5. Run green, commit.

Done when

cargo test -p panoptes-harness anthropic passes. Adding a second provider (OpenAI, a local gateway) is now "copy this file, change the request/response shapes" — the trait makes providers uniform.

Build: The Dispatch Loop + Run Binary

Maps to: Task 9. Kind: Build.

Objective

Write dispatch.rs: the opaque response_id function and the dispatch loop over vignettes × clients × epochs, writing one ResponseRecord per call. Then the panoptes-run binary that loads the manifest, builds the pinned client list, and runs the loop. Test the ID's opacity and the loop's record count against a mock.

Scaffold

Create: crates/panoptes-harness/src/dispatch.rs and src/main.rs. Modify: crates/panoptes-harness/Cargo.toml (add [[bin]] name = "panoptes-run" path = "src/main.rs") and src/lib.rs (re-export dispatch, response_id).

Dependencies: sha2 for the response-id hash — already in the manifest if you followed the earlier Scaffold note (the original task plan forgets it). chrono supplies Utc::now(); #[tokio::main] runs the binary.

Expected result: cargo test -p panoptes-harness → 4 tests pass crate-wide; dispatch_writes_one_record_per_call proves 2 vignettes × 1 client × 3 epochs → 6 records.

The spec (givens)

• response_id(vignette_id: &str, model: &str, epoch: u32) -> String: the first 16 hex chars of SHA-256 over the vignette id bytes, then the model bytes, then epoch.to_le_bytes().
• dispatch(vignettes: &[Vignette], clients: &[Box<dyn ModelClient>], epochs: u32, log_path: &Path) -> anyhow::Result<usize> — loop order vignette → client → epoch, one append_record per call, returns the count.
• Binary CLI args: --scenarios (default "scenarios/generated"), --log (default "harness/logs/responses.jsonl"), --epochs (default 5). Real runs read ANTHROPIC_API_KEY and optional ANTHROPIC_BASE_URL from the environment.

Full code: Answer Key, Task 9.

Concepts exercised

• Iterating over &[Box<dyn ModelClient>] and calling an async trait method.
• Deterministic-but-opaque ID construction (the blinding property, in the harness).
• An integration test that drives the whole loop against a mock server.

The build loop (you drive)

1. Write failing tests: response_id_is_deterministic_and_opaque (same inputs → same id; epoch changes it; model name and params do NOT appear in it); dispatch_writes_one_record_per_call (2 vignettes × 1 client × 3 epochs → 6 records).
2. Predict: why must response_id be a hash rather than a readable concatenation? What property would a readable id break?
3. Run, check.
4. Implement dispatch and the binary.
5. Run green, commit.

Done when

cargo test -p panoptes-harness is fully green, including the opacity guard and the integration test.

Concept-Check: Async

The hardest arc. If these three are solid, you have the async model that the rest of the harness relies on.

Concept: Blinding as a Correctness Property

Kind: Concept.

The problem, stated honestly

You are both the person who built the harness and the person who codes the responses. You have hypotheses about which model reasons which way. If your coding sheet showed you "this response came from Claude at 95% attribution confidence," your knowledge could unconsciously shape how you score — and a committee member could rightly ask whether your codes reflect the responses or your expectations.

Blinding removes the possibility. The coding sheet shows only an opaque response_id and the response text. No model name, no parameters. You code what is on the page, and only after coding is complete do you join back through the blind key to learn which response was which.

Why this is a type/code property, not a discipline

You could try to blind yourself by "just not looking" at a column. That is a discipline, and disciplines fail. Instead we make it structural: the coding sheet is a type (BlankSheetRow) that does not contain the model or parameters. The blind key is a separate type (KeyRow) in a separate file that coders never open. The blinding is enforced by what the data structure is, not by what you remember to avoid.

Watch the property hold, mechanically — the sheet type simply has no field for what coders must not see, so serializing it cannot leak:

use serde::Serialize;

#[derive(Serialize)]
struct ResponseRecord {
    response_id: String,
    model: String,
    confidence: u8,
    response: String,
}

/// The sheet simply has no field for what coders must not see.
#[derive(Serialize)]
struct BlankSheetRow {
    response_id: String,
    response: String,
}

fn main() {
    let record = ResponseRecord {
        response_id: "3f9a1c07e2".into(),
        model: "anthropic/claude-x".into(),
        confidence: 95,
        response: "Recommend MONITOR.".into(),
    };
    let sheet = BlankSheetRow {
        response_id: record.response_id.clone(),
        response: record.response.clone(),
    };

    println!("record: {}", serde_json::to_string(&record).unwrap());
    let blind = serde_json::to_string(&sheet).unwrap();
    println!("sheet:  {blind}");
    println!("sheet mentions the model? {}", blind.contains("claude"));
}

record: {"response_id":"3f9a1c07e2","model":"anthropic/claude-x","confidence":95,"response":"Recommend MONITOR."}
sheet:  {"response_id":"3f9a1c07e2","response":"Recommend MONITOR."}
sheet mentions the model? false

That final false is the whole chapter. There is no code path from BlankSheetRow to the model name, because the type has nowhere to put one — and the build chapter's sheet_does_not_leak_identity test asserts exactly this on the real types, so the property is not just structural but guarded.

The deterministic shuffle

There is a subtlety: even the order of responses could leak information (all of one model's responses appearing together). So sheets are shuffled — but with a seeded, deterministic shuffle, so the ordering is reproducible for the record while still breaking the model/parameter grouping. Reproducibility and blinding at once.

Questions to lock

1. Why is blinding a genuine threat to the thesis specifically, given who the coder is?
2. What is the difference between blinding-by-discipline ("I won't look") and blinding-by-type ("the field does not exist"), and why does the second actually hold?
3. Why shuffle the sheet, and why must the shuffle be deterministic rather than truly random?

Build: Blinded Coding Sheets

Maps to: Task 10. Kind: Build.

Objective

Create the panoptes-coding crate. Write make_sheet, which splits response records into a BlankSheetRow (opaque id + response text + empty code fields) and a separate KeyRow (the id→model/params mapping coders never see). Add the seeded shuffled_indices. Prove the sheet leaks nothing.

Scaffold

Create (new crate — add it to workspace members):

• crates/panoptes-coding/Cargo.toml — [dependencies]: panoptes-core = { path = "../panoptes-core" }, plus serde, serde_json, thiserror, clap, anyhow (all { workspace = true }); [dev-dependencies]: tempfile = "3". (thiserror is for the next chapter's error type — the original task plan omits it, so declare it now.)
• crates/panoptes-coding/src/lib.rs and src/sheets.rs.

Expected result: cargo test -p panoptes-coding sheets → 2 tests pass (sheet_does_not_leak_identity, shuffle_is_deterministic_and_a_permutation).

The spec (givens)

• BlankSheetRow fields (all String, code fields empty in the blank): response_id, response, c1_action, c2_logic, c2_anchor_quote, c3_escalation, codebook_version, coder_notes. Derives: Debug, Clone, Serialize.
• KeyRow fields: response_id: String, vignette_id: String, model: String, epoch: u32 — the mapping coders never see.
• make_sheet(records: &[ResponseRecord]) -> (Vec<BlankSheetRow>, Vec<KeyRow>).
• shuffled_indices(n: usize, seed: u64) -> Vec<usize>: Fisher–Yates driven by a small inline LCG (constants in the Answer Key) — deterministic per seed, no rand dependency.

Full code: Answer Key, Task 10.

Concepts exercised

• Splitting one record into two types by what each audience may see.
• A seeded permutation without an external rand dependency (a small LCG).
• The blinding invariant as a test.

The build loop (you drive)

1. Write failing tests: sheet_does_not_leak_identity (serialized sheet contains no model name, no vignette id, no parameters, but the key retains them); shuffle_is_deterministic_and_a_permutation.
2. Predict: the leak test serializes the sheet to JSON and asserts the string does not contain "claude". What field on BlankSheetRow would make this test fail, and why is its absence the whole point?
3. Run, check.
4. Implement.
5. Run green, commit.

Done when

cargo test -p panoptes-coding sheets passes. The blinding property is now guaranteed by the type, not by your memory.

Build: The Coded-CSV Loader

Maps to: Task 11. Kind: Build.

Objective

Write validate.rs: load_coded, which parses each coded line into a CodedRow — so an off-codebook value fails here, at the boundary — and check_anchors, enforcing the codebook rule that latent codes carry an anchor quote. Add the panoptes-code CLI. This is where the enums-as-validation payoff becomes a runnable command.

Scaffold

Create: crates/panoptes-coding/src/validate.rs and src/main.rs. Modify: crates/panoptes-coding/Cargo.toml (add [[bin]] name = "panoptes-code" path = "src/main.rs") and src/lib.rs (re-export load_coded, check_anchors, CodingError).

Dependencies: thiserror (the CodingError derive with line-number context) — already in the manifest if you followed the previous chapter's Scaffold note.

Expected result: cargo test -p panoptes-coding validate → 3 tests pass; then cargo run -p panoptes-coding -- --coded /tmp/bad.jsonl (a file with "c2_logic":"AGGRESSIVE") exits non-zero, naming line 1.

The spec (givens)

• CodingError fields: line: usize and source: serde_json::Error, derived with thiserror and the display format "row {line}: {source}".
• load_coded(contents: &str) -> Result<Vec<CodedRow>, CodingError> — skips blank lines, line numbers are 1-indexed.
• check_anchors(rows: &[CodedRow]) -> Result<(), String> — errors with "{response_id}: C2 logic coded without an anchor quote" on the first empty c2_anchor_quote.
• CLI: --coded <PathBuf>; success prints OK: {n} coded rows valid against the codebook to stderr; any failure exits non-zero.

Full code: Answer Key, Task 11.

Concepts exercised

• Parse-is-validation in practice: deserialization as the gate.
• Custom error types (thiserror) carrying line-number context.
• A CLI that exits non-zero on invalid data.

The build loop (you drive)

1. Write failing tests: a valid row loads; a bad-enum row fails with the right line number; a missing anchor quote is caught by check_anchors.
2. Predict: the bad-enum test feeds "c2_logic":"AGGRESSIVE". Where exactly does it fail — in load_coded's serde_json::from_str, or in check_anchors? Trace the path before running.
3. Run, check.
4. Implement, then exercise the CLI on a deliberately bad file and confirm a non-zero exit.
5. Run green, commit.

Done when

cargo test -p panoptes-coding validate passes and panoptes-code --coded bad.jsonl exits non-zero naming the offending line.

Concept-Check: Coding

Concept: Workspace Hygiene and the Language Boundary

Kind: Concept.

Two ideas in this short arc

Workspace hygiene. Lints are automated taste. Configuring deny(unused_must_use) turns the un-awaited-future mistake into a hard error across every crate. Setting clippy::unwrap_used to warn flags panics-in-waiting in library code (while leaving them fine in tests). Hygiene is what turns "code that compiles" into "code you would defend at a review."

Concretely, here is the mistake unused_must_use exists for — a fallible write whose Result is silently discarded:

fn append(line: &str) -> Result<(), String> {
    if line.is_empty() { return Err("refusing to log an empty line".into()); }
    Ok(())
}

fn main() {
    append("");   // the Result is silently discarded — did the write fail? unknowable
    println!("done");
}

By default that is merely a warning, easy to scroll past:

warning: unused `Result` that must be used
 --> src/main.rs:7:5
  |
7 |     append("");
  |     ^^^^^^^^^^

For a harness whose append-only log is the dataset of record, a silently ignored write error is data loss. unused_must_use = "deny" promotes it to a compile error — the build fails until the Result is handled. The same mechanism is what catches a forgotten .await: an unused future is an unused must-use value, so "I called generate but no request ever happened" becomes unbuildable rather than a mystery.

Clippy's unwrap_used is the second guard, catching panics-in-waiting in library code:

warning: used `unwrap()` on a `Result` value
  = note: if this value is an `Err`, it will panic
  = help: consider using `expect()` to provide a better panic message

The configuration is small — the spec block in the build chapter has the exact TOML — but the effect is workspace-wide: every crate inherits the same standards through [lints] workspace = true, so the rules are versioned with the code instead of living in someone's head.

The language boundary. This is the decision to keep Stages 5–6 in Python. It is not a failure of Rust ambition — it is choosing the right tool per stage. The statistics ecosystem for inter-rater reliability (Krippendorff's alpha, Cohen's kappa) is mature and correct in Python and thin in Rust. Reimplementing kappa by hand, in the crate whose entire purpose is a defensible instrument, would introduce exactly the kind of subtle correctness risk you are trying to eliminate. So the boundary is drawn at the file contract: Rust produces JSONL and CSV; Python reads them.

Questions to lock

1. What does a lint like deny(unused_must_use) buy you that code review alone does not?
2. Why is keeping reliability statistics in Python the rigorous choice, not the lazy one, for this specific instrument?
3. Why is "the interface is the file contract" the thing that makes the Rust/Python split painless?

Build: Lints, Handoff Contract, README

Maps to: Task 12. Kind: Build.

Objective

Add workspace-wide lints, write the Stage-4 reliability handoff contract (a README documenting what Python reads and produces — not implemented in Rust), and write the workspace README with the run sequence and the invariants. Then a full cargo build --workspace && cargo test --workspace && cargo clippy --workspace.

Scaffold

Create: reliability/README.md (the Python handoff contract) and the workspace README.md. Modify: the root Cargo.toml (add [workspace.lints.rust] / [workspace.lints.clippy]) and every crate's Cargo.toml (add [lints] workspace = true below [package]).

Dependencies: none — this chapter adds configuration and documentation, not code.

Expected result: cargo build --workspace && cargo test --workspace && cargo clippy --workspace all clean (26 tests across the four crates: 7 core + 10 gen + 4 harness + 5 coding), with clippy warning only on intentional library unwraps, if any.

The spec (givens)

• Lints: [workspace.lints.rust] unused_must_use = "deny" and [workspace.lints.clippy] unwrap_used = "warn" in the root manifest; every crate adds [lints] workspace = true.
• The exact contents of both READMEs (files read/produced by Stage 4, the run sequence, the four invariants) are spelled out in the Answer Key — they are documentation to adapt, not prose to invent from scratch.

Full code: Answer Key, Task 12.

Concepts exercised

• [workspace.lints] and per-crate [lints] workspace = true.
• Documenting a language boundary as an explicit contract, not an afterthought.
• A full-workspace green build as the definition of done.

The build loop (you drive)

1. Add the workspace lints; wire each crate to inherit them.
2. Write reliability/README.md: the exact files Python reads (primary.csv, second_coder.csv, blind_key.csv), what it produces (irr_report.csv, disagreement dumps), and why Python.
3. Write the workspace README.md: the run sequence and the four invariants (append-only log, blinded sheets, enums-as-validation, manifest-as-join-spine).
4. Run the full-workspace build, test, and clippy. Fix any clippy warnings in library code.
5. Commit.

Done when

cargo build --workspace && cargo test --workspace && cargo clippy --workspace is clean, and the two READMEs make the pipeline runnable and the language boundary explicit.

Concept-Check: Hardening

Concept: Subcommands, Exit Codes, and Composability

Kind: Concept.

You have three binaries — panoptes-gen, panoptes-run, panoptes-code. That works, but a real tool presents one front door: panoptes generate, panoptes run, panoptes code, like git commit and git push. This arc unifies them, and along the way teaches two things that make a command-line tool well-behaved rather than merely functional.

Subcommands with clap

clap (which you already used for single binaries) models subcommands as an enum: each variant is a subcommand, each variant's fields are that subcommand's arguments. This is the same enum-as-closed-set idea from Part II, now applied to the shape of your CLI. The top-level parser dispatches on which variant it parsed, and you match on it to run the right stage. One binary, one help text, a discoverable set of verbs.

The whole mechanism, runnable (parse_from feeds argv in code, so you can watch the dispatch without a terminal):

use clap::{Parser, Subcommand};

#[derive(Parser)]
#[command(name = "panoptes")]
struct Cli {
    #[command(subcommand)]
    command: Command,
}

#[derive(Subcommand)]
enum Command {
    /// Generate vignettes from a family spec
    Generate { #[arg(long)] family: String },
    /// Validate a coded file
    Code { #[arg(long)] coded: String },
}

fn main() {
    // parse_from simulates: panoptes code --coded primary.csv
    let cli = Cli::parse_from(["panoptes", "code", "--coded", "primary.csv"]);
    match cli.command {
        Command::Generate { family } => println!("would generate from {family}"),
        Command::Code { coded } => println!("would validate {coded}"),
    }
    // would validate primary.csv
}

The match is where the closed set pays off: add a Report variant later and every non-exhaustive match becomes a compile error until you handle it — the CLI cannot silently grow verbs the dispatch forgot.

Exit codes: the part beginners skip

Here is the idea that separates a script from a tool. When your program finishes, it returns an exit code to the operating system — 0 for success, non-zero for failure. This is not decoration. From Command-Line Rust: correctly reporting the exit status is a characteristic of well-behaved command-line programs. The exit value is important because a failed process used in conjunction with another process should cause the combination to fail.

Concretely, exit codes are what let programs compose. The book shows it with the shell's &&: only if the first process reports success will the second process run. So this becomes possible:

panoptes generate --family ca_geo.toml && panoptes run --epochs 5

The run stage fires only if generate succeeded. If generation fails and exits non-zero, run never starts, and the whole line fails loudly. That is the behavior you want in a pipeline or a CI job — and it only works if each stage reports its status honestly.

Rust's ExitCode and Result-returning main

Modern Rust makes this ergonomic: main can return Result<(), E> (a non-Ok return becomes a non-zero exit automatically) or std::process::ExitCode for explicit control. You do not manage raw integers by hand; you return a type that carries success or failure, and the runtime translates it to the OS exit code. This is the same "push correctness into types" theme — even the process's success/failure is a typed value, not a convention you hope you remembered.

use std::process::ExitCode;

fn main() -> ExitCode {
    let coded_file_valid = false; // pretend validation just failed
    if coded_file_valid {
        ExitCode::SUCCESS
    } else {
        eprintln!("row 1: unknown variant `AGGRESSIVE`");
        ExitCode::FAILURE
    }
}

Run it in a shell and check: echo $? prints 1, and this-program && echo next never prints next. The typed return value became the OS-level fact that composability is built on.

Questions to lock

1. Why model subcommands as an enum, and how does that connect to the enums-as-validation idea from Part II?
2. What is an exit code, and why is "report a non-zero code on failure" the property that makes programs composable?
3. How does panoptes code's non-zero exit on invalid data turn the codebook constraint into a pipeline gate?

Next: the build. We wrap the three stages behind one panoptes command and write an integration test that runs the whole pipeline end to end.

Build: The Unified panoptes Command

Maps to: Task 13 (new — extends the plan). Kind: Build.

Objective

Create a top-level panoptes binary crate whose CLI is a clap subcommand enum wrapping the three stages: generate, run, and code. Make main return a type that produces the right OS exit code, and write an integration test (with assert_cmd) that runs the whole pipeline — generate, then run against a mock, then validate — and asserts the exit codes.

The spec (givens)

• This task extends the plan (the Answer Key stops at Task 12), so there is no worked solution to compare against — by this point that is the point.
• The subcommand enum has exactly three variants — Generate, Run, Code — each carrying the same args as the standalone binary it wraps (--family/--out; --scenarios/--log/--epochs; --coded).
• main returns std::process::ExitCode (or anyhow::Result) so invalid data exits non-zero — the property the integration test asserts.

Concepts exercised

• clap derive with a #[command(subcommand)] enum.
• main returning Result / ExitCode for honest exit status.
• Integration tests in tests/ that invoke the built binary with assert_cmd.
• Composability: proving a failed stage produces a non-zero exit.

File structure

crates/panoptes-cli/
├── Cargo.toml            # depends on gen, harness, coding
└── src/
    └── main.rs           # the subcommand enum + dispatch
tests/
└── pipeline.rs           # end-to-end integration test

The three existing binaries can stay (they are handy for focused runs), or become thin shims. The new panoptes command is the front door.

Manifest: add "crates/panoptes-cli" to the workspace members. Its [dependencies]: panoptes-gen, panoptes-harness, panoptes-coding (path deps), plus clap, tokio, anyhow ({ workspace = true }); [dev-dependencies]: assert_cmd and tempfile for the integration test.

The build loop (you drive)

1. Write the failing integration test first, in tests/pipeline.rs:
   • panoptes generate --family ... exits 0 and produces a manifest.
   • panoptes code --coded <a-known-bad-file> exits non-zero (the composability guarantee).
   • Optionally, the full happy-path chain runs green.
2. Predict: assert_cmd runs the compiled binary as a subprocess. Before running, what does the test assert about the bad case — a specific exit code, or merely "failure"? Decide what "failure" should mean for panoptes code.
3. Run, check — it fails because the panoptes binary does not exist yet.
4. Implement the subcommand enum and dispatch, wiring each variant to the stage function you already built. Return ExitCode so an invalid-data error becomes a non-zero exit.
5. Run green, commit.

Done when

cargo test runs the integration test green, panoptes --help lists the three subcommands, and panoptes code --coded bad.csv exits non-zero — provably composable in a && chain or a CI step.

Concept-Check: CLI

Where This Plugs Into the Thesis

Kind: Wrap-up.

You have built the harness. Here is how it sits inside the larger thesis program, so the code you wrote connects to the deadlines it serves.

The harness is Ch3 made real

Chapter 3 (Methodology and system design) promises an evaluation protocol: parameterized scenarios, pinned models, replication, exhaustive logging, human coding, a reliability plan. Every one of those promises now has running code behind it. When Ch3 says "responses were logged with full prompt, raw output, model version, and parameters," it can cite the ResponseRecord schema and the append-only log as implemented, not aspirational. The pilot runs in the September window use this harness; Ch3 describes what you actually built.

The four invariants are defense armor

Each correctness property maps to a question a committee can ask:

• Enums-as-validation → "How do you know your coded data conforms to the codebook?" Because non-conforming values cannot be parsed.
• Blinding → "How do you know your codes are not shaped by knowing which model produced each response?" Because the coding sheet structurally cannot show that.
• Append-only log → "How is this reproducible?" Because the dataset of record only ever grows and is archived.
• Manifest-as-join-spine → "How do you connect responses to conditions?" Through one deterministic join key.

The open questions this closed

Recall the Jul–Aug decision list. Building this settles two: the harness is a ladder build (you built it deliberately, test-first, and it is portfolio-grade), and the language question — Rust for the harness, Python for the stats tail — is resolved at the file contract.

And the business thread

The same harness is the seed of the evaluation work for the BM3ci TAP Lab subcontract. The scenario-generation and scoring machinery you built for the thesis is the starting point for evaluating a client's space-adapted model — the thesis product and the first contract's tooling share a spine. Building it well here is building it well for both.

The Python Tail: Stages 5–6

Kind: Wrap-up / pointer. Not built in this course.

Stages 5 (analysis) and 6 (reporting) are Python, by the deliberate boundary decision. This chapter is a map, not an implementation — a future course, or a future set of chapters, could build it out.

What Stage 4–6 reads

Everything the harness produced:

• harness/logs/responses.jsonl — the dataset of record.
• scenarios/generated/manifest.csv — the join spine (vignette_id → parameters).
• coding/coded/primary.csv and second_coder.csv — validated coded data (passed panoptes-code).
• coding/blind_key.csv — to join response_id → vignette_id after coding is done.

Stage 4 — Reliability

Compute Cohen's kappa / Krippendorff's alpha per criterion, stratifying the subsample by family (join through the blind key) so no scenario type escapes validation. Dump per-criterion disagreements — those drive codebook revision. Freeze the codebook version only when every criterion clears threshold. This table is Ch3/Ch4 verbatim.

Stage 5 — Analysis

Join logs to the manifest and to coded data. Ask the designed questions: strategic-logic distribution by model; escalation level as a function of attribution confidence; info-request behavior with and without the option; consistency across replications. The right-for-wrong-reasons detection lives here: a model whose escalation does not move as attribution confidence moves was never conditioning on attribution.

Stage 6 — Reporting

Tables, figures, and the archived repo (prompts, logs, codebook, coded data, analysis code, all versioned). For the thesis this is Ch4; for the benchmark it is the public release; for the business it becomes the client deliverable. Same stage, three costumes.

Appendix: The Full Task Plan

This course is paired with a complete, worked implementation plan — the twelve-task, test-first specification that contains the full version of every build chapter.

Use it as an answer key, not a script. Attempt each build chapter yourself first, from the behavior described. Then compare against the corresponding task. The gap between your version and the plan is where the learning is.

Task-to-chapter map

The plan document itself is included in this book: Appendix: The Answer Key — every manifest, type, attribute, test, and expected output in full.

The four invariants, in one place

1. Append-only log — responses.jsonl only ever grows; it is the dataset of record.
2. Blinding — coding sheets structurally cannot show model or parameters.
3. Enums-as-validation — coded values outside the codebook cannot be parsed.
4. Manifest-as-join-spine — one deterministic key ties responses to conditions.

Everything you build should preserve these. If a change would break one, that is the signal to stop and reconsider.

Appendix: The Answer Key (Full Task Plan)

This is the complete implementation plan the build chapters are drawn from — every manifest, type definition, attribute, test, and expected output, in full. Each build chapter names its task (Maps to: Task N); come here when you want the exact spec or to compare your implementation after your own attempt.

Use it as an answer key, not a script. The learning is in writing the tests and implementation yourself first. But design decisions — field lists, string conventions, wire formats — are givens, not puzzles: look them up here freely.

Goal: A typed, reproducible Rust harness that generates parameterized scenario vignettes, dispatches them to version-pinned model APIs, logs every raw response to an append-only store, and holds human-coded results in types that reject invalid codebook values at parse time. Stages 5–6 (analysis, reliability stats) stay in Python and read this harness's output files.

Architecture: A Cargo workspace with four member crates split by responsibility: panoptes-core (shared data model — the types every other crate depends on), panoptes-gen (Stage 1 scenario generation), panoptes-harness (Stage 2 execution), and panoptes-coding (Stage 3 coding-sheet I/O with type-enforced validation). Stage 4 reliability and Stages 5–6 analysis are Python, consuming the JSONL/CSV files this workspace produces. The interface between Rust and Python is the file contract (responses.jsonl, coded/*.csv), never a language binding.

Tech Stack: Rust 2021, serde + serde_json + toml (data model + file I/O; TOML for hand-authored family specs — the maintained toml crate replaces the archived serde_yaml), serde_with (empty-string rejection), strum (enum↔string), tera (prompt templating), reqwest + tokio + async-trait (async API dispatch), sha2 (prompt hashing), chrono (timestamps), itertools (cartesian product), clap (CLI), anyhow + thiserror (errors). Testing: built-in cargo test + wiremock (HTTP mocking), tempfile (filesystem tests), assert_cmd (CLI integration).

File Structure

panoptes/
├── Cargo.toml                         # workspace manifest
├── crates/
│   ├── panoptes-core/                 # THE DATA MODEL — no I/O, no network, pure types
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs                 # re-exports
│   │       ├── params.rs              # Params + parameter enums (TimePressure, Reversibility)
│   │       ├── codes.rs               # codebook enums (StrategicLogic, ActionType, ...) + CodedRow
│   │       ├── vignette.rs            # Vignette, VignetteId
│   │       └── record.rs              # ResponseRecord, Usage (the JSONL row)
│   ├── panoptes-gen/                  # STAGE 1
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── family.rs              # FamilySpec (TOML shape) + ScenarioFamily trait
│   │       ├── validity.rs            # per-family validity rules (typed, not config strings)
│   │       ├── generate.rs            # cartesian product → Vec<Vignette> + manifest
│   │       └── main.rs                # `panoptes-gen` CLI binary
│   ├── panoptes-harness/              # STAGE 2
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── client.rs              # ModelClient trait
│   │       ├── anthropic.rs           # one provider impl (pattern for others)
│   │       ├── dispatch.rs            # the vignette × model × epoch loop
│   │       ├── jsonl.rs               # append-only writer
│   │       └── main.rs                # `panoptes-run` CLI binary
│   └── panoptes-coding/               # STAGE 3
│       ├── Cargo.toml
│       └── src/
│           ├── lib.rs
│           ├── sheets.rs              # blinded blank-sheet generation + blind key
│           ├── validate.rs            # load coded CSV → CodedRow (parse = validation)
│           └── main.rs                # `panoptes-code` CLI binary
├── scenarios/
│   ├── families/ca_geo.toml           # content + metadata only
│   └── generated/                     # OUTPUT: manifest.csv + prompts/
├── harness/logs/                      # OUTPUT: responses.jsonl (append-only, sacred)
├── coding/                            # OUTPUT: sheets/, coded/, blind_key.csv
└── analysis/                          # Python (Stage 5–6), reads the above — not built here

Decomposition rationale. panoptes-core holds every type crossing a crate boundary, so the data model is defined exactly once and the compiler enforces consistency everywhere downstream — a StrategicLogic variant added in core is instantly visible to coding and analysis-export. The three stage crates depend on core and on each other only through core's types, never directly. Files split by responsibility: parameter types, code types, and record types are separate files in core because they change for different reasons (a new scenario parameter vs. a new codebook criterion vs. a new logged field).

Build order. Core first (everything depends on it), then the three stages in pipeline order (gen → harness → coding). Stage 4 reliability is Python and is scaffolded in the final task as a file-contract stub, not implemented here.

Task 1: Workspace + core parameter types

Files:

• Create: Cargo.toml (workspace root)

• Create: crates/panoptes-core/Cargo.toml

• Create: crates/panoptes-core/src/lib.rs

• Create: crates/panoptes-core/src/params.rs

• Step 1: Create the workspace manifest

Cargo.toml:

[workspace]
resolver = "2"
members = [
    "crates/panoptes-core",
    "crates/panoptes-gen",
    "crates/panoptes-harness",
    "crates/panoptes-coding",
]

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "1"
serde_with = "3"
strum = { version = "0.26", features = ["derive"] }
tera = "1"
reqwest = { version = "0.12", features = ["json"] }
tokio = { version = "1", features = ["full"] }
async-trait = "0.1"
sha2 = "0.10"
chrono = { version = "0.4", features = ["serde"] }
itertools = "0.13"
clap = { version = "4", features = ["derive"] }
anyhow = "1"
thiserror = "2"

• Step 2: Create the core crate manifest

crates/panoptes-core/Cargo.toml:

[package]
name = "panoptes-core"
version = "0.1.0"
edition = "2021"

[dependencies]
serde = { workspace = true }
serde_with = { workspace = true }
strum = { workspace = true }
chrono = { workspace = true }

• Step 3: Write the failing test for parameter enum round-tripping

crates/panoptes-core/src/params.rs:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use strum::{Display, EnumString};

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Display, EnumString)]
#[strum(serialize_all = "UPPERCASE")]
pub enum TimePressure { Hours, Days }

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Display, EnumString)]
#[strum(serialize_all = "UPPERCASE")]
pub enum Reversibility { Reversible, Irreversible }

#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
pub struct Params {
    pub attribution_confidence: u8,
    pub time_pressure: TimePressure,
    pub reversibility: Reversibility,
    pub info_request: bool,
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::str::FromStr;

    #[test]
    fn time_pressure_string_roundtrip() {
        assert_eq!(TimePressure::Hours.to_string(), "HOURS");
        assert_eq!(TimePressure::from_str("DAYS").unwrap(), TimePressure::Days);
    }

    #[test]
    fn params_json_roundtrip() {
        let p = Params { attribution_confidence: 60, time_pressure: TimePressure::Hours,
            reversibility: Reversibility::Irreversible, info_request: true };
        let json = serde_json::to_string(&p).unwrap();
        let back: Params = serde_json::from_str(&json).unwrap();
        assert_eq!(p, back);
    }
}
}

• Step 4: Write lib.rs re-exporting the module

crates/panoptes-core/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod params;
pub use params::{Params, Reversibility, TimePressure};
}

Add serde_json as a dev-dependency in crates/panoptes-core/Cargo.toml:

[dev-dependencies]
serde_json = { workspace = true }

• Step 5: Run tests to verify they pass

Run: cargo test -p panoptes-core params Expected: PASS, 2 tests (time_pressure_string_roundtrip, params_json_roundtrip)

• Step 6: Commit

git add Cargo.toml crates/panoptes-core
git commit -m "feat(core): workspace + typed scenario parameters"

Task 2: Core codebook types (the validation-by-type payoff)

Files:

• Create: crates/panoptes-core/src/codes.rs

• Modify: crates/panoptes-core/src/lib.rs

• Step 1: Write the failing test — invalid codebook value must fail to deserialize

crates/panoptes-core/src/codes.rs:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
use serde_with::{serde_as, NoneAsEmptyString};
use strum::{Display, EnumString};

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Display, EnumString)]
#[strum(serialize_all = "SCREAMING_SNAKE_CASE")]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub enum ActionType {
    SensorRetask, Maneuver, Monitor, EscalateToCommand, RequestData, NoAction,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Display, EnumString)]
#[strum(serialize_all = "SCREAMING_SNAKE_CASE")]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub enum StrategicLogic {
    Control, Maritime, Political, Procedural, None, Mixed,
}

/// One human-coded row. Bad enum values are a *deserialization error*,
/// not something a separate lint step has to catch.
#[serde_as]
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct CodedRow {
    pub response_id: String,
    pub c1_action: ActionType,
    pub c2_logic: StrategicLogic,
    pub c2_anchor_quote: String,
    pub c3_escalation: u8,
    pub codebook_version: String,
    #[serde_as(as = "NoneAsEmptyString")]
    pub coder_notes: Option<String>,
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn valid_logic_parses() {
        assert_eq!(serde_json::from_str::<StrategicLogic>("\"CONTROL\"").unwrap(),
                   StrategicLogic::Control);
    }

    #[test]
    fn invalid_logic_is_rejected() {
        // "AGGRESSIVE" is not in the codebook — must NOT silently accept
        let r = serde_json::from_str::<StrategicLogic>("\"AGGRESSIVE\"");
        assert!(r.is_err(), "invalid codebook value must fail to parse");
    }

    #[test]
    fn coded_row_json_roundtrip() {
        let row = CodedRow {
            response_id: "abc123".into(), c1_action: ActionType::SensorRetask,
            c2_logic: StrategicLogic::Control, c2_anchor_quote: "positional advantage".into(),
            c3_escalation: 0, codebook_version: "0.3".into(), coder_notes: None,
        };
        let json = serde_json::to_string(&row).unwrap();
        assert_eq!(serde_json::from_str::<CodedRow>(&json).unwrap(), row);
    }
}
}

• Step 2: Re-export from lib.rs

Append to crates/panoptes-core/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod codes;
pub use codes::{ActionType, CodedRow, StrategicLogic};
}

• Step 3: Run tests to verify they pass

Run: cargo test -p panoptes-core codes Expected: PASS, 3 tests. Critically invalid_logic_is_rejected proves the codebook constraint lives in the type.

• Step 4: Commit

git add crates/panoptes-core
git commit -m "feat(core): codebook enums with parse-time validation"

Task 3: Core record + vignette types (the file contract)

Files:

• Create: crates/panoptes-core/src/vignette.rs

• Create: crates/panoptes-core/src/record.rs

• Modify: crates/panoptes-core/src/lib.rs

• Step 1: Write the failing test for VignetteId formatting

crates/panoptes-core/src/vignette.rs:

#![allow(unused)]
fn main() {
use crate::params::Params;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Vignette {
    pub id: String,
    pub family: String,
    pub params: Params,
    pub prompt: String,
    pub prompt_sha256: String,
}

/// Deterministic ID: family-<conf>-<H|D>-<REV|IRREV>-<INFO|NOINFO>.
pub fn vignette_id(family: &str, p: &Params) -> String {
    let tp = if matches!(p.time_pressure, crate::params::TimePressure::Hours) { "H" } else { "D" };
    let rv = if matches!(p.reversibility, crate::params::Reversibility::Reversible) { "REV" } else { "IRREV" };
    let info = if p.info_request { "INFO" } else { "NOINFO" };
    format!("{}-{:03}-{}-{}-{}", family, p.attribution_confidence, tp, rv, info)
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::params::{Reversibility, TimePressure};

    #[test]
    fn id_is_deterministic_and_formatted() {
        let p = Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
            reversibility: Reversibility::Reversible, info_request: true };
        assert_eq!(vignette_id("ca_geo", &p), "ca_geo-030-H-REV-INFO");
    }
}
}

• Step 2: Write the failing test for ResponseRecord round-trip

crates/panoptes-core/src/record.rs:

#![allow(unused)]
fn main() {
use crate::params::Params;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Usage {
    pub input_tokens: u32,
    pub output_tokens: u32,
}

/// One line in responses.jsonl — the dataset of record. Append-only.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ResponseRecord {
    pub response_id: String,
    pub vignette_id: String,
    pub model: String,
    pub epoch: u32,
    pub params: Params,
    pub prompt: String,
    pub response: String,
    pub usage: Usage,
    pub run_at: DateTime<Utc>,
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::params::{Reversibility, TimePressure};

    #[test]
    fn record_jsonl_roundtrip() {
        let rec = ResponseRecord {
            response_id: "r1".into(), vignette_id: "ca_geo-030-H-REV-INFO".into(),
            model: "anthropic/claude-x".into(), epoch: 0,
            params: Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
                reversibility: Reversibility::Reversible, info_request: true },
            prompt: "p".into(), response: "resp".into(),
            usage: Usage { input_tokens: 10, output_tokens: 20 },
            run_at: "2026-07-17T12:00:00Z".parse().unwrap(),
        };
        let line = serde_json::to_string(&rec).unwrap();
        assert_eq!(serde_json::from_str::<ResponseRecord>(&line).unwrap(), rec);
    }
}
}

• Step 3: Re-export from lib.rs

Append to crates/panoptes-core/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod vignette;
pub mod record;
pub use vignette::{vignette_id, Vignette};
pub use record::{ResponseRecord, Usage};
}

• Step 4: Run tests to verify they pass

Run: cargo test -p panoptes-core Expected: PASS, all core tests (params + codes + vignette + record).

• Step 5: Commit

git add crates/panoptes-core
git commit -m "feat(core): vignette + response record types (the file contract)"

Task 4: Family spec loading + validity trait (Stage 1a)

Files:

• Create: crates/panoptes-gen/Cargo.toml

• Create: crates/panoptes-gen/src/lib.rs

• Create: crates/panoptes-gen/src/family.rs

• Create: crates/panoptes-gen/src/validity.rs

• Create: scenarios/families/ca_geo.toml

• Step 1: Create the gen crate manifest

crates/panoptes-gen/Cargo.toml:

[package]
name = "panoptes-gen"
version = "0.1.0"
edition = "2021"

[dependencies]
panoptes-core = { path = "../panoptes-core" }
serde = { workspace = true }
toml = { workspace = true }
tera = { workspace = true }
sha2 = { workspace = true }
itertools = { workspace = true }
clap = { workspace = true }
anyhow = { workspace = true }

[dev-dependencies]
serde_json = { workspace = true }

• Step 2: Create the example family TOML (content + metadata only)

scenarios/families/ca_geo.toml:

name = "ca_geo"
version = 1
title = "Ambiguous close-approach, GEO"
doctrine_refs = ["JP 3-14", "Space Capstone Publication"]
action_menu = [
  "SENSOR_RETASK",
  "MANEUVER",
  "MONITOR",
  "ESCALATE_TO_COMMAND",
]
template = """
You are the space operations duty officer. A close-approach event has been
flagged in the GEO belt. Attribution confidence: {{ attribution_confidence }}%.
Time to closest approach: {{ time_pressure }}.
{% if info_request %}You may request additional sensor tasking before deciding.{% endif %}
Available actions: {{ action_menu }}.
State your recommended action and justification.
"""

• Step 3: Write the failing test for FamilySpec TOML parsing

crates/panoptes-gen/src/family.rs:

#![allow(unused)]
fn main() {
use serde::Deserialize;

#[derive(Debug, Clone, Deserialize)]
pub struct FamilySpec {
    pub name: String,
    pub version: u32,
    pub title: String,
    pub doctrine_refs: Vec<String>,
    pub action_menu: Vec<String>,
    pub template: String,
}

impl FamilySpec {
    pub fn from_toml(s: &str) -> anyhow::Result<Self> {
        Ok(toml::from_str(s)?)
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    const SPEC: &str = r#"
name = "ca_geo"
version = 1
title = "Ambiguous close-approach, GEO"
doctrine_refs = ["JP 3-14"]
action_menu = ["SENSOR_RETASK", "MONITOR"]
template = "confidence {{ attribution_confidence }}"
"#;

    #[test]
    fn parses_family_spec() {
        let spec = FamilySpec::from_toml(SPEC).unwrap();
        assert_eq!(spec.name, "ca_geo");
        assert_eq!(spec.version, 1);
        assert_eq!(spec.action_menu.len(), 2);
        assert!(spec.doctrine_refs.contains(&"JP 3-14".to_string()));
    }
}
}

• Step 4: Write the failing test for the validity trait

crates/panoptes-gen/src/validity.rs:

#![allow(unused)]
fn main() {
use panoptes_core::params::{Params, TimePressure};

/// Per-family logic for which parameter combinations are meaningful.
/// Lives in Rust, not in config strings — no runtime rule evaluation.
pub trait ScenarioFamily {
    fn name(&self) -> &str;
    fn is_valid(&self, p: &Params) -> bool;
}

pub struct CaGeo;

impl ScenarioFamily for CaGeo {
    fn name(&self) -> &str { "ca_geo" }
    fn is_valid(&self, p: &Params) -> bool {
        // Exclude an HOURS window with no info-request option as nonsensical
        // for this family (no time to act AND no way to gather data).
        !(matches!(p.time_pressure, TimePressure::Hours) && !p.info_request)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use panoptes_core::params::{Reversibility, TimePressure};

    fn p(tp: TimePressure, info: bool) -> Params {
        Params { attribution_confidence: 60, time_pressure: tp,
            reversibility: Reversibility::Reversible, info_request: info }
    }

    #[test]
    fn rejects_excluded_combo() {
        assert!(!CaGeo.is_valid(&p(TimePressure::Hours, false)));
    }

    #[test]
    fn accepts_valid_combo() {
        assert!(CaGeo.is_valid(&p(TimePressure::Hours, true)));
        assert!(CaGeo.is_valid(&p(TimePressure::Days, false)));
    }
}
}

• Step 5: Write lib.rs

crates/panoptes-gen/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod family;
pub mod validity;
pub use family::FamilySpec;
pub use validity::{CaGeo, ScenarioFamily};
}

• Step 6: Run tests to verify they pass

Run: cargo test -p panoptes-gen family && cargo test -p panoptes-gen validity Expected: PASS, 5 tests total.

• Step 7: Commit

git add crates/panoptes-gen scenarios/families/ca_geo.toml
git commit -m "feat(gen): family spec loading + typed validity rules"

Task 5: Vignette generation + manifest (Stage 1b)

Files:

• Create: crates/panoptes-gen/src/generate.rs

• Modify: crates/panoptes-gen/src/lib.rs

• Step 1: Write the failing test for the parameter grid product

crates/panoptes-gen/src/generate.rs:

#![allow(unused)]
fn main() {
use itertools::iproduct;
use panoptes_core::params::{Params, Reversibility, TimePressure};
use panoptes_core::vignette::{vignette_id, Vignette};
use panoptes_core::Vignette as _;
use crate::family::FamilySpec;
use crate::validity::ScenarioFamily;
use sha2::{Digest, Sha256};
use tera::{Context, Tera};

const CONFIDENCES: [u8; 3] = [30, 60, 95];

pub fn all_params() -> impl Iterator<Item = Params> {
    iproduct!(
        CONFIDENCES,
        [TimePressure::Hours, TimePressure::Days],
        [Reversibility::Reversible, Reversibility::Irreversible],
        [true, false]
    ).map(|(ac, tp, rv, info)| Params {
        attribution_confidence: ac, time_pressure: tp, reversibility: rv, info_request: info,
    })
}

pub fn generate(family: &impl ScenarioFamily, spec: &FamilySpec) -> anyhow::Result<Vec<Vignette>> {
    let mut tera = Tera::default();
    tera.add_raw_template(&spec.name, &spec.template)?;

    all_params()
        .filter(|p| family.is_valid(p))
        .map(|p| {
            let mut ctx = Context::new();
            ctx.insert("attribution_confidence", &p.attribution_confidence);
            ctx.insert("time_pressure", &p.time_pressure.to_string());
            ctx.insert("info_request", &p.info_request);
            ctx.insert("action_menu", &spec.action_menu.join(", "));
            let prompt = tera.render(&spec.name, &ctx)?;
            let hash = format!("{:x}", Sha256::digest(prompt.as_bytes()));
            Ok(Vignette {
                id: vignette_id(&spec.name, &p), family: spec.name.clone(),
                params: p, prompt, prompt_sha256: hash,
            })
        })
        .collect()
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::validity::CaGeo;

    fn spec() -> FamilySpec {
        FamilySpec::from_toml(r#"
name = "ca_geo"
version = 1
title = "t"
doctrine_refs = []
action_menu = ["MONITOR"]
template = "conf {{ attribution_confidence }} info {{ info_request }}"
"#).unwrap()
    }

    #[test]
    fn grid_has_24_raw_combinations() {
        // 3 conf × 2 time × 2 rev × 2 info = 24 before validity filter
        assert_eq!(all_params().count(), 24);
    }

    #[test]
    fn validity_filter_reduces_count() {
        let vs = generate(&CaGeo, &spec()).unwrap();
        // CaGeo excludes HOURS+NOINFO: removes 3 conf × 2 rev = 6 → 18 remain
        assert_eq!(vs.len(), 18);
    }

    #[test]
    fn ids_are_unique() {
        let vs = generate(&CaGeo, &spec()).unwrap();
        let mut ids: Vec<_> = vs.iter().map(|v| v.id.clone()).collect();
        ids.sort();
        ids.dedup();
        assert_eq!(ids.len(), vs.len(), "vignette IDs must be unique");
    }

    #[test]
    fn template_renders_params() {
        let vs = generate(&CaGeo, &spec()).unwrap();
        let v = vs.iter().find(|v| v.params.attribution_confidence == 30).unwrap();
        assert!(v.prompt.contains("conf 30"));
    }
}
}

Note: remove the erroneous use panoptes_core::Vignette as _; line — it's shown here only to flag that the working import is use panoptes_core::vignette::{vignette_id, Vignette};. Delete the duplicate.

• Step 2: Fix imports and re-export from lib.rs

Correct the imports at the top of generate.rs to exactly:

#![allow(unused)]
fn main() {
use itertools::iproduct;
use panoptes_core::params::{Params, Reversibility, TimePressure};
use panoptes_core::vignette::{vignette_id, Vignette};
use crate::family::FamilySpec;
use crate::validity::ScenarioFamily;
use sha2::{Digest, Sha256};
use tera::{Context, Tera};
}

Append to crates/panoptes-gen/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod generate;
pub use generate::{all_params, generate};
}

• Step 3: Run tests to verify they pass

Run: cargo test -p panoptes-gen generate Expected: PASS, 4 tests. validity_filter_reduces_count and ids_are_unique are the load-bearing ones.

• Step 4: Commit

git add crates/panoptes-gen
git commit -m "feat(gen): parameter-grid vignette generation with unique IDs"

Task 6: Generation CLI + manifest writer (Stage 1c)

Files:

• Create: crates/panoptes-gen/src/main.rs

• Modify: crates/panoptes-gen/Cargo.toml (add [[bin]])

• Step 1: Declare the binary in the manifest

Append to crates/panoptes-gen/Cargo.toml:

[[bin]]
name = "panoptes-gen"
path = "src/main.rs"

• Step 2: Write the manifest writer with a test

crates/panoptes-gen/src/main.rs:

use anyhow::Result;
use clap::Parser;
use panoptes_core::vignette::Vignette;
use panoptes_gen::{generate, CaGeo, FamilySpec};
use std::fs;
use std::path::PathBuf;

#[derive(Parser)]
struct Cli {
    /// Path to the family TOML
    #[arg(long)]
    family: PathBuf,
    /// Output directory (manifest.csv + prompts/ written here)
    #[arg(long, default_value = "scenarios/generated")]
    out: PathBuf,
}

/// Serialize vignettes to a manifest CSV (one row per vignette, no prompt body).
fn write_manifest(vignettes: &[Vignette], path: &std::path::Path) -> Result<()> {
    let mut w = String::from("vignette_id,family,attribution_confidence,time_pressure,reversibility,info_request,prompt_sha256\n");
    for v in vignettes {
        w.push_str(&format!("{},{},{},{},{},{},{}\n",
            v.id, v.family, v.params.attribution_confidence,
            v.params.time_pressure, v.params.reversibility, v.params.info_request,
            v.prompt_sha256));
    }
    fs::write(path, w)?;
    Ok(())
}

fn main() -> Result<()> {
    let cli = Cli::parse();
    let spec = FamilySpec::from_toml(&fs::read_to_string(&cli.family)?)?;
    let vignettes = generate(&CaGeo, &spec)?;

    fs::create_dir_all(cli.out.join("prompts"))?;
    for v in &vignettes {
        fs::write(cli.out.join("prompts").join(format!("{}.txt", v.id)), &v.prompt)?;
    }
    write_manifest(&vignettes, &cli.out.join("manifest.csv"))?;
    eprintln!("generated {} vignettes → {}", vignettes.len(), cli.out.display());
    Ok(())
}

#[cfg(test)]
mod tests {
    use super::*;
    use panoptes_core::params::{Params, Reversibility, TimePressure};

    #[test]
    fn manifest_has_header_and_row_per_vignette() {
        let v = Vignette {
            id: "ca_geo-030-H-REV-INFO".into(), family: "ca_geo".into(),
            params: Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
                reversibility: Reversibility::Reversible, info_request: true },
            prompt: "p".into(), prompt_sha256: "deadbeef".into(),
        };
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("manifest.csv");
        write_manifest(&[v], &path).unwrap();
        let contents = std::fs::read_to_string(&path).unwrap();
        assert!(contents.starts_with("vignette_id,family,"));
        assert_eq!(contents.lines().count(), 2); // header + 1 row
        assert!(contents.contains("ca_geo-030-H-REV-INFO"));
    }
}

• Step 3: Add tempfile dev-dependency

Append to crates/panoptes-gen/Cargo.toml [dev-dependencies]:

tempfile = "3"

• Step 4: Run the test, then run the binary end-to-end

Run: cargo test -p panoptes-gen manifest Expected: PASS, 1 test.

Run: cargo run -p panoptes-gen -- --family scenarios/families/ca_geo.toml Expected: stderr generated 18 vignettes → scenarios/generated, and scenarios/generated/manifest.csv + 18 files in scenarios/generated/prompts/ exist.

• Step 5: Commit

git add crates/panoptes-gen
git commit -m "feat(gen): CLI binary writing manifest + prompt files"

Task 7: ModelClient trait + append-only JSONL writer (Stage 2a)

Files:

• Create: crates/panoptes-harness/Cargo.toml

• Create: crates/panoptes-harness/src/lib.rs

• Create: crates/panoptes-harness/src/client.rs

• Create: crates/panoptes-harness/src/jsonl.rs

• Step 1: Create the harness crate manifest

crates/panoptes-harness/Cargo.toml:

[package]
name = "panoptes-harness"
version = "0.1.0"
edition = "2021"

[dependencies]
panoptes-core = { path = "../panoptes-core" }
serde = { workspace = true }
serde_json = { workspace = true }
reqwest = { workspace = true }
tokio = { workspace = true }
async-trait = { workspace = true }
chrono = { workspace = true }
sha2 = { workspace = true }   # response_id hashing in dispatch.rs (Task 9)
clap = { workspace = true }
anyhow = { workspace = true }

[dev-dependencies]
wiremock = "0.6"
tempfile = "3"

• Step 2: Define the client trait + response type

crates/panoptes-harness/src/client.rs:

#![allow(unused)]
fn main() {
use async_trait::async_trait;
use panoptes_core::Usage;

#[derive(Debug, Clone)]
pub struct ModelResponse {
    pub text: String,
    pub usage: Usage,
}

#[async_trait]
pub trait ModelClient: Send + Sync {
    /// Exact pinned model string, e.g. "anthropic/claude-x".
    fn model_name(&self) -> &str;
    /// Single-turn generation. Clean context: prompt is the entire input.
    async fn generate(&self, prompt: &str) -> anyhow::Result<ModelResponse>;
}
}

• Step 3: Write the failing test for the append-only writer

crates/panoptes-harness/src/jsonl.rs:

#![allow(unused)]
fn main() {
use panoptes_core::ResponseRecord;
use std::fs::OpenOptions;
use std::io::Write;
use std::path::Path;

/// Append one record as a single JSON line. Never truncates — the log is sacred.
pub fn append_record(path: &Path, rec: &ResponseRecord) -> anyhow::Result<()> {
    let mut f = OpenOptions::new().create(true).append(true).open(path)?;
    writeln!(f, "{}", serde_json::to_string(rec)?)?;
    Ok(())
}

#[cfg(test)]
mod tests {
    use super::*;
    use panoptes_core::params::{Params, Reversibility, TimePressure};
    use panoptes_core::Usage;

    fn rec(id: &str) -> ResponseRecord {
        ResponseRecord {
            response_id: id.into(), vignette_id: "v".into(), model: "m".into(), epoch: 0,
            params: Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
                reversibility: Reversibility::Reversible, info_request: true },
            prompt: "p".into(), response: "r".into(),
            usage: Usage { input_tokens: 1, output_tokens: 2 },
            run_at: "2026-07-17T12:00:00Z".parse().unwrap(),
        }
    }

    #[test]
    fn appends_without_truncating() {
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("responses.jsonl");
        append_record(&path, &rec("a")).unwrap();
        append_record(&path, &rec("b")).unwrap();
        let contents = std::fs::read_to_string(&path).unwrap();
        assert_eq!(contents.lines().count(), 2, "second append must not overwrite the first");
        // each line is independently valid JSON
        for line in contents.lines() {
            serde_json::from_str::<ResponseRecord>(line).unwrap();
        }
    }
}
}

• Step 4: Write lib.rs

crates/panoptes-harness/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod client;
pub mod jsonl;
pub use client::{ModelClient, ModelResponse};
pub use jsonl::append_record;
}

• Step 5: Run tests to verify they pass

Run: cargo test -p panoptes-harness jsonl Expected: PASS, 1 test. appends_without_truncating guards the append-only invariant.

• Step 6: Commit

git add crates/panoptes-harness
git commit -m "feat(harness): ModelClient trait + append-only JSONL writer"

Task 8: Anthropic client against a mock server (Stage 2b)

Files:

• Create: crates/panoptes-harness/src/anthropic.rs

• Modify: crates/panoptes-harness/src/lib.rs

• Step 1: Write the failing test using wiremock

crates/panoptes-harness/src/anthropic.rs:

#![allow(unused)]
fn main() {
use crate::client::{ModelClient, ModelResponse};
use async_trait::async_trait;
use panoptes_core::Usage;
use serde::Deserialize;

pub struct AnthropicClient {
    pub api_key: String,
    pub model: String,
    pub base_url: String, // injectable so tests can point at a mock
    http: reqwest::Client,
}

impl AnthropicClient {
    pub fn new(api_key: String, model: String, base_url: String) -> Self {
        Self { api_key, model, base_url, http: reqwest::Client::new() }
    }
}

#[derive(Deserialize)]
struct RawUsage { input_tokens: u32, output_tokens: u32 }
#[derive(Deserialize)]
struct RawBlock { text: String }
#[derive(Deserialize)]
struct RawResp { content: Vec<RawBlock>, usage: RawUsage }

#[async_trait]
impl ModelClient for AnthropicClient {
    fn model_name(&self) -> &str { &self.model }

    async fn generate(&self, prompt: &str) -> anyhow::Result<ModelResponse> {
        let raw: RawResp = self.http
            .post(format!("{}/v1/messages", self.base_url))
            .header("x-api-key", &self.api_key)
            .header("anthropic-version", "2023-06-01")
            .json(&serde_json::json!({
                "model": self.model, "max_tokens": 1024,
                "messages": [{"role": "user", "content": prompt}]
            }))
            .send().await?
            .error_for_status()?
            .json().await?;
        let text = raw.content.into_iter().map(|b| b.text).collect::<Vec<_>>().join("");
        Ok(ModelResponse { text, usage: Usage {
            input_tokens: raw.usage.input_tokens, output_tokens: raw.usage.output_tokens } })
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use wiremock::matchers::{method, path};
    use wiremock::{Mock, MockServer, ResponseTemplate};

    #[tokio::test]
    async fn parses_a_mocked_completion() {
        let server = MockServer::start().await;
        Mock::given(method("POST")).and(path("/v1/messages"))
            .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
                "content": [{"type": "text", "text": "Recommend SENSOR_RETASK."}],
                "usage": {"input_tokens": 42, "output_tokens": 7}
            })))
            .mount(&server).await;

        let client = AnthropicClient::new("k".into(), "claude-x".into(), server.uri());
        let resp = client.generate("test prompt").await.unwrap();
        assert_eq!(resp.text, "Recommend SENSOR_RETASK.");
        assert_eq!(resp.usage.input_tokens, 42);
        assert_eq!(resp.usage.output_tokens, 7);
    }
}
}

• Step 2: Re-export from lib.rs

Append to crates/panoptes-harness/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod anthropic;
pub use anthropic::AnthropicClient;
}

• Step 3: Run the test to verify it passes

Run: cargo test -p panoptes-harness anthropic Expected: PASS, 1 test. The mock proves parsing without spending API credits. Adding OpenAI/other providers = copy this file, change the request/response shapes.

• Step 4: Commit

git add crates/panoptes-harness
git commit -m "feat(harness): Anthropic client with mock-server test"

Task 9: Dispatch loop over manifest × models × epochs (Stage 2c)

Files:

• Create: crates/panoptes-harness/src/dispatch.rs

• Create: crates/panoptes-harness/src/main.rs

• Modify: crates/panoptes-harness/Cargo.toml (add [[bin]])

• Modify: crates/panoptes-harness/src/lib.rs

• Step 1: Write the failing test for opaque response IDs

crates/panoptes-harness/src/dispatch.rs:

#![allow(unused)]
fn main() {
use panoptes_core::{ResponseRecord, Vignette};
use crate::client::ModelClient;
use crate::jsonl::append_record;
use chrono::Utc;
use sha2::{Digest, Sha256};
use std::path::Path;

/// Opaque, deterministic response id — coders must not be able to read the
/// model or parameters off it (blinding). Hash of (vignette_id, model, epoch).
pub fn response_id(vignette_id: &str, model: &str, epoch: u32) -> String {
    let mut h = Sha256::new();
    h.update(vignette_id.as_bytes());
    h.update(model.as_bytes());
    h.update(epoch.to_le_bytes());
    format!("{:x}", h.finalize())[..16].to_string()
}

pub async fn dispatch(
    vignettes: &[Vignette],
    clients: &[Box<dyn ModelClient>],
    epochs: u32,
    log_path: &Path,
) -> anyhow::Result<usize> {
    let mut count = 0;
    for v in vignettes {
        for client in clients {
            for epoch in 0..epochs {
                let resp = client.generate(&v.prompt).await?;
                append_record(log_path, &ResponseRecord {
                    response_id: response_id(&v.id, client.model_name(), epoch),
                    vignette_id: v.id.clone(), model: client.model_name().to_string(),
                    epoch, params: v.params, prompt: v.prompt.clone(),
                    response: resp.text, usage: resp.usage, run_at: Utc::now(),
                })?;
                count += 1;
            }
        }
    }
    Ok(count)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn response_id_is_deterministic_and_opaque() {
        let a = response_id("ca_geo-030-H-REV-INFO", "claude-x", 0);
        let b = response_id("ca_geo-030-H-REV-INFO", "claude-x", 0);
        assert_eq!(a, b, "same inputs → same id");
        assert_ne!(a, response_id("ca_geo-030-H-REV-INFO", "claude-x", 1), "epoch changes id");
        assert!(!a.contains("claude"), "model name must not leak into the id");
        assert!(!a.contains("030"), "params must not leak into the id");
        assert_eq!(a.len(), 16);
    }
}
}

• Step 2: Write an integration test for the full loop against a mock

Append to the tests module in dispatch.rs:

#![allow(unused)]
fn main() {
    use crate::anthropic::AnthropicClient;
    use panoptes_core::params::{Params, Reversibility, TimePressure};
    use wiremock::matchers::method;
    use wiremock::{Mock, MockServer, ResponseTemplate};

    fn vig(id: &str) -> Vignette {
        Vignette { id: id.into(), family: "ca_geo".into(),
            params: Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
                reversibility: Reversibility::Reversible, info_request: true },
            prompt: "p".into(), prompt_sha256: "x".into() }
    }

    #[tokio::test]
    async fn dispatch_writes_one_record_per_call() {
        let server = MockServer::start().await;
        Mock::given(method("POST")).respond_with(ResponseTemplate::new(200).set_body_json(
            serde_json::json!({"content":[{"type":"text","text":"ok"}],
                "usage":{"input_tokens":1,"output_tokens":1}})))
            .mount(&server).await;

        let clients: Vec<Box<dyn ModelClient>> = vec![
            Box::new(AnthropicClient::new("k".into(), "claude-x".into(), server.uri())),
        ];
        let vignettes = vec![vig("a"), vig("b")];
        let dir = tempfile::tempdir().unwrap();
        let log = dir.path().join("responses.jsonl");

        // 2 vignettes × 1 client × 3 epochs = 6 records
        let n = dispatch(&vignettes, &clients, 3, &log).await.unwrap();
        assert_eq!(n, 6);
        assert_eq!(std::fs::read_to_string(&log).unwrap().lines().count(), 6);
    }
}

• Step 3: Write the run binary

crates/panoptes-harness/src/main.rs:

use anyhow::Result;
use clap::Parser;
use panoptes_core::Vignette;
use panoptes_harness::{dispatch::dispatch, AnthropicClient, ModelClient};
use std::fs;
use std::path::PathBuf;

#[derive(Parser)]
struct Cli {
    /// Directory containing manifest.csv + prompts/ (from panoptes-gen)
    #[arg(long, default_value = "scenarios/generated")]
    scenarios: PathBuf,
    /// Append-only log path
    #[arg(long, default_value = "harness/logs/responses.jsonl")]
    log: PathBuf,
    #[arg(long, default_value_t = 5)]
    epochs: u32,
}

fn load_vignettes(dir: &std::path::Path) -> Result<Vec<Vignette>> {
    // Reconstruct vignettes from manifest + prompt files.
    // (Parsing left as a small CSV read; the manifest schema is fixed in Task 6.)
    let manifest = fs::read_to_string(dir.join("manifest.csv"))?;
    let mut out = Vec::new();
    for line in manifest.lines().skip(1) {
        let cols: Vec<&str> = line.split(',').collect();
        let id = cols[0].to_string();
        let prompt = fs::read_to_string(dir.join("prompts").join(format!("{id}.txt")))?;
        out.push(Vignette {
            id: id.clone(), family: cols[1].to_string(),
            params: panoptes_core::params::Params {
                attribution_confidence: cols[2].parse()?,
                time_pressure: cols[3].parse().map_err(|e| anyhow::anyhow!("{e}"))?,
                reversibility: cols[4].parse().map_err(|e| anyhow::anyhow!("{e}"))?,
                info_request: cols[5].parse()?,
            },
            prompt, prompt_sha256: cols[6].to_string(),
        });
    }
    Ok(out)
}

#[tokio::main]
async fn main() -> Result<()> {
    let cli = Cli::parse();
    fs::create_dir_all(cli.log.parent().unwrap())?;
    let vignettes = load_vignettes(&cli.scenarios)?;

    // Pinned model list — final list is Open Question #6. base_url from env for real runs.
    let base = std::env::var("ANTHROPIC_BASE_URL").unwrap_or("https://api.anthropic.com".into());
    let key = std::env::var("ANTHROPIC_API_KEY").unwrap_or_default();
    let clients: Vec<Box<dyn ModelClient>> = vec![
        Box::new(AnthropicClient::new(key, "claude-x-pinned".into(), base)),
    ];

    let n = dispatch(&vignettes, &clients, cli.epochs, &cli.log).await?;
    eprintln!("wrote {n} response records → {}", cli.log.display());
    Ok(())
}

• Step 4: Declare the binary + re-export dispatch

Append to crates/panoptes-harness/Cargo.toml:

[[bin]]
name = "panoptes-run"
path = "src/main.rs"

Append to crates/panoptes-harness/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod dispatch;
pub use dispatch::{dispatch, response_id};
}

• Step 5: Run all harness tests

Run: cargo test -p panoptes-harness Expected: PASS — response_id_is_deterministic_and_opaque and dispatch_writes_one_record_per_call are the critical two.

• Step 6: Commit

git add crates/panoptes-harness
git commit -m "feat(harness): dispatch loop + run binary over manifest"

Task 10: Blinded coding sheets + blind key (Stage 3a)

Files:

• Create: crates/panoptes-coding/Cargo.toml

• Create: crates/panoptes-coding/src/lib.rs

• Create: crates/panoptes-coding/src/sheets.rs

• Step 1: Create the coding crate manifest

crates/panoptes-coding/Cargo.toml:

[package]
name = "panoptes-coding"
version = "0.1.0"
edition = "2021"

[dependencies]
panoptes-core = { path = "../panoptes-core" }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }   # CodingError derive in validate.rs (Task 11)
clap = { workspace = true }
anyhow = { workspace = true }

[dev-dependencies]
tempfile = "3"

• Step 2: Write the failing test — sheet must not leak model or params

crates/panoptes-coding/src/sheets.rs:

#![allow(unused)]
fn main() {
use panoptes_core::ResponseRecord;
use std::collections::BTreeMap;

/// A blank coding row: response_id + the response text, and nothing that
/// reveals which model produced it or what the parameters were.
#[derive(Debug, Clone, serde::Serialize)]
pub struct BlankSheetRow {
    pub response_id: String,
    pub response: String,
    // coder fills these; empty in the blank
    pub c1_action: String,
    pub c2_logic: String,
    pub c2_anchor_quote: String,
    pub c3_escalation: String,
    pub codebook_version: String,
    pub coder_notes: String,
}

/// The blind key, kept separate and NOT given to coders.
#[derive(Debug, Clone, serde::Serialize)]
pub struct KeyRow {
    pub response_id: String,
    pub vignette_id: String,
    pub model: String,
    pub epoch: u32,
}

pub fn make_sheet(records: &[ResponseRecord]) -> (Vec<BlankSheetRow>, Vec<KeyRow>) {
    let mut sheet = Vec::new();
    let mut key = Vec::new();
    for r in records {
        sheet.push(BlankSheetRow {
            response_id: r.response_id.clone(), response: r.response.clone(),
            c1_action: String::new(), c2_logic: String::new(),
            c2_anchor_quote: String::new(), c3_escalation: String::new(),
            codebook_version: String::new(), coder_notes: String::new(),
        });
        key.push(KeyRow {
            response_id: r.response_id.clone(), vignette_id: r.vignette_id.clone(),
            model: r.model.clone(), epoch: r.epoch,
        });
    }
    (sheet, key)
}

/// Deterministic shuffle by seed so ordering can't be reconstructed but runs
/// are reproducible. Uses a simple index permutation keyed on the seed.
pub fn shuffled_indices(n: usize, seed: u64) -> Vec<usize> {
    // Fisher–Yates with a tiny LCG — no external rand dependency.
    let mut idx: Vec<usize> = (0..n).collect();
    let mut state = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
    for i in (1..n).rev() {
        state = state.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
        let j = (state >> 33) as usize % (i + 1);
        idx.swap(i, j);
    }
    idx
}

#[cfg(test)]
mod tests {
    use super::*;
    use panoptes_core::params::{Params, Reversibility, TimePressure};
    use panoptes_core::Usage;

    fn rec(id: &str, model: &str) -> ResponseRecord {
        ResponseRecord {
            response_id: id.into(), vignette_id: "ca_geo-030-H-REV-INFO".into(),
            model: model.into(), epoch: 0,
            params: Params { attribution_confidence: 30, time_pressure: TimePressure::Hours,
                reversibility: Reversibility::Reversible, info_request: true },
            prompt: "p".into(), response: "Recommend MONITOR.".into(),
            usage: Usage { input_tokens: 1, output_tokens: 1 },
            run_at: "2026-07-17T12:00:00Z".parse().unwrap(),
        }
    }

    #[test]
    fn sheet_does_not_leak_identity() {
        let (sheet, key) = make_sheet(&[rec("r1", "anthropic/claude-x")]);
        let serialized = serde_json::to_string(&sheet).unwrap();
        assert!(!serialized.contains("claude"), "model must not appear on the sheet");
        assert!(!serialized.contains("ca_geo"), "vignette id must not appear on the sheet");
        assert!(!serialized.contains("30"), "params must not appear on the sheet");
        // but the key retains the mapping
        assert_eq!(key[0].model, "anthropic/claude-x");
        assert_eq!(key[0].vignette_id, "ca_geo-030-H-REV-INFO");
    }

    #[test]
    fn shuffle_is_deterministic_and_a_permutation() {
        let a = shuffled_indices(18, 20261);
        let b = shuffled_indices(18, 20261);
        assert_eq!(a, b, "same seed → same order");
        let mut sorted = a.clone(); sorted.sort();
        assert_eq!(sorted, (0..18).collect::<Vec<_>>(), "must be a permutation of 0..n");
    }
}
}

• Step 3: Write lib.rs

crates/panoptes-coding/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod sheets;
pub use sheets::{make_sheet, shuffled_indices, BlankSheetRow, KeyRow};
}

• Step 4: Run tests to verify they pass

Run: cargo test -p panoptes-coding sheets Expected: PASS, 2 tests. sheet_does_not_leak_identity is the integrity guarantee for blind coding.

• Step 5: Commit

git add crates/panoptes-coding
git commit -m "feat(coding): blinded coding sheets + separate blind key"

Task 11: Coded-CSV loader — parse is validation (Stage 3b)

Files:

• Create: crates/panoptes-coding/src/validate.rs

• Create: crates/panoptes-coding/src/main.rs

• Modify: crates/panoptes-coding/Cargo.toml (add [[bin]])

• Modify: crates/panoptes-coding/src/lib.rs

• Step 1: Write the failing test — bad codebook value fails loudly with row context

crates/panoptes-coding/src/validate.rs:

#![allow(unused)]
fn main() {
use panoptes_core::CodedRow;

#[derive(Debug, thiserror::Error)]
#[error("row {line}: {source}")]
pub struct CodingError {
    pub line: usize,
    #[source]
    pub source: serde_json::Error,
}

/// Load coded rows from a JSON-lines file. Each line deserializes into a
/// CodedRow, so any value outside the codebook enums fails here — the
/// validation IS the parse. Empty anchor quotes for latent codes are caught
/// by the check below.
pub fn load_coded(contents: &str) -> Result<Vec<CodedRow>, CodingError> {
    let mut rows = Vec::new();
    for (i, line) in contents.lines().enumerate() {
        if line.trim().is_empty() { continue; }
        let row: CodedRow = serde_json::from_str(line)
            .map_err(|source| CodingError { line: i + 1, source })?;
        rows.push(row);
    }
    Ok(rows)
}

/// Latent codes (C2 strategic logic) must anchor to quoted text — the codebook rule.
pub fn check_anchors(rows: &[CodedRow]) -> Result<(), String> {
    for r in rows {
        if r.c2_anchor_quote.trim().is_empty() {
            return Err(format!("{}: C2 logic coded without an anchor quote", r.response_id));
        }
    }
    Ok(())
}

#[cfg(test)]
mod tests {
    use super::*;

    const VALID: &str = r#"{"response_id":"r1","c1_action":"SENSOR_RETASK","c2_logic":"CONTROL","c2_anchor_quote":"positional advantage","c3_escalation":0,"codebook_version":"0.3","coder_notes":""}"#;
    const BAD_ENUM: &str = r#"{"response_id":"r2","c1_action":"SENSOR_RETASK","c2_logic":"AGGRESSIVE","c2_anchor_quote":"x","c3_escalation":0,"codebook_version":"0.3","coder_notes":""}"#;
    const NO_ANCHOR: &str = r#"{"response_id":"r3","c1_action":"MONITOR","c2_logic":"CONTROL","c2_anchor_quote":"","c3_escalation":0,"codebook_version":"0.3","coder_notes":""}"#;

    #[test]
    fn valid_row_loads() {
        let rows = load_coded(VALID).unwrap();
        assert_eq!(rows.len(), 1);
        assert_eq!(rows[0].response_id, "r1");
    }

    #[test]
    fn bad_enum_fails_with_line_number() {
        let err = load_coded(BAD_ENUM).unwrap_err();
        assert_eq!(err.line, 1);
        // proves an out-of-codebook value can't slip through
    }

    #[test]
    fn missing_anchor_is_caught() {
        let rows = load_coded(NO_ANCHOR).unwrap();
        let err = check_anchors(&rows).unwrap_err();
        assert!(err.contains("r3"));
        assert!(err.contains("anchor"));
    }
}
}

• Step 2: Write the validation CLI

crates/panoptes-coding/src/main.rs:

use anyhow::Result;
use clap::Parser;
use panoptes_coding::validate::{check_anchors, load_coded};
use std::fs;
use std::path::PathBuf;

#[derive(Parser)]
struct Cli {
    /// Path to a coded JSON-lines file (primary or second coder)
    #[arg(long)]
    coded: PathBuf,
}

fn main() -> Result<()> {
    let cli = Cli::parse();
    let contents = fs::read_to_string(&cli.coded)?;
    let rows = load_coded(&contents).map_err(|e| anyhow::anyhow!("{e}"))?;
    check_anchors(&rows).map_err(|e| anyhow::anyhow!(e))?;
    eprintln!("OK: {} coded rows valid against the codebook", rows.len());
    Ok(())
}

• Step 3: Declare the binary + re-export

Append to crates/panoptes-coding/Cargo.toml:

[[bin]]
name = "panoptes-code"
path = "src/main.rs"

Append to crates/panoptes-coding/src/lib.rs:

#![allow(unused)]
fn main() {
pub mod validate;
pub use validate::{check_anchors, load_coded, CodingError};
}

• Step 4: Run tests, then exercise the CLI on a good and a bad file

Run: cargo test -p panoptes-coding validate Expected: PASS, 3 tests.

Create a temp bad file and confirm the CLI rejects it:

echo '{"response_id":"r2","c1_action":"SENSOR_RETASK","c2_logic":"AGGRESSIVE","c2_anchor_quote":"x","c3_escalation":0,"codebook_version":"0.3","coder_notes":""}' > /tmp/bad.jsonl
cargo run -p panoptes-coding -- --coded /tmp/bad.jsonl; echo "exit: $?"

Expected: non-zero exit with an error naming line 1 — the codebook constraint enforced at the boundary.

• Step 5: Commit

git add crates/panoptes-coding
git commit -m "feat(coding): coded-CSV loader where parsing enforces the codebook"

Task 12: Workspace lints, Stage-4 handoff stub, README

Files:

• Create: reliability/README.md (Python handoff contract — not implemented in Rust)

• Create: README.md (workspace overview + run sequence)

• Modify: root Cargo.toml (workspace lints)

• Step 1: Add workspace-wide lint config

Append to root Cargo.toml:

[workspace.lints.rust]
unused_must_use = "deny"

[workspace.lints.clippy]
unwrap_used = "warn"     # unwraps are fine in tests; flag them in library code

Add to each crate's Cargo.toml (below [package]):

[lints]
workspace = true

• Step 2: Write the Stage-4 handoff contract

reliability/README.md:

# Stage 4 — Reliability (Python, not Rust)

Reads:
- `coding/coded/primary.csv`   (validated by `panoptes-code`)
- `coding/coded/second_coder.csv`
- `coding/blind_key.csv`       (to join response_id → vignette_id)

Produces:
- `reliability/irr_report.csv`  — per-criterion Cohen's κ / Krippendorff's α
- `reliability/disagreements_<criterion>.csv`

Why Python: `krippendorff` and `scikit-learn`'s `cohen_kappa_score` are correct,
maintained, and not worth reimplementing. C3 escalation uses ordinal α.
Stratify the reliability subsample by `family` (join via blind_key) so no
scenario type escapes validation. Freeze the codebook version only when every
criterion clears ~0.7 (0.8 comfort). This table is Ch3/Ch4 verbatim.

• Step 3: Write the workspace README with the run sequence

README.md:

# Panoptes Evaluation Harness

Stages 1–4 in Rust (typed, reproducible); Stages 5–6 in Python (stats + analysis),
reading the file contract this workspace produces.

## Run sequence
1. Generate:  `cargo run -p panoptes-gen -- --family scenarios/families/ca_geo.toml`
2. Execute:   `ANTHROPIC_API_KEY=... cargo run -p panoptes-run -- --epochs 5`
3. Sheets:    `cargo run -p panoptes-code -- ...`  (blank sheets → human codes → validate)
4. Validate:  `cargo run -p panoptes-code -- --coded coding/coded/primary.csv`
5–6. Python analysis reads harness/logs/responses.jsonl + coding/coded/*.csv

## Invariants
- responses.jsonl is append-only (the dataset of record)
- coding sheets never contain model identity or parameters (blinding)
- codebook values are enums — invalid values fail to parse, not lint
- the manifest is the join spine: analysis joins on vignette_id / response_id

• Step 4: Full workspace build + test + clippy

Run: cargo build --workspace && cargo test --workspace && cargo clippy --workspace Expected: builds clean, all tests pass, clippy warns only on any intentional library unwraps.

• Step 5: Commit

git add Cargo.toml crates README.md reliability/README.md
git commit -m "chore: workspace lints, Stage-4 handoff contract, README"

Self-Review

Spec coverage. Every stage from the architecture is implemented: Stage 1 generation (Tasks 4–6), Stage 2 execution (Tasks 7–9), Stage 3 coding (Tasks 10–11), with the shared data model front-loaded (Tasks 1–3) so types are defined once. Stage 4 is deliberately a Python handoff contract (Task 12), matching the decision to keep reliability statistics in Python. The three design invariants from earlier sessions each have a guarding test: append-only (appends_without_truncating), blinding (sheet_does_not_leak_identity, response_id_is_deterministic_and_opaque), and codebook-as-type (invalid_logic_is_rejected, bad_enum_fails_with_line_number).

Type consistency. Params, Vignette, ResponseRecord, CodedRow, and the enums are defined once in panoptes-core and imported everywhere else. vignette_id (core) and response_id (harness) are distinct by design: the former is human-readable and encodes parameters; the latter is opaque and hides them, which is the blinding property. model_name() on ModelClient is the single source of the pinned model string used in both dispatch and records.

Known simplifications to harden during the build, not blockers. The manifest CSV is written and parsed by hand (Tasks 6, 9) rather than via a csv crate — fine at this scale, but swap to the csv crate if quoting/escaping in prompts ever leaks into manifest fields (prompts live in separate files, so this is low-risk). The Anthropic request/response shapes (Task 8) are current-API-shaped but must be checked against live docs at build time. Provider list in main (Task 9) is a placeholder pending Open Question #6 (final pinned model list).

Appendix: Workspace Scaffold

This is the orientation map for the harness workspace you are building (a separate repo from this book). It answers three questions the build chapters assume: where does each file go, what is it called, and which dependencies does it need? Every build chapter also carries its own Scaffold block; this page is the whole picture in one place.

The harness workspace is its own project directory (e.g. ~/projects/rust/panoptes/) — do not create it inside this course's repo.

The full tree

Everything you create across the course, annotated with the chapter that creates it:

panoptes/
├── Cargo.toml                         # workspace manifest        (Part II · build-params)
├── crates/
│   ├── panoptes-core/                 # THE DATA MODEL — no I/O, no network, pure types
│   │   ├── Cargo.toml                 #                           (Part II · build-params)
│   │   └── src/
│   │       ├── lib.rs                 # re-exports                (Part II, grows each chapter)
│   │       ├── params.rs              # Params + parameter enums  (Part II · build-params)
│   │       ├── codes.rs               # codebook enums + CodedRow (Part II · build-codes)
│   │       ├── vignette.rs            # Vignette, vignette_id     (Part II · build-records)
│   │       └── record.rs              # ResponseRecord, Usage     (Part II · build-records)
│   ├── panoptes-gen/                  # STAGE 1 — generation
│   │   ├── Cargo.toml                 #                           (Part III · build-family)
│   │   └── src/
│   │       ├── lib.rs                 #                           (Part III · build-family)
│   │       ├── family.rs              # FamilySpec (TOML shape)   (Part III · build-family)
│   │       ├── validity.rs            # ScenarioFamily trait      (Part III · build-family)
│   │       ├── generate.rs            # grid → Vec<Vignette>      (Part III · build-generate)
│   │       └── main.rs                # `panoptes-gen` binary     (Part III · build-cli)
│   ├── panoptes-harness/              # STAGE 2 — execution
│   │   ├── Cargo.toml                 #                           (Part IV · build-client-log)
│   │   └── src/
│   │       ├── lib.rs                 #                           (Part IV · build-client-log)
│   │       ├── client.rs              # ModelClient trait         (Part IV · build-client-log)
│   │       ├── jsonl.rs               # append-only writer        (Part IV · build-client-log)
│   │       ├── anthropic.rs           # one provider impl         (Part IV · build-anthropic)
│   │       ├── dispatch.rs            # vignette×model×epoch loop (Part IV · build-dispatch)
│   │       └── main.rs                # `panoptes-run` binary     (Part IV · build-dispatch)
│   ├── panoptes-coding/               # STAGE 3 — coding I/O
│   │   ├── Cargo.toml                 #                           (Part V · build-sheets)
│   │   └── src/
│   │       ├── lib.rs                 #                           (Part V · build-sheets)
│   │       ├── sheets.rs              # blinded sheets + key      (Part V · build-sheets)
│   │       ├── validate.rs            # coded loader = validation (Part V · build-loader)
│   │       └── main.rs                # `panoptes-code` binary    (Part V · build-loader)
│   └── panoptes-cli/                  # the unified front door    (Part VII · build-cli)
│       ├── Cargo.toml
│       ├── src/main.rs                # `panoptes` subcommand enum
│       └── tests/pipeline.rs          # assert_cmd integration test
├── scenarios/
│   ├── families/ca_geo.toml           # content + metadata only   (Part III · build-family)
│   └── generated/                     # OUTPUT: manifest.csv + prompts/  (gitignore this)
├── harness/logs/                      # OUTPUT: responses.jsonl — append-only, sacred
├── coding/                            # OUTPUT: sheets/, coded/, blind_key.csv
├── reliability/README.md              # Stage-4 Python handoff    (Part VI · build-hardening)
└── README.md                          # run sequence + invariants (Part VI · build-hardening)

Build order matches the parts: core first (everything imports it), then gen → harness → coding in pipeline order, then the unified CLI on top.

The workspace manifest — every dependency, declared once

The root Cargo.toml you write in the very first build chapter declares all shared dependencies under [workspace.dependencies]. Later crate manifests then just write dep = { workspace = true } — so if a chapter seems to use a crate "out of nowhere" (strum, serde_with, …), it was declared here on day one:

[workspace]
resolver = "2"
members = [
    "crates/panoptes-core",
    "crates/panoptes-gen",
    "crates/panoptes-harness",
    "crates/panoptes-coding",
]

[workspace.dependencies]
serde = { version = "1", features = ["derive"] }   # derive Serialize/Deserialize (everywhere)
serde_json = "1"        # JSON + JSONL encoding (records, tests)
toml = "1"              # family-spec parsing (Part III)
serde_with = "3"        # NoneAsEmptyString for optional free-text (Part II codes)
strum = { version = "0.26", features = ["derive"] } # enum ↔ string Display/EnumString (Part II)
tera = "1"              # prompt templating (Part III)
reqwest = { version = "0.12", features = ["json"] } # HTTP client (Part IV)
tokio = { version = "1", features = ["full"] }      # async runtime (Part IV)
async-trait = "0.1"     # async fn in the ModelClient trait (Part IV)
sha2 = "0.10"           # prompt hashing (Part III) + opaque response ids (Part IV)
chrono = { version = "0.4", features = ["serde"] }  # run_at timestamps (Part II records)
itertools = "0.13"      # iproduct! cartesian grid (Part III)
clap = { version = "4", features = ["derive"] }     # every CLI binary
anyhow = "1"            # application-level errors
thiserror = "2"         # typed library errors (Part V loader)

(Add "crates/panoptes-cli" to members when you reach Part VII.)

Who depends on what

The two bold entries are corrections to the original task plan, which omitted them: dispatch.rs (Part IV) hashes response ids with sha2, and validate.rs (Part V) derives its error type with thiserror. Declare them when you create each crate's manifest and Parts IV–V will compile without surprises.

wiremock = "0.6", tempfile = "3", and assert_cmd are dev-dependencies declared per-crate (not in the workspace block).

Expected test progression

A quick reality check for the end of each part — if your counts differ, look for a missed test, not a missed feature:

Appendix: Reference Books

This course is grounded in five books. Where a chapter anchors a concept to a specific book, it cites it inline. This appendix says what each one is for, so you know where to go deeper.

Async Rust — Maxwell Flitton & Caroline Morton (O'Reilly)

The primary source for Part IV. Its treatment of the future lifecycle (idle → polled → Pending/Ready), the runtime's polling loop, and the concurrency model underpins the async concept chapters. If any async idea in this course feels thin, this is the book to open — its chapter on futures, pinning, and context goes deeper than a working harness strictly needs, which is exactly why it is the right reference when the compiler surprises you.

Command-Line Rust — Ken Youens-Clark (O'Reilly)

The source for Part VII. Its early chapters model exactly what the CLI arc does: building tools with clap, writing integration tests that run the compiled binary, and — the idea that matters most — exit codes and composability ("Exit Values Make Programs Composable"). The book's discipline around honest exit status is what turns panoptes from three scripts into a composable tool.

Rust for Rustaceans — Jon Gjengset (No Starch)

The source for the Foundations arc (ownership, moves, borrowing) and a best-practices reference throughout. Its Chapter 1 "Foundations" gives the flows mental model for ownership and lifetimes that the course teaches, and its intermediate-idioms chapters inform trait and API design decisions. This is the book to grow into after the course.

Effective Rust — David Drysdale (O'Reilly)

A best-practices reference, organized as discrete numbered "items" (in the tradition of Effective C++). Consulted for idiomatic choices around the type system, error handling, and API design. Where a design decision in the harness has an idiomatic "right answer," this book usually has an item on it.

AI Engineering — Chip Huyen (O'Reilly)

Foundational knowledge for the evaluation methodology the harness serves, and the broader thread connecting Panoptes to the thesis and to Norion. It sharpens the wrap-up's framing of what a capability benchmark is and why the codebook, reliability, and analysis stages are shaped the way they are. Read it alongside the eval-engineering canon (Husain, Yan, Shankar) for the discipline this harness is an instance of.