Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Build: The Dispatch Loop + Run Binary

Maps to: Task 9. Kind: Build.

Objective

Write dispatch.rs: the opaque response_id function and the dispatch loop over vignettes × clients × epochs, writing one ResponseRecord per call. Then the panoptes-run binary that loads the manifest, builds the pinned client list, and runs the loop. Test the ID's opacity and the loop's record count against a mock.

Scaffold

Create: crates/panoptes-harness/src/dispatch.rs and src/main.rs. Modify: crates/panoptes-harness/Cargo.toml (add [[bin]] name = "panoptes-run" path = "src/main.rs") and src/lib.rs (re-export dispatch, response_id).

Dependencies: sha2 for the response-id hash — already in the manifest if you followed the earlier Scaffold note (the original task plan forgets it). chrono supplies Utc::now(); #[tokio::main] runs the binary.

Expected result: cargo test -p panoptes-harness4 tests pass crate-wide; dispatch_writes_one_record_per_call proves 2 vignettes × 1 client × 3 epochs → 6 records.

The spec (givens)

  • response_id(vignette_id: &str, model: &str, epoch: u32) -> String: the first 16 hex chars of SHA-256 over the vignette id bytes, then the model bytes, then epoch.to_le_bytes().
  • dispatch(vignettes: &[Vignette], clients: &[Box<dyn ModelClient>], epochs: u32, log_path: &Path) -> anyhow::Result<usize> — loop order vignette → client → epoch, one append_record per call, returns the count.
  • Binary CLI args: --scenarios (default "scenarios/generated"), --log (default "harness/logs/responses.jsonl"), --epochs (default 5). Real runs read ANTHROPIC_API_KEY and optional ANTHROPIC_BASE_URL from the environment.

Full code: Answer Key, Task 9.

Concepts exercised

  • Iterating over &[Box<dyn ModelClient>] and calling an async trait method.
  • Deterministic-but-opaque ID construction (the blinding property, in the harness).
  • An integration test that drives the whole loop against a mock server.

The build loop (you drive)

  1. Write failing tests: response_id_is_deterministic_and_opaque (same inputs → same id; epoch changes it; model name and params do NOT appear in it); dispatch_writes_one_record_per_call (2 vignettes × 1 client × 3 epochs → 6 records).
  2. Predict: why must response_id be a hash rather than a readable concatenation? What property would a readable id break?
  3. Run, check.
  4. Implement dispatch and the binary.
  5. Run green, commit.
Milestone Stage 2 is complete. You can generate scenarios and run them against pinned models, logging every raw response to an append-only store, with response IDs that keep coders blind. Combined with Stage 1, you now have a working evaluation pipeline through execution — the parts that produce the dataset of record.

Done when

cargo test -p panoptes-harness is fully green, including the opacity guard and the integration test.