The full technical stack behind the Panoptes SDA decision benchmark and its bounded RLVR demonstration: three Rust components, one evidentiary file boundary, one uniform model plane, and a deliberately thin Python statistical tail.
The suite is three Rust components joined by two clean interfaces. panoptes_etl supplies ground truth: Source-trait extractors pull public feeds, a typed transform normalizes them, and content-addressed loads plus a run ledger make every artifact's provenance a checkable claim. panoptes (the harness, the Inspect analog) generates parameterized vignettes from that grounded data, executes them against version-pinned models with clean-context calls, logs every raw exchange, and hosts the typed, blinded coding stage. The distributed API backend serves the open benchmark after the December gate, and not before.
The first interface is the file boundary: Rust is the sole writer of the JSONL evidentiary record and the tidy Parquet summaries; Python only reads. The second is the model plane: local open weights and frontier APIs both present as pinned REST endpoints, so a fine-tuned checkpoint re-enters evaluation as just another subject model, through the identical path as every baseline. No special evaluation code exists for the trained model, which is itself a validity argument.
Model plane edges are directed: models feed Execute, and Execute alone writes the log. The checkpoint's re-entry arrow (16 → 7a) closes the eval-to-reward loop.
Open, no-auth GP element sets over plain GET. Bulk catalog coverage. TLE-derived elements carry kilometer-class accuracy: fine for grounding scenario realism, indefensible for claiming real conjunction-screening fidelity. Ch3 says exactly that in one sentence.
Simplest Source implThe authoritative 18 SDS catalog, and the source that forces auth, cookie sessions, secrets, and rate limits. Its user agreement restricts redistribution: raw pulls stay local in the content-addressed store; the open benchmark ships derived synthetic scenarios plus provenance references, never raw rows.
Firewall-adjacent · check agreement before releaseLaunch and event context for enrichment and attribution: who launched what, operated by whom. Supplies the attribution ambiguity the vignettes turn into decision pressure. Lowest criticality of the three feeds; ca_geo stands on 1a and 1b if this slips.
Pagination + backoff arcThe one input that isn't code. Version-pinned texts (JP 3-14, SCP 2020, SWF Apr 2025, SDP 3-100, SDP 2-0) plus Dolman, Klein, Bowen. Flows into exactly two places, the codebook (9) and the rewards (14), and that both consume the same artifact is the thesis's closed-loop novelty.
Human intellectual laborThe Source trait: every feed is a uniform async fn, so a new source is a new impl and nothing else changes. That seam makes the NASA and RAG arcs additive rather than rewrites. Scientifically, extraction is where "what was pulled, from where, when" becomes recorded fact.
Append-only record of every ETL run: source, watermark, timestamp, outcome. A Ch3 reproducibility sentence made mechanical, and the debugging tool when a feed silently changes shape. Kept boring: a table or JSONL file, never a service.
Typed normalization: NORAD IDs as newtypes, epochs parsed not stringly-typed, malformed records rejected at the boundary with rejections logged. Nothing downstream ever defensively re-validates, which keeps 6 and beyond simple.
Idempotent, content-addressed writes: re-running is always safe, and every artifact's hash is its identity. With 2a, this lets you assert "scenario 0451 derives from CDM pull a3f9… of 12 Sep 2026" as a checkable claim, worth more at defense than any framework feature.
The interface between the two Rust codebases. The keystone insight: a real CDM is a collision-avoidance decision waiting to be posed; objects and elements exist to enrich and attribute conjunctions. ITAR discipline lives here: real public behavior grounds parameters, published assets stay generic.
Templates crossed with a parameter grid, seeded deterministically. Parameterization makes Panoptes a benchmark rather than a quiz: ambiguity, attribution confidence, time pressure, and severity become controlled experimental knobs, which lets Ch4 report tendencies across conditions instead of trivia scores.
Async tokio dispatch against raw REST, pinned model strings, logged temperatures and replication counts, clean-context calls carrying only the scenario prompt. That last property is the advisor-subject conflict mitigation in mechanical form. Full-matrix runs are deliberate, budgeted events.
The 5080 serving open weights (14B dense and under, gpt-oss-20b) behind an OpenAI-compatible endpoint, indistinguishable from remote APIs to the provider client. Open-weights anchors are deprecation insurance: the benchmark stays re-runnable regardless of what vendors retire.
Frontier references and judge candidates, pinned by version string. Retirement risk is accepted and mitigated by 8a's exhaustive logging: even after a model dies, its recorded behavior remains analyzable. Budget line item per open question 6.
One serialized run struct per line: prompt, raw response, parameters, dates. The single interface between Rust and Python and the immutable evidentiary record. The harness is the sole writer; models feed Execute, and Execute alone writes the log. The one artifact to protect with your life.
NVMe carries active models, datasets, in-flight checkpoints; the 5TB NAS archives finished runs, unused quants, completed checkpoints via the sweep. Upgrade triggers as recorded: RAM at swap-hit, second NVMe when the sweep costs time, 2.5GbE if transfers bottleneck.
The typed, blinded coding stage and the harness's defense-day property: a coder is structurally unable to record a value outside the codebook, because it fails to parse. Blinding hides model identity from coders, killing brand bias. This is where 1d's criteria become operational measurements.
Pilot responses coded independently by Trevor and the second coder. The September critical-path item: the coder must be named and warm before pilot data exists, and the subsample must span scenario families and the score range. Everything downstream rests on this artifact.
Critical path · Sep 2026Rust owns everything through data shaping: the harness emits tidy Parquet summaries, scores per criterion per run, already joined. Python shrinks to a script that reads clean tables. The natural extension of the typed pipeline, and the agreed language boundary.
Krippendorff's alpha on the double-coded subsample, 0.67 floor, computed with the vetted Python package, twenty lines. Below floor means the criteria are ambiguous, not that the coders are bad, and the mandated response is codebook revision before anything proceeds.
October gate conditionThe judge is a version-pinned experimental subject calibrated against 10: agreement statistics plus position, verbosity, and self-enhancement bias checks. Only a judge that agrees with validated humans earns the right to score the full matrix and feed 15. Claude is excluded from this role; disclosed in Ch3.
The calibrated judge scores the full run matrix, producing Ch4's middle act. Ch4's order is fixed: validation results first because everything rests on them, then model comparisons as tendencies across the grid, then the RLVR delta.
The codebook decomposed into weighted checklist criteria, RaR-style, never a single scalar, because decomposition is the first defense against reward hacking. Standing job applies in full force: every reward criterion gets challenged for hackability the way codebook criteria get challenged for ambiguity.
One policy model at or under 4B, QLoRA-based, comfortably inside 16GB, standard Unsloth configuration, no bespoke RL infrastructure. Pre-registered success: the method improves codebook-scored reasoning and/or the eval detects reward hacking, so flat or negative results are findings. Instability at this scale is expected and cited.
The elegant closure: the trained model re-enters through 7a as just another pinned subject, evaluated on held-out items through the identical path as every baseline. The released artifact stays small, synthetic-only, and deliberately licensed, since model outputs don't inherit published status.
The open benchmark's delivery vehicle and the business-track on-ramp, backend only. It consumes 13's results and 5's scenario repository; it contributes nothing to any research question before December, making it the diagram's designated plumbing-trap candidate. It stays a stub until the gate is behind you.
Plumbing-trap watch · post-Dec 2026The consistency argument for all-Rust analysis has a hole that can't be closed: Unsloth is Python, so the GRPO stage keeps Python in the stack regardless. More importantly, the analysis layer is where validity-critical statistics live. Krippendorff's alpha has no vetted Rust implementation; hand-rolling it invites the exact defense question you don't want, and golden-testing a Rust port against reference implementations means Python re-enters as the test oracle anyway. Numbers that land in Ch4 tables come from implementations the field already trusts.
Stages 1 through 8 are pure engineering, buildable any time. Stages 9 through 12 form the validity spine: the only stages with a human dependency (the second coder, September) and a hard gate attached (pilot IRR at the 0.67 floor, October). The diagram's risk isn't distributed left to right; it's concentrated in that one vertical. Protect it from harness and pipeline tinkering, and from stage 17 entirely.