# Ephemera β€” Adversarial cross-model review gate (external LLM CLIs) > Stand up a pre-merge **adversarial code review** that runs a change through two *external* agentic CLIs (a second and third opinion from vendors that don't share your primary model's blind spots), has your primary agent synthesize their findings, and **gates the merge** on a fresh passing verdict. Self-executing Markdown. The repo + its tests are the source of truth; this file = intent + install ledger + teardown audit. > **Provides / Requires** > - **Provides**: a `/adversarial-review` skill (on-demand) + a `PreToolUse` merge gate (blocking) + a reusable review engine, installed into a target repo. > - **Requires**: (a) a target repo with a **checkpoint/diff tool** (Arc *or* git); (b) at least **one external headless agentic CLI** on PATH (`agy`=Gemini, `grok`=xAI, `ollama`=local, `claude`, …); (c) `jq`; (d) a **project-invariants file** the reviewers are told to attack (your threat model / "crown jewels"). Discover each at `verify` time β€” don't assume. > **Engine-pattern sibling.** `sdd-fleet.local.md` reuses this plan's multi-CLI headless-fan-out engine to *build* slices from a spec β€” this plan reviews a diff; that one produces one. Same dispatch shape, opposite direction. --- ## πŸ€– Director Prompt (read first, every run) You are the Director. Execute this plan: - Observe-before-act on every step (does the file/hook already exist? is the CLI on PATH?) - Verify each step before advancing (a step's test must pass before the next) - Stop at every πŸ”΄ GATE and πŸ’₯ for a human "go" - Write realized values + verify results back into Live State - Teardown removes **only what this plan created** β€” never a pre-existing hook, test, or CI step - Use only the commands in this plan - On any failure, annotate Live State and stop ``` Legend 🟒 create Β· 🟑 config Β· πŸ”΄ GATE (human go) Β· πŸ’₯ destructive (human go) Β· ⏳ wait Β· βœ” verify ``` **The one non-negotiable invariant this whole plan defends: value-never-logged.** No secret VALUE may reach engine stdout, the gate's stderr, the verdict artifact, an error path, or the external models. Every component below is built to hold that line; the tests prove it; teardown doesn't weaken it. --- ## What you need, and why | Prerequisite | Why | How to tell you have it (`verify`) | |---|---|---| | A checkpoint/diff tool (Arc or git) | The engine resolves *what changed* to review; the gate keys freshness off a **content hash of the tree** | `arc --version` **or** `git rev-parse --show-toplevel` | | β‰₯1 external headless CLI | The independent reviewers. Headless = a `-p`/`--print`/`--single` mode that prints one response and exits | `command -v agy grok ollama claude` (need at least one) | | `jq` | JSON assembly/validation in the engine, gate, hooks | `command -v jq` | | A project-invariants file | The reviewers must be told *what to attack* β€” your security/correctness invariants. Generic prompts find generic nothing | you point `$INVARIANTS_FILE` at it (e.g. a `SECURITY.md`, a threat-model doc, or a review-agent definition) | > **Egress reality β€” read before choosing cloud reviewers.** A cloud reviewer (`agy`β†’Google, `grok`β†’xAI) is given **read access to your whole working tree**, not just the diff. If a plaintext secret file can sit in the tree (`.env`, `.dev.vars`, local dev state), that is an egress surface. Layer-0 of this plan fails closed on the common cases; for a repo whose entire purpose is not leaking credentials, prefer the **local** reviewer (`ollama`) or a scrubbed egress. --- ## Provisioning Inputs *(resolve once, up front; the plan is a pure function of these)* | # | Question | Options (closed enum) | Default | Sets | Gates | |---|----------|-----------------------|---------|------|-------| | 1 | Which independent reviewers? | `dual-cloud` (two vendors) / `cloud+local` / `local-only` (ollama) | `dual-cloud` | `REVIEWERS` | which CLIs the engine invokes | | 2 | Enforcement style | `skill-only` (advisory) / `skill+gate` (blocking pre-merge) | `skill+gate` | `INSTALL_HOOK` | whether Β§4 (hooks) is applied | | 3 | Checkpoint/diff tool | `arc` / `git` | `arc` | `VCS` | the range-resolution + freshness-hash commands | | 4 | Egress posture | `cloud-diff-only` / `local-only` / `cloud+scrub` | `cloud-diff-only` | `EGRESS` | Layer-0 guard strictness + which CLIs run | | 5 | Adjudication | `synthesize` (primary agent merges + verifies) / `raw` (print both verbatim) | `synthesize` | `ADJUDICATE` | the skill body | ```yaml # β†’ written into Live State once resolved resolved_inputs: reviewers: dual-cloud install_hook: true vcs: arc egress: cloud-diff-only adjudicate: synthesize resolved_by: resolved_at: ``` > **Provider-agnostic reviewers** β€” the engine branches only the *invocation*, not the *prompt*: the same adversarial prompt + invariants + changed-file list goes to every reviewer; only the CLI flags differ. Adding a fourth reviewer = one more branch, never a forked prompt. Always keep a **`local-only`** member (ollama) so a repo that must not egress still fits the contract. ## TAGS β€” *deleted (this plan creates no taggable cloud resources; local files only).* --- ## Knobs β€” parameterize, don't fork One knob: `$PROJECT_DIR` (the target repo root; hooks reference it via the harness-injected `$CLAUDE_PROJECT_DIR`, never a hardcoded path β€” that is what makes the install portable to another machine). "Install into repo B" = same file, different `$PROJECT_DIR` β€” never a copied, path-edited plan. --- ## Live State *(the write-back install ledger β€” keep it true)* ```yaml status: not-created # published template - run it to realize state last_action: [what the last step did] last_verified: [what the last verify observed] resolved_inputs: { ... } ``` | key | value (filled on apply) | |---|---| | REVIEWERS_ON_PATH | `β€”` | | ENGINE_INSTALLED | `β€”` | | GATE_INSTALLED | `β€”` | | HOOK_WIRED | `β€”` | | SKILL_INSTALLED | `β€”` | | βœ” check | expected | observed | result | |---|---|---|---| | engine dry-run embeds invariants | a token UNIQUE to `$INVARIANTS_FILE` appears in the prompt | β€” | β€” | | engine never echoes raw model text | planted secret in a mock model's prose is absent from stdout | β€” | β€” | | gate fails **closed** | checkpoint-tool unavailable / stale / missing verdict β†’ exit 2 (block) | β€” | β€” | | **(negative)** gate does NOT fail open | a checkpoint-tool error yields exit 2, **not** a raw non-2 code | β€” | β€” | | **(negative)** engine refuses secret egress | a planted `.env`/dev-secret in the tree β†’ engine aborts path-only, no model runs | β€” | β€” | | hook matcher is anchored | a benign command that *mentions* the merge phrase is NOT gated; a real (even chained) merge IS | β€” | β€” | > Assert the **negatives** β€” a fail-*open* gate and a secret-egress path are the two ways this tool silently betrays its purpose. A dry-run that records a *failed* negative is the plan earning its keep. --- ## 0. Variables ```bash export PROJECT_DIR="" # never hardcode into hooks; hooks use $CLAUDE_PROJECT_DIR export INVARIANTS_FILE="$PROJECT_DIR/" # what the reviewers are told to attack export VCS="arc" # arc | git (from Provisioning Input 3) export REVIEWERS="dual-cloud" # from Input 1 # Reviewer invocations (headless). Adjust to the CLIs you resolved: # agy -p "" --add-dir "$EGRESS_ROOT" --print-timeout # grok -p "" --cwd "$EGRESS_ROOT" --disable-web-search --output-format json --json-schema '' # ollama run (stdin prompt; local, no egress) ``` ## Dependency frontier ``` findings-schema + assert-helper ─┐ β”œβ”€> engine (range β†’ EGRESS GUARD β†’ prompt β†’ parallel reviewers β†’ tagged JSON) project-invariants file β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”œβ”€> freshness-gate script ─> 🟑 hook wiring ─> βœ” integration └─> /adversarial-review skill (synthesis + verdict artifact) └─> runbook + discoverability pointer ─> βœ” acceptance ``` Non-negotiable edges: the **egress guard precedes any reviewer call** (a secret must never leave before the scan); the **gate reads an artifact the skill writes** (schema must agree); the **hook execs the gate** (gate must exist + be executable first). Teardown reverses this. ## 1. Findings schema + test harness 🟒 ```bash # a JSON contract every reviewer must emit: {findings:[{file,line,severity,invariant,claim}], verdict:"pass"|"fail"} # + a shared bash assert helper (assert_eq / assert_contains / assert_exit) the later tests source. ``` ```bash # βœ” verify: jq validates the schema; the contract fields + severities are present. ``` > β†’ Live State: schema present. ## 2. Review engine 🟒 The reusable core. Ordered stages (the order is load-bearing): ```bash # 1. resolve changed files from $VCS (arc: `arc intent diff --json`; git: `git diff --name-only `) # 2. build ONE adversarial prompt = + changed-file list + JSON contract # 3. --dry-run exits here (prints range + prompt; NO egress) # 4. LAYER-0 EGRESS GUARD (see Gotchas): abort path-only if a plaintext secret file is in the egress surface # 5. run every resolved reviewer IN PARALLEL; each reads the repo itself; capture each verdict # 6. emit one JSON object {reviewerA:…, reviewerB:…}; unparseable/empty/error β†’ {"error":…} sentinel, NEVER raw text ``` ```bash # βœ” verify (no network): --dry-run asserts an invariant-unique token is embedded; # mock-reviewer tests assert (a) both verdicts pass through, (b) a down reviewer degrades to a sentinel, # (c) a planted secret in prose output never reaches stdout, (d) empty output doesn't crash the merge, # (e) runs unchanged under the system's oldest bash (portability). ``` > β†’ Live State: engine present + all engine tests green. ## 3. Freshness gate 🟒 ```bash # exit 0 IFF a verdict artifact exists, verdict=="pass", AND its tree-hash == the CURRENT tree-hash # (force a fresh checkpoint first to fold in uncommitted edits). Else exit 2 (block), PATH-ONLY reason on stderr. # The tree-hash is a content hash of the working tree (arc: snapshot tree_hash; git: `git write-tree`). ``` ```bash # βœ” verify: mock-VCS test β€” missing artifactβ†’2, fail verdictβ†’2, stale hashβ†’2, fresh passβ†’0, # VCS-errorβ†’2 (NOT a raw non-2 code), reason never contains the hash value. ``` > β†’ Live State: gate present + executable + gate tests green. ## 4. Hook wiring 🟑 *(only if `INSTALL_HOOK=true`)* ```bash # Wire a PreToolUse hook (harness config, e.g. .claude/settings.json) that execs the gate on an EXECUTING merge. # - Reference the gate via $CLAUDE_PROJECT_DIR (portable), NOT a hardcoded path. # - ANCHOR the command matcher (start-of-command or after ;&| ) so a benign *mention* of the merge phrase # isn't gated β€” mirror any existing guard hook's anchoring. Preserve all pre-existing hooks byte-for-byte. ``` ```bash # βœ” verify: drive the hook command with fake payloads β€” executing-mergeβ†’exit 2, previewβ†’0, unrelatedβ†’0, # benign-mentionβ†’0, chained (`cd x && …`)β†’2; pre-existing hooks still fire. ``` > β†’ Live State: HOOK_WIRED = true (+ which matcher). ## 5. Synthesis skill 🟒 ```bash # /adversarial-review (or your harness's equivalent): resolve+confirm range β†’ snapshot β†’ pre-egress secret # check on changed files β†’ run engine β†’ VERIFY each finding at the source (external models hallucinate) β†’ # dedup + map to $INVARIANTS_FILE β†’ present merged report β†’ write the verdict artifact stamped with tree_hash. # Invariants in the prose: no secret VALUE in report/artifact; inconclusive/single-reviewer never auto-passes. ``` ```bash # βœ” verify: frontmatter parses; the five safety clauses are present (secret gate, value-never-logged, # single-reviewer-no-auto-pass, verdict schema w/ tree_hash, verify-at-source). ``` > β†’ Live State: skill installed. ## 6. Runbook + pointer 🟑 ```bash # A runbook (when/how/caveats β€” incl. the whole-tree egress reality) + a one-line pointer in the repo's # agent guide so the gate is discoverable. Docs, no verify beyond "present + accurate." ``` ## Update (idempotent reconcile) 🟑 Re-running is safe: files are overwritten in place; the hook entry is matched-and-skipped if already present (append only if absent β€” never duplicate). Re-run the full test suite; a drifted component fails its own βœ” and is re-applied. ## Teardown (observe-first, resumable) πŸ’₯ > Reverse of install. Observe what's present, act, re-observe. Removes **only** what this plan created. ```bash # πŸ’₯ remove the hook ENTRY this plan added (leave every pre-existing hook intact β€” surgical, not a file wipe) # πŸ’₯ rm the engine, gate, schema, tests, skill, runbook, and the verdict artifact + its ignore entry # πŸ’₯ remove the one-line pointer from the agent guide ``` ```bash # βœ” teardown verify: the hook no longer fires on a merge; pre-existing hooks still fire; no tool files remain. ``` > β†’ Live State: `status: gone`. ## Deliberately not included - **An MCP server** β€” the reviewers are already headless CLIs; shelling out is strictly simpler than a transport to maintain. Reach for MCP only if a reviewer is networked/stateful. - **An autonomous blocking hook that runs the models inline** β€” it can't synthesize and would block a merge for minutes. The gate checks a *freshness artifact* the skill produced; the expensive review stays in the skill where a model is in the loop. - **CI/remote integration** β€” local dev-flow only; a CI variant is a separate plan. - **Trusting the verdict artifact as a security boundary** β€” it's a local, hand-editable convenience tripwire, not a control. The real gate is human judgment on the synthesized report. --- ## Gotchas & countermeasures (the durable payload β€” learned the hard way) These are the bugs an adversarial review of *this very pattern* caught before merge. They are latent in the "obvious" implementation β€” bake the countermeasure in from the start. 1. **Fail-*open* gate under `set -euo pipefail`.** A plain `CUR="$(vcs … | jq …)"` assignment that fails **aborts the script with the pipeline's raw exit code** *before* reaching your `block()` call. If the harness treats hook exit-2 as "block" but any *other* non-zero as "non-blocking," a VCS hiccup lets the merge **through**. Countermeasure: `… | jq … 2>/dev/null || true` so an empty result routes to `block()` (exit 2). Test it: point the mock VCS to fail *after* the freshness snapshot and assert exit 2. 2. **Egress surface βŠ‹ the diff.** Agentic CLIs get read over the **whole tree** (`--add-dir`/`--cwd`), but a naive secret scan only covers *changed* files. An unchanged/untracked `.env`/dev-secret is then egressible-but-unscanned. Countermeasure: a **Layer-0 engine guard** that refuses to run (path-only, exit non-zero) if a plaintext secret file is present in the egress root β€” existence check only, never read contents; exempt `--dry-run` (no egress). This also covers direct-engine use, which the skill's gate doesn't. 3. **Empty model output crashes the merge.** `jq empty` accepts empty input, so an empty reviewer reply (a soft-killed timeout) makes `jq -n --argjson x ""` crash with **zero** output β€” no error sentinel, breaking "stdout is always one JSON object." Countermeasure: guard `[ -s "$f" ] && jq empty "$f"` before passing through; else emit `{"error":"unparseable"}`. 4. **Portability: empty bash array under `set -u`.** `"${arr[@]}"` on an *empty* array throws `unbound variable` on bash < 4.4 (macOS stock `/bin/bash` is 3.2) β€” silently disabling a reviewer on another machine. Countermeasure: `"${arr[@]+"${arr[@]}"}"`, or avoid arrays (string accumulators). Test under the oldest bash you target. 5. **Un-anchored hook matcher = false-positive blocks.** A `grep 'the merge phrase'` that matches *anywhere* in a command blocks benign commands that merely mention it (docs, `echo`, this very plan). Countermeasure: anchor to a leading command token (start, or after `;&|`) exactly like a git-guard; a chained real invocation still matches off the separator. Note: a string matcher can't parse shell β€” a quoted fixture containing the phrase is correctly matched; that's inherent, document it. 6. **Value-never-logged, everywhere.** Engine prints paths/prompt/schema only; the gate's block reasons are static strings (never the hash); the verdict artifact carries `file/line/invariant/severity/claim` only. Trace the resolved/decrypted value to every sink and confirm none is a stdout/stderr/artifact/error path. ## Acceptance contract Installed correctly ⇔ **all** component βœ”s green (incl. the two negatives), a real executing-merge is blocked without a fresh passing verdict and allowed with one, and a one-time **live smoke** (one real `/adversarial-review` on a small known change) shows the external CLIs invoked, findings verified-at-source, and a clean artifact written β€” with no secret value anywhere in the output.