Interface EvalSpec<M>

Tools pre-approved for the run (--allowedTools, a permission allowlist); the default tools stay available either way. [] approves none. Default: Read Edit Write Bash.

The arms to compare, by name.

Record/replay cache mode. Default "off" (always re-sample). "readwrite" records each trial (output + post-run files) and replays it on a matching re-run — so editing measure re-scores for free; the model is re-called only when a model-affecting input changes. "read" replays but never records. The cache key excludes measure, so changing your metric still hits.

Where cache records live. Default .vigiles/eval-cache under cwd.

How many trials to run at once (across all arms × trials). Default 1 (fully sequential — the safe, no-surprise default). Raise it to cut wall-clock time; rate-limit bursts are absorbed by the retry/backoff below.

Reasoning budget for the run (claude --effort, e.g. "low" or an integer). Lives here beside model because it is part of the MEASUREMENT — it moves the output distribution, not the sample size — so it is hashed into the lock and the cache, and never read from an env var. Omit for the harness default.

Where each trial runs — required, no default.

• { kind: "ephemeral" } — a throwaway HOME (under the trial's temp dir) and a scrubbed environment: the OS essentials, the harness's own auth (runEnv.keep, plus its auth file copied in), the eval's own VIGILES_* variables, and nothing else. home: { kind: "files", files } seeds that HOME first (HOME-relative paths; the harness's auth file cannot be overwritten). Running a model-driven skill is itself a side effect, so a git push or a write to ~ lands in a disposable HOME.
• { kind: "inherit", reason } — your real HOME and environment, minus the parent session's identity. reason is printed with the report.

There is no default because both are wrong silently in one case: inheriting measures the author's machine (their config, their tokens), and an ephemeral run fails if a user's credential is not where the harness says it is — loudly, on trial 1, which is why it is the one to try first. See docs/safety.md (what a child run inherits).

Base fixture files (path → contents), written fresh for every run.

The eval LOCK — a committed staleness gate for CI (see src/eval-lock.ts). With a name set, vigiles eval --update (local, on your subscription) records the report to .vigiles/eval-locks/<name>.lock.json; --check (CI) verifies the committed result against the current inputs WITHOUT a model call, failing "stale" when they diverge. Mode normally comes from the CLI; set lock to drive it programmatically or point a test at a throwaway dir. The lock engages only when name is set (it keys the lock file).

Abort the run once measured cost reaches this many USD. In-flight trials finish; remaining ones are skipped and report.aborted is set. Needs the model to report total_cost_usd (the eval tier does).

Compute this run's metrics from its outcome.

Model alias. Default "haiku".

Retries on a detected rate-limit/overload before giving up. Default 3.

Base backoff ms (doubled each retry). Default 1000.

Seconds to wait between runs (avoid rate-limit bursts). Default 4.

Tool stubs on PATH (rung R2) — fake binaries that answer PER INVOCATION. Build each with experimental_stub: a list of rules over the argv tokens, first match wins. vigiles writes a shim per stub into a temp dir BESIDE the trial's working dir and puts it first on the run's PATH, so a skill/hook/agent — or a script it runs — that calls gh/psql/git gets the recorded answer instead of the real service.

An invocation no rule answers is UNANSWERED: the model sees one neutral stderr line and a non-zero exit, no new trials start, the report lists the call (ArmReport.unansweredStubCalls), vigiles eval exits 2 and --update writes no lock. Every call is recorded in RunContext.stubCalls, so experimental_stub.called(...) can assert one happened.

The answers are author/recorded fixtures, never model-synthesized — a synthesized tool output looks plausible but diverges from the real tool.

Stub each arm's skill BODIES (frontmatter kept) before the run — for firing comparisons (does the skill get SELECTED?), where a selected skill should stop at selection instead of running its (often expensive) procedure. Every arm with a pluginDir / skillsDir is repackaged with bodies stripped; arms without one are untouched. Don't combine with quality metrics: the body is gone, so there is nothing to grade. See stubSkillBody.

The task prompt given to the agent.

Per-run timeout ms. Default 240000.

Trials per arm. Default 5.