anvil · reference

Harness quality checklist

Use it to review anvil, or any harness. Each item names the chapter that builds it. Prints on two pages. ← course ·

Revised contracts. Native filesystem tools enforce FileAccess grants; plan mode caps mutation authority; project executable config is trusted separately; every session mutation has a reducer record; unfinished side effects recover as unknown; required verifiers and eval graders are independent evidence. Provider switching is conditional on supported replay. See the matching course boundary exercises for the build contract.

Provider-neutral (01–06)

  • Canonical types only above the provider layer
  • Opaque provider state preserved and replayed to its own provider only
  • Usage normalized to one meaning (uncached input)
  • Unknown stream events ignored, mid-stream errors surfaced
  • Retries with jitter, honoring retry-after, only before output
  • Conformance suite passes on every configured provider
  • Keys only in env vars, config names them

Loop correctness (07–09)

  • Transcript integrity validated in tests after every edge case
  • Tool failures become actionable error results
  • Step limit, cost budget and cancellation are stop conditions
  • Ctrl-C at any point leaves a resumable, valid transcript
  • Core emits events and has no UI code
  • Headless mode: clean stdout, JSON output, meaningful exit codes

Tools (07, 10)

  • Few, non-overlapping, well-described tools
  • Deterministic tool ordering
  • Exact-string edits with uniqueness errors
  • Read-before-write freshness check
  • Output truncated with guidance on how to narrow
  • Bash: timeout, process-group kill, no pipe deadlock

Safety (11–12)

  • valid effects → deny → plan ceiling → escalation/outside grant → ask → allow → mode, with deny winning everywhere
  • Compound shell commands checked per segment
  • Denials carry user feedback back to the model
  • OS sandbox: workspace-only writes, network off by default
  • Escalation always requires explicit approval
  • Env scrubbed for every subprocess. Secrets redacted from transcripts.
  • Escape tests run in CI on macOS and Linux

Context (13–14, 17)

  • Minimal, right-altitude base prompt, justified by evals
  • AGENTS.md discovered and labelled
  • Stable cache prefix. Hit rate measured.
  • Usage baseline plus conservative estimates, invalidated on context mutation
  • Tool-result clearing, then summarization compaction
  • Append-only, crash-safe, resumable sessions
  • Todo list survives compaction

Extensibility (15–17)

  • MCP tools namespaced, permission-gated, output untrusted
  • Broken MCP servers don't block startup
  • Hooks at five lifecycle events with bounded stop continuations
  • Skills by progressive disclosure
  • Subagents: fresh context, filtered tools, no recursion, same safety

Observability (18)

  • Spans for turn / step / chat / tool with GenAI attributes
  • TTFT, steps, tool error rate, cache hit rate, cost tracked
  • Logs to file, never to stdout
  • Session replay for humans

Evals (19–20)

  • Scripted-provider loop tests and golden transcripts on every commit
  • Task evals with computational graders, k trials, pass@k and pass^k
  • Baseline comparison flags regressions
  • Harness changes justified by experiments, not by feel
  • Workarounds for model weaknesses are configurable and re-evaluated per model

Boundary regression suite