# Rust agent harness course review

Reviewed 2026-10-10. Original lessons, reference pages, shared assets, and learning records are preserved. This review covers all 21 chapters and the four reference pages. It distinguishes executable defects, contradictions visible in the text, and suggested production hardening. There is no complete learner-built harness in this workspace; its end-to-end behavior cannot be certified from the course alone.

## Verdict

Keep the course. Its dependency direction, provider adapters, sans-IO approach, tool-error feedback, deterministic loop tests, permissions/sandbox distinction, and eval pyramid make a coherent foundation. The hidden solutions and retrieval questions respect the learner's preference to write the code themselves.

The course is broad enough for a production-shaped prototype. It is not yet a sufficient production correctness or security specification. Several reference snippets violate their own contracts, and recovery and authority boundaries need more explicit exercises. These are substantive issues, not reasons to replace the architecture.

## Most consequential findings

### 1. Transcript validator misses duplicate calls and wrong roles — confirmed by Rust tests

Chapter 01, step 1.2. Converting calls into a HashSet before checking uniqueness discards duplicate IDs within one message. Two calls with the same ID and only one result pass validation. The validator also accepts tool calls on user messages and results on assistant messages.

Check every call before deduplicating, enforce block-role restrictions, and then compare pending IDs with the next message's results. Keep `allow_pending_tail` as an explicit temporary state. Strict role alternation is a useful anvil convention; it is not a universal requirement imposed identically by all provider APIs.

Alternative lecture: [Transcript and stream integrity](../lessons/0002-transcript-and-stream-integrity.html).

### 2. Permission implementation contradicts promised authority constraints — static evidence

Chapter 11, step 11.2. `all_allowed` returns before `touches_outside_workspace`, allowing an explicit allow rule to bypass the stated requirement that outside paths are at least Ask. Plan mode is checked after allow rules, so an explicitly allowed mutation can execute in plan mode despite the checkpoint promising proposals only. This needs a deliberate policy decision rather than silently inheriting the reference order.

Specify the order as: validate requested operation; enforce hard denials and mode ceilings; identify canonical resources; require escalation approval; then apply user rules within those ceilings. Reject invalid/empty shell parses rather than accepting `all()` on an empty segment list. Evaluate every affected resource, not a single path string. Preserve deny precedence when escalation is requested.

Acceptance cases: plan + allow(edit) still denies edits; allow(read_file) + outside path still asks unless a separate explicit outside grant exists; deny + network escalation still denies; unsupported shell syntax asks; an empty/invalid command cannot auto-allow.

### 3. Shell sandbox does not protect native file tools — architectural gap

Chapters 10–12. The sandbox wraps bash subprocesses. Native read/write/edit/glob/grep operate in the harness process. Lexically normalizing `..` does not resolve symlinks, and a check followed by open/rename can race with filesystem changes. A broad read-only default also needs explicit sensitive-file controls for searches, not only read_file.

Add a file-access boundary shared by all native filesystem tools. Define allowed reads and writes, protect metadata and credential paths, check symlink behavior, and select a capability/descriptor-relative implementation or sandboxed worker for stronger enforcement. A simple canonicalize-and-check implementation can be a staged prototype if its race limitations are stated.

Acceptance cases: native write through an inside-workspace symlink to an outside directory is blocked; new-file creation checks the parent; searches cannot reveal denied file contents; `.git`, `.anvil`, hooks and skills cannot be rewritten to acquire new authority without a user grant.

### 4. Project configuration can carry executable authority — missing trust boundary

Chapters 06, 15, 16. Project config wins during merging; hooks execute shell commands; MCP config launches processes; provider config selects endpoints and environment-variable names. An untrusted repository can therefore change more than instructions. In particular, overriding a provider's base URL while retaining its credential variable can send that credential to a different host.

Separate repository data from trusted execution/security config. Require a trust decision before activating repository-supplied commands, environment passthrough, endpoint overrides, or permission expansions. Bind credentials to approved endpoints. Keep this distinction in the loader, not only in a prompt.

### 5. Session record model cannot replay every mutation — completeness gap

Chapter 14. `push_user` can extend an existing message after cancellation or tool execution. A log containing only Message append records cannot represent that change unless it records a patch/replacement. Tool-result clearing, `/clear`, model switches, todos, and `/rewind` also need explicit persistence semantics. Some later chapters add records, but no complete reducer contract is specified.

Define append, extend/replace, context-reset, model-change, todo-update, tool-start, and tool-finish records as needed. Test the invariant `replay(records) == live_state` after each mutation. A torn final JSONL line may be recoverable; malformed interior records should produce an error. Flush is not a power-loss durability guarantee; define when sync is required.

After a crash, a tool that started without a durable result has an **unknown outcome**, not necessarily a failed or unexecuted action. Do not automatically repeat side effects. Reconcile files or external state, report uncertainty, and obtain direction when necessary. Synthesized results preserve transcript structure but cannot prove what happened in the world.

### 6. Schema sanitizer corrupts legitimate schemas — confirmed by Rust tests

Chapter 05, step 5.1. The generic recursive walk treats property-name maps as schemas. A field named `title` is deleted from `properties` while remaining in `required`. Removing `$defs` leaves `$ref` dangling. A union of string and integer silently becomes string.

Walk schema positions rather than arbitrary JSON objects. Resolve supported references before removing definitions, detect recursion, preserve unions when the target dialect supports them, and reject unsupported constructs with a precise path. The GenerateContent API offers `parametersJsonSchema` as an alternative to `parameters`; verify target model support and limitations rather than blindly stripping keys. [Google API reference](https://ai.google.dev/api/generate-content#FunctionDeclaration).

### 7. SSE decoder deviates from its cited specification — confirmed by tests and primary spec

Chapter 02, step 2.3. `finish()` dispatches an incomplete event at EOF, whereas WHATWG discards pending data. A leading UTF-8 BOM causes the first `data` field to be lost. Lone CR is explicitly documented as unsupported, so that limitation is not a hidden defect, but the claims of a fully proven decoder are too strong. The existing split test compares the implementation against itself and covers one fixture.

Use independently expected outputs; test BOM, LF/CRLF/CR, bytewise chunking, empty data, comments, and incomplete EOF. Bound line/event sizes. If a known provider needs permissive EOF behavior, put it behind a named compatibility policy and test it separately. [WHATWG SSE interpretation](https://html.spec.whatwg.org/multipage/server-sent-events.html#event-stream-interpretation).

### 8. MessageBuilder hides malformed stream transitions — confirmed by Rust tests

Chapter 03, step 3.1. A delta with no start is ignored. A duplicate start overwrites the slot mapping and leaves an earlier ToolCall with null arguments. Repeated Finish and events after Finish are accepted. Reasoning and text lack block indices, limiting faithful reconstruction when blocks interleave or signatures arrive separately.

Make protocol transitions fallible. Distinguish malformed model arguments (tool feedback) from malformed event sequencing (adapter/protocol error). Add indexed block events when a supported provider requires them. Execute calls only after successful stream completion and argument validation. Handling truncation/refusal with tool blocks needs an explicit policy before the loop dispatches them.

### 9. Provider switching is conditional, not automatically safe

Chapters 01, 05, 09 and 20. Dropping foreign metadata is necessary but does not guarantee that a new provider will accept imported tool history or that returning to the original provider restores a valid reasoning chain. Preserve same-provider state intact; define unsupported-history behavior and offer a deliberate text handoff when needed.

The old Gemini thought-signatures URL now directs readers to a Thinking guide centered on the Interactions API. Do not combine that protocol's step/signature events with the course's GenerateContent adapter. Record the endpoint family with fixtures. [Gemini Thinking guide](https://ai.google.dev/gemini-api/docs/thinking), [GenerateContent reference](https://ai.google.dev/api/generate-content).

Mapping Anthropic `pause_turn` to EndTurn would prematurely stop a server-tool continuation. Server-side tools are outside the current core scope, so this is a future compatibility boundary rather than evidence that the course's client-tool demo fails. Reject unsupported continuations instead of declaring success. [Anthropic stop reasons](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons).

### 10. Resource limits and cancellation need stronger specifications

Chapters 08, 10, 15, 17, 18. Unbounded event channels and `wait_with_output()` can retain unbounded output before truncation. Process-group kills cover members of the group, not descendants that detach. Cancellation of a future is not an exact-once side-effect guarantee. MCP calls, approval waits and hooks need bounded cancellation behavior, cleanup and ownership. `Tool::read_only()` is independent of arguments, but chapter 17 makes task read-only depend on the selected agent definition; the interface cannot directly express that.

Use bounded output collectors while continuing to drain pipes, bounded/coalesced progress delivery, process cleanup with documented OS limits, and per-call effect classification. For child requests, share a session budget with admission/reservation; checking spend after steps is a soft ceiling and can overshoot, especially with parallel subagents. Missing usage or pricing must be represented as unknown rather than zero.

Dropping a client stream does not establish that provider generation and billing stop instantly. Avoid teaching that as a guarantee.

### 11. Evals have a sound foundation, but need trusted graders and uncertainty

Chapter 19. The pass@k estimator is correct for valid inputs; the pass^k expression is also appropriate for sampling k trials without replacement from n outcomes. Keep these. Add checks for `0 < k <= n` and `c <= n`. When n equals k, the estimates collapse to whether any/all trials passed, so three trials provide very coarse evidence, not strong proof of improvement.

Keep graders and hidden tests outside the agent's writable tree. Distinguish task failure from infrastructure failure; retain both rather than silently dropping costly errors. Add per-task uncertainty and paired baseline comparisons, pin task/dependency versions, and record grader provenance. A test suite inside the agent's repo can be modified along with implementation files unless independently protected.

### 12. Checkpoint restore needs a user-edit conflict policy

Chapter 20. A shadow git directory isolates index/history, but restoring files can still overwrite user edits made after the snapshot. Define a three-way restore, a conflict preview, or an explicit refusal when a file changed outside the agent's recorded edits. Keep ignored/secret files out of snapshots and test permissions, symlinks and file creation/deletion. Snapshot before the first mutation, including bash/hook actions, not only direct edit tools.

## Chapter coverage

| Chapter | Assessment |
|---|---|
| 00 Orientation | Strong framing and dependency direction. Production-shaped is a justified goal; production safety remains to be demonstrated. |
| 01 Canonical conversation | Good abstraction; repair validator and qualify cross-provider replay promises. |
| 02 Provider/SSE | Good sans-IO teaching; repair framing contracts and add independent expected-output tests. |
| 03 Anthropic | Good decomposition; strengthen builder transitions and state preservation. |
| 04 OpenAI-compatible | Useful explicit quirks; feature support varies by endpoint/model, so configure and test capabilities. |
| 05 Gemini/Responses | Most fragile adapter chapter: schema corruption and protocol-specific replay need attention; Responses is marked stretch despite broad claims of four completed protocols. |
| 06 Registry/retries | Decorator and jitter are sound. Add endpoint trust, total retry deadline, and HTTP-date Retry-After if required. Tokio paused-time tests need `test-util`; `full` alone does not enable it. |
| 07 Tools | Good errors/spec ordering. Truncation counts bytes while saying characters, and inserted marker exceeds nominal maximum. Specify whether the limit is content or total output. |
| 08 Loop | Good scripted tests. Add abnormal-finish cases, stream-error events, bounded UI delivery and explicit per-call effects. |
| 09 CLI | Solid headless design. Specify renderer drain/EOF before exit, error outcomes, signal-listener cleanup and conditional provider switching. |
| 10 Coding tools | Useful contracts; add native filesystem confinement, content freshness checks, empty old_string handling and bounded pipe capture. Mtime is a heuristic, not a complete freshness guarantee. |
| 11 Permissions | Strong concepts; reference ordering breaks outside-path and plan-mode promises. |
| 12 Sandbox/secrets | Valuable OS work; separate subprocess enforcement from native tool authority. Regex env scrubbing/exact redaction is useful defense, not a guarantee against all secrets or exfiltration. |
| 13 Context/cache | Stable prefix is a good aim. Historical usage is a useful estimate of the next input, not exact across compaction, model switches, omitted reasoning or changed schemas. Reset the baseline when context changes. Cache thresholds make small-prompt live checks conditional. |
| 14 Sessions | Good recovery motivation; record mutations, distinguish transcript repair from side-effect recovery, specify torn records and durability. |
| 15 MCP | Current rmcp/spec facts verified. Add name-collision tests after sanitizing/truncating; wildcard tool-name matching is not implemented by the literal-tool rule description in ch11. |
| 16 Hooks/skills | Good extension model; trust repository commands. Fail-open may suit advisory hooks but not a mandatory verifier. Ensure verification status is reported when hooks fail/time out. |
| 17 Subagents/todos | Good context isolation; per-call effects, concurrency caps and shared admission budget need explicit design. Todo concepts say exactly one active item, build says at most one; at most one permits a completed list. |
| 18 Observability/cost | Useful measurements. Normalize to each standard's meaning; anvil uncached input and OTel total input are different quantities. Track interrupted requests and child spend; name soft vs hard budget behavior. |
| 19 Evals | Keep pyramid and estimators. Protect graders and add uncertainty, infrastructure-error categories and valid-input checks. |
| 20 Capstone | Good long-running pattern. Harden restore conflicts, snapshot coverage, authority on generated init scripts and autonomous verifier integrity. |

## Pedagogical assessment

The course already uses useful retrieval, explicit acceptance criteria and hidden solutions. Changes should preserve them. Six provider chapters before first interactive use delay feedback; a good optional route is 00 → 01 → 02 → one adapter → 07 → 08 → 09, then return to adapters and conformance. Keep coding tools disabled until their safety boundary exists.

Several quizzes are easy because distractors are obviously impossible. Add debugging and transfer tasks: diagnose an unseen trace, propose a counterexample to an invariant, compare two policies, and implement a failing regression case. Use staged checkpoints rather than treating a single passing fixture as proof of correctness. The time estimates are planning guesses; there is no learner performance evidence here to validate 40–60 hours.

## Teaching skill discovery

Used the requested find-skills workflow: checked the skills.sh leaderboard, then inspected the teaching skill and its source repository rather than recommending from a name alone. The leaderboard already surfaced a relevant candidate, so the fallback CLI search was unnecessary.

Candidate: **mattpocock/skills — teach**, listed at about 788.5K installs when checked. Its source repository displayed about 284K stars. Those are adoption signals, not proof of teaching effectiveness. Its relevant guidance is to tie lessons to the mission, use focused practical wins, retrieval practice and immediate feedback. This workspace already follows much of that structure. No skill was installed globally.

Optional installation command: `npx skills add mattpocock/skills --skill teach`.

Sources: [leaderboard](https://skills.sh/), [skill listing](https://skills.sh/mattpocock/skills/teach), [skill source](https://github.com/mattpocock/skills/blob/main/skills/productivity/teach/SKILL.md), [repository](https://github.com/mattpocock/skills).

## Verification and limitations

- Extracted representative original snippets into `verification/`; added minimal module/import glue, without repairing their behavior.
- `cargo test --offline --manifest-path audit/verification/Cargo.toml`: **15 passing tests**. Nine assert the observed original defects; six check corrected transcript/SSE behavior. Passing reproductions confirm bugs; they do not mean the original solutions are correct.
- MCP 2026-07-28 and rmcp 3.5.1 are supported by the current [spec](https://modelcontextprotocol.io/specification/2026-07-28) and [SDK docs](https://docs.rs/rmcp/3.5.1/rmcp/). They should not be “corrected” to older versions.
- [Tokio pause docs](https://docs.rs/tokio/latest/tokio/time/fn.pause.html) confirm the `test-util` feature requirement.
- [Bubblewrap maintainers](https://github.com/containers/bubblewrap#sandbox-security) explicitly distinguish the sandbox builder from the policy supplied by its caller.
- No live model requests, credentials, paid evals, or OS escape attacks were run. Findings about native file access, policy ordering and persistence are code/text analysis, not a penetration-test certificate.
- The original SHA-256 manifest is saved in `original-sha256.json` for preservation checks.