mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Fable 5.2
381b862135
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run ends its task with a Paperclip terminal call
(`paperclip_finish` or `paperclip_block`)
> - The Runner refuses a terminal call that the agent decided before it
saw earlier tool results (`terminal_call_before_results`). This protects
against stale calls held across a restart.
> - On staging, Codex cancelled a nested exec cell. Codex never sent
`item/completed` for one Paperclip tool call in that cell. The Runner
then refused every later finish and block in the turn, and the turn
ended with no semantic result.
> - The same-run recovery then resumed the run on a new controller
container. The run's admitted instruction bundle is stored on the
controller's local disk, so it was missing there, and runnerd failed
with ENOENT.
> - This pull request makes a refusal name a returned call once, and
rebuilds a missing admitted instruction bundle before the Runner backend
starts.
> - The agent-directory lock part of the same failure chain ("Agent
directory operation is already active") is fixed separately by #20829.
> - The benefit is that the agent can finish after one refusal, and a
resumed run does not fail because a deploy moved it to a host without
the bundle.
## Linked Issues or Issue Description
Refs #20829 (the agent-directory lock wait and dead-owner release; this
PR does not change that lock).
**What happened?**
Staging Codex runs on Daytona sandboxes failed after about 30 minutes of
work:
1. Early in the turn, the agent called `paperclip_search_assigned_tools`
from a nested exec cell. Codex cancelled that cell. The controller
returned the result, but Codex never sent `item/completed`, so the
Runner never committed `tool.execution.completed` for it.
2. At the end, the agent called `paperclip_finish` four times and
`paperclip_block` once. The Runner refused each one with
`terminal_call_before_results` ("Those calls have now returned"). The
old call stayed "returned but unseen" for the rest of the turn.
3. The turn completed with no semantic result
(`native_finalization_missing`). Same-run recovery scheduled another
attempt.
4. On the new controller container, the attempt failed with
`runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir
'/paperclip/instances/default/runtime-context-assets/bundles/<sha>'`
(one run) or `ENOENT ... lstat` of the same kind of path (a run during a
rolling deploy). In both, `<sha>` is the run's admitted instruction
bundle. The asset root is controller-local, and the new containers had
no volume.
5. The last attempt failed with "Agent directory operation is already
active" (fixed by #20829).
**Expected behavior**
The agent finishes after one refusal. A resumed run rebuilds its exact
admitted instruction bundle on the new host.
**Steps to reproduce**
1. Run a Codex task that calls a Paperclip tool, then
`paperclip_finish`.
2. Drop Codex's `item/completed` for the first tool call (new chaos
fault `--faults PV2 --provider-fault-message tool_completed`).
3. On master, every `paperclip_finish` is refused and the run ends with
`native_finalization_missing`.
4. Separately, delete `runtime-context-assets/bundles/<instruction
digest>` and resume a run that pinned it: runnerd fails with ENOENT.
**Paperclip version or commit**
master at 1d67e309e0.
## What Changed
- `packages/paperclip-runner/src/live/terminal-call-freshness.ts` and
`runnerd-codex-transport.ts`: when the Runner refuses a terminal call
with `terminal_call_before_results`, it records the refused call id with
the returned-but-unseen call ids it named. A later terminal call ignores
those calls only when the refused call's `tool.execution.completed` (the
refusal reached the provider) is committed before the later call starts.
A terminal call queued before the refusal reached the provider, a call
that was pending at the refusal, and a call made after it are judged as
before. The record is in memory, so after a Runner restart the agent can
get one more refusal.
- `server/src/services/native-runtime/runtime-context.ts`: new
`rehydratePinnedInstructionAsset`. When the admitted instruction bundle
is not present on this host, it rebuilds it from the agent's instruction
revision whose content has the admitted digest (single-file bundles), or
from the current instructions when they still produce the admitted
digest. It never substitutes different content. When nothing matches, it
leaves the context unchanged.
- `server/src/services/native-runtime/native-session-executor.ts`:
`createRunnerdBackendWithinSessionClaim` calls it before the backend
starts. When the bundle is present this is two local file reads.
- Restart chaos: new provider message `tool_completed` for the Codex
shim (PV1/PV2/PV3). It targets Codex's `item/completed` for the first
Paperclip tool call that is not a finish or block. Config accepts it
with A1 and C1 on Codex only. The README documents it.
## Verification
- RED on master (fix reverted, new tests kept): `vitest run
src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner`
fails with `expected [ 'paperclip_search_assigned_tools' ] to deeply
equal []`.
- GREEN: `terminal-call-freshness.test.ts` 13 passed; the transport
tests for terminal calls pass, including a new test where the first
finish is refused and the second is dispatched, and tests where a queued
stale terminal call and a late-returning pending write are still
refused.
- `server/src/__tests__/runtime-skill-preservation.test.ts`: 9 passed,
including the new test that deletes an admitted instruction bundle and
its manifest, then rebuilds the exact bundle from the revision history
while a newer revision exists.
- `vitest --config tests/restart-chaos/vitest.config.ts
provider-faults.test.ts`: 22 passed.
- Chaos wave `w-20261111T150212-s301` (`--env local --adapters codex
--workloads A1 --faults PV2 --provider-fault-message tool_completed
--tasks 1 --seed 301`): 1/1 PASS, `provider_fault_applied` applied=1,
R1–R6 and the A1 oracle pass.
- `pnpm verify` asks for broader verification (a restart-chaos README
change is a global CI input); required CI supplies it.
## Risks
- The terminal-call change lets a later finish through when a returned
call never reaches the agent. The earlier refusal named that call and
reached the provider first, so the agent decides with that knowledge.
- The instruction rebuild adds two local file reads before the Runner
backend starts. It writes only when the bundle is missing, and only
bytes whose digest equals the admitted digest.
- Not covered here: a multi-file (legacy) instruction bundle whose
content changed since admission still cannot be rebuilt; a skill bundle
with neither a version id nor a preservation receipt is not rebuilt; the
runner-side quarantine after an S3 `SlowDown` during the post-turn
harness backup.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>