mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:38:22 +01:00
DottaandClaude Fable 5.2
f9171ef621
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native turn in a remote sandbox, closing the session saves
the provider session (the harness backup) to object storage. The next
attempt resumes from it.
> - On staging, object storage throttled that save ("Please reduce your
request rate", S3 `SlowDown`). The close failed, was retried 3 times 1 s
apart, and the session was quarantined. The next attempt of the same run
was refused with `native_session_cleanup_quarantined`.
> - The retries could never succeed: the Runner transport memoizes
`close()`, so every retry returned the first rejected promise and never
ran the checkpoint again. Quarantine recovery and admission recovery
call the same `close()`.
> - Also, the backup upload did not wait out a throttle or outage longer
than one request's own retries.
> - This pull request makes a close that failed only in its checkpoint
run that checkpoint again on the next `close()`, and makes the backup
upload wait out storage unavailability inside the existing bounded
storage wait.
> - The benefit is that a transient storage throttle no longer ends in a
quarantine that refuses the next attempt of the same run.
## Linked Issues or Issue Description
Refs #20772 (made the backup upload resendable after `SlowDown`). Refs
#20165 (per-request throttle backoff). Refs #20833 (earlier steps of the
same failure chain).
**What happened?**
1. A staging Codex run on Daytona finished its turn. The post-turn close
ran the remote checkpoint.
2. The checkpoint failed: `remote checkpoint failed: Please reduce your
request rate.`
3. The Runner retried the close 3 times 1 s apart (`native session
cleanup recovery after close failure (1..3)`), then 3 quarantine
recoveries. All failed within about 5 s.
4. The next attempt of the same run was refused:
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Expected behavior**
A throttle slows the save down. A retry of the close saves the session
again. The next attempt of the same run proceeds once the save succeeds.
**Steps to reproduce**
1. Make a transport close whose checkpoint throws once.
2. Call `close()` again.
3. On master the second call returns the first failure without running
the checkpoint again.
4. Separately, answer each harness-backup `PUT` with 9 `503 SlowDown`
responses. On master `persistNativeHarnessBackup` throws `SlowDown:
Please reduce your request rate.`
**Paperclip version or commit**
master at 381b862135.
## What Changed
- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: when
a close fails because its checkpoint failed (and the control-route
release did not fail), the transport keeps the checkpoint and the rest
of the close. The next `close()` runs the checkpoint again and then
finishes the close. All other close failures (for example a missing
suspension proof) stay memoized as before.
- `server/src/services/native-runtime/native-harness-backup-storage.ts`:
`persistNativeHarnessBackup` runs inside the existing storage outage
wait (`waitOutStorageOutages`, purpose "saving the provider session"). A
storage-unavailable failure (outage, throttle, rejected credentials)
that outlasts one request's retries is retried with the bounded
exponential storage wait (2 s doubling to 30 s, 5 minutes in total).
Each request still uses the jittered per-request throttle backoff from
#20165.
- Restart chaos: storage fault `ST8` answers the next 9 to 12 (seeded)
harness-backup `PUT`s with `503 SlowDown`. The storage audit reports it
when it never changed a request. The README documents it (for `--env
daytona --storage s3`).
## Verification
- RED on master (fix reverted, new tests kept):
- `runnerd-codex-transport-suspension.test.ts` "runs a failed close
checkpoint again on the next close": the second `close()` returns the
first `SlowDown` error.
- `native-harness-backup-throttle.integration.test.ts` "waits out a
throttle that outlasts one request's backoff": `SlowDown: Please reduce
your request rate.`
- GREEN: both pass. `native-harness-backup-throttle.integration.test.ts`
3 passed. `runnerd-codex-transport-suspension.test.ts` 12 passed.
`runnerd-codex-transport-release-resume.test.ts` 22 passed (twice).
`vitest --config tests/restart-chaos/vitest.config.ts`: 46 files, 554
tests passed, including the new ST8 planning and proxy test.
- `tsc --noEmit` clean for `packages/paperclip-runner`, `server` and
`tests/restart-chaos`.
- `runnerd-codex-transport-settlement.test.ts` has one case that also
fails on unchanged master in this worktree.
## Risks
- A checkpoint retry runs the same server checkpoint callback again
after the runner process was stopped. It reads the settled state from
the sandbox. If the sandbox is gone, it fails as before and the
quarantine stays.
- A close attempt can now take up to the 5-minute storage wait during a
long outage. This is on the post-turn path only, not before
`turn.start`.
- No change to the quarantine admission rules: admission still
accelerates one bounded recovery batch and refuses if it fails. Those
recoveries now really retry the checkpoint.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>