mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
88469bd2bc57add904ec8fc93a4265be47e8e8f4
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
62bc0519d9 |
fix(native): bound per-turn harness backups by inactivity, not total time (#22382)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a remote native turn is accepted, the executor saves the provider's harness state to durable storage, so another web server can continue the same conversation. > - That save runs under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers archive capture, digest, encryption, the upload, any storage-outage wait and a full verifying read-back. > - On staging, five accepted runs failed this save 337–388 s after it began. The run then reported `native_workspace_sync_out_failed` with "The operation was aborted" or tar-stream's "already finalized or destroyed", and workspace copy-back went to retry. > - #19906 fixed the same fixed-deadline rule on the restore side. The write side still uses it. Since #20772/#20871 the save also waits out storage throttling and outages for up to five minutes, which cannot finish inside a five-minute total. > - This pull request bounds the upload and the read-back by inactivity instead of total time. A slow transfer continues while the store makes progress. A transfer that stops fails with a named code. > - The benefit is that an accepted turn no longer loses its provider backup only because the backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19906 (same idle-deadline rule for failover-backup restores), #20772 and #20871 (file-source resends and storage-outage waits during the save), #19108 (stops tar-stream from hiding the real abort), #18932 (durable harness backups). **What happened?** On staging, five native per-turn runs failed within one hour, all after the provider turn was accepted. In each run, `harness_state.backup.persist` failed after 359,588 / 365,422 / 387,471 / 383,801 / 337,335 ms, and `session.checkpoint.persist` failed with it. The only fixed limit on that path is `AbortSignal.timeout(300_000)` around `persistNativeHarnessBackup` in `native-session-executor.ts`. The durations sit just above 300 s, plus a variable sandbox sync-out step before the save. Errors were reported as "The operation was aborted" and, before #19108, as "already finalized or destroyed". **Expected behavior** An accepted turn saves its provider backup while the store keeps making progress, even when the total time is more than five minutes. A write or read-back that stalls fails with a named, diagnosable code. A failed save publishes no backup. **Steps to reproduce** 1. Persist a native harness backup through a store that takes upload chunks slowly, so the full save takes longer than the deadline. 2. With the executor's fixed total deadline, the save fails with "The operation was aborted" and publishes nothing. 3. A store that stops taking bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: generalize `withDownloadProgressDeadline` into `withTransferProgressDeadline`. It still wraps `getObject` exactly as before, and it now also wraps `putFile` with a five-minute idle timer: - A provider that reads the stream body (local disk) renews it with each chunk it takes. - A provider that sends the file source itself (S3 with `sourceFile`, which never reads the stream) renews it with each request the store answers, each multipart part it takes, and each resend or outage wait that starts. - After the last of them the timer stays armed until the store answers, so a write that never completes is named too. - `persistNativeHarnessBackup` gains `transferIdleTimeoutMs` and `persistMaxMs` (default 30 minutes). The outer bound wraps the whole save, including the storage-outage wait. With an idle limit, a stalled upload throws `native_harness_storage_upload_stalled`, a stalled read-back throws `native_harness_storage_download_stalled`, and the whole save throws `native_harness_storage_persist_deadline_exceeded` at the outer bound. Caller cancellation keeps its own reason, and a storage outage that outlasts its own budget still surfaces as the storage error. Callers that pass no idle limit (controller archive, offline migration, preservation) are unchanged. - `storage/request-stats.ts`: add `observeStorageProgress(onProgress, work)`, an `AsyncLocalStorage` observer called from the existing `recordStorageRequest`, `recordStoragePart` and `recordStorageBackoff` hooks. It is scoped to the wrapped `putFile` call only, and a throwing observer is ignored. Stats collection is unchanged. - `native-session-executor.ts`: the per-turn backup passes `NATIVE_HARNESS_UPLOAD_IDLE_TIMEOUT_MS` (five minutes) instead of `AbortSignal.timeout(300_000)`. - Restore behavior and error codes are unchanged: `withDownloadProgressDeadline` now delegates to the shared helper with the same defaults. - `doc/database-backups.md`: document the write-side idle limit, what counts as progress, and the new codes. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backup rows, references, receipts and checkpoints are unchanged. A deploy that lands during a save stops the process, and the pending row is never published, as before. ## Verification Run from `server/` (provider env vars unset): - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-upload-deadline.integration.test.ts`: 6 tests passed. They use embedded PostgreSQL, local-disk storage and an incompressible 1 MB acpx backup. - "keeps saving a backup whose upload outlasts the idle window while the store keeps taking bytes". Under the former fixed-total policy the same slow upload fails with "The operation was aborted" and publishes nothing. With the idle limit it completes after more than twice the window and restores. - "names an upload that stops taking bytes and publishes no backup", then saves the same source once the store responds. - "keeps a file-source upload going while the store answers its requests, and names one that stops". The fake store never reads the stream and records each request answer the way the S3 provider does. - "names a read-back that stops sending bytes and publishes no backup". - "stops at the outer bound while an upload still progresses". - "lets Stop end a progressing upload with its own reason". - RED, part 1: with `native-harness-backup-storage.ts` and `request-stats.ts` reverted to `master`, 4 tests fail. Three stall tests hang until the 10 s test timeout, and the outer-bound test resolves instead of rejecting. - RED, part 2: with only the progress observer disabled, the file-source test fails with `native_harness_storage_upload_stalled` on an upload that is still progressing. - `native-harness-backup-*` suites, `native-harness-online-preservation`, `src/storage/`, `storage-request-stats` and `storage-s3-multipart-backoff`: 200 of 201 passed. The one failure, `native-harness-backup-cleanup.test.ts` ("removes retired backups containing immutable skills…", expected mode 256 to be 292), fails the same way on unmodified `master` in this sandbox. - `native-session-executor.test.ts`: 1541 of 1542 passed. The new assertion checks that the per-turn backup passes `transferIdleTimeoutMs` and no fixed signal, and flipping it fails the `opencode-new-durable*` cases. The one failure, "compares export process compatibility without changing conversation identity (changed: false)" (`db.transaction is not a function`), fails the same way on unmodified `master`. - `node scripts/typecheck-layers.mjs`: no errors in `server/`. The local filtered install lacked some CLI/UI/demo packages, which reported unrelated errors. CI covers the full typecheck. Not verified: a live staging run with this build. The staging evidence shows the save failing just after 300 s on several runs. It does not show which step was slow (upload, read-back or local capture), so the fix removes the fixed total for every step on that path. ## Risks - A save that progresses very slowly can now take longer than five minutes. Before, it failed. The settle step waits for it, up to the 30-minute outer bound. - A local step that hangs (for example a stalled disk during capture) is now cut off at 30 minutes instead of 5 minutes, with `native_harness_storage_persist_deadline_exceeded`. - For S3 file-source uploads, progress is measured per request, not per byte. A single request that takes more than five minutes counts as stalled. For a 16 MiB single PUT, that is a store slower than about 55 KB/s. Multipart parts are 8 MiB, eight in flight. - If a successor takes over during a long save, the existing fence checks before and after the upload reject it with `owner_superseded`, and nothing is published. This is unchanged behavior. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc9ad17515 |
perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280)
## Thinking Path
- Cloud durable storage is a major source of failures and slowness on
staging. For example, a staging task failed when saving files after
automatic retries.
- Before measuring or fixing it, we need to see where storage time goes.
Today a saved-workspace restore before the turn appears only inside the
coarse `workspace_environment_realize` phase.
- After the turn, the save's encrypt, put, verify and multipart throttle
backoff have no timing at all. Only `captureMs`, `transferMs`,
`verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`.
- Storage retries happen in the SDK and in the multipart uploader,
invisibly.
- This adds the measurement first, so the storage-mode chaos waves
(MinIO behind a fault proxy) can report save and restore p50/p95 and
point at the dominant wait.
## Linked Issues or Issue Description
**What happened?**
Durable-storage saves and restores were opaque in the run log and save
timing. A slow or repeatedly retried storage call could not be told
apart from slow capture, encryption or sandbox work.
**Steps to reproduce**
1. Run a remote task with S3-compatible durable storage, so it saves
after the turn and restores on the next run.
2. Inspect the run log and the `remote_workspace_snapshot` operation's
`saveTiming`. The restore's download, decrypt and verify, and the save's
put, verify and throttle backoff, have no timing.
**Expected behavior**
Each storage step has a duration, and storage request retries and
backoff are counted.
## What Changed
- `server/src/storage/request-stats.ts` (new): an async-local collector
of storage request statistics. Counts only:
- `requests`, `attempts` and `failedRequests`;
- `retryDelayMs` (SDK waits);
- `parts`, `throttled` and `backoffMs` (the multipart uploader's waits).
- `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which
wraps every S3 send, records each request's SDK `$metadata.attempts` and
`totalRetryDelay`, on success and on failure.
- `server/src/storage/s3-multipart.ts`: records each uploaded part and
each resend wait, flagged throttled when the backend returned 503, 429
or SlowDown.
- `server/src/services/remote-workspace-snapshot-storage.ts`:
- Each step is a `storage.snapshot.*` run phase:
- save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`;
- restore: `download`, `decrypt`, `verify_plaintext`, `publish`.
- A caller can pass a `timing` record that receives the step durations,
the bytes and the request statistics.
- `server/src/services/remote-workspaces.ts`:
- The save passes a timing record, kept in `saveTiming.storage`.
- The sandbox unpack during a restore is its own phase,
`workspace.snapshot.sandbox_restore`.
- `execution-preparation-timing.ts`: the new phase names.
- `doc/run-log-events.md`: a new "Durable storage timing" section.
Because the restore runs inside run preparation, its steps of 50 ms or
more become `run.performance.span` rows with no extra plumbing. This is
run-log and instance-database data only; there's no Telemetry change.
## Verification
- `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`,
`storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`,
`remote-workspace-snapshot-storage` and `execution-preparation-timing`
tests: 86 passed. They include:
- collected stats for a throttled multipart upload: 12 requests, 7
failed, 7 throttled, 35.5 s backoff and 3 parts;
- a save and restore that fill every step duration and the byte count.
- `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming`
assertion now includes `storage`.
- Server typecheck: no errors in the touched files.
## Risks
- **Overhead:** one async-local lookup per S3 request, plus timestamps
per snapshot step.
- **Collector scope:** the collector only exists inside a snapshot save
or restore. Other storage callers are unaffected; they record nothing.
- **`saveTiming`** gains a `storage` object. Readers that compare it
exactly (one test, updated) will see the new field.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|