mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
17e8bd7735
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox save two things after each turn: the provider checkpoint and the workspace export > - An agent can fill the sandbox disk during its own work, for example with a package install > - The turn still finishes, but both saves need free sandbox disk, so they fail and the task becomes blocked with no clear reason > - This pull request makes the checkpoint work on a full disk and makes a full disk visible in the export failure > - The benefit is fewer stranded checkpoints, and an operator sees at once why an export failed ## Linked Issues or Issue Description Related: #18715 added the shared-principal response-recovery control synchronized below. #18644 reduced the scratch disk that the workspace capture uses. #18172 added the disk-free streaming fallback to the Daytona directory sync-out. Both were deployed when this failure occurred. Neither changes the checkpoint copy, and neither can help when the disk has no free space. **What happened?** A native agent ran a package install for a large monorepo in a 10 GB Daytona sandbox. `df` in the run showed `10G 10G 4.6M 100%`. The agent committed its work and reported `done`. Then: 1. The provider checkpoint made a filtered copy of the provider directory inside the sandbox (`tar -cf archive; tar -xf archive -C snapshot`) to apply its exclusions. On the full disk, tar failed. The run recorded `runner_remote_checkpoint_snapshot_failed` without the reason, and the partial copy stayed on the disk. 2. Workspace finalization failed three times, and file preservation failed six times. The capture builds its archive inside the sandbox, so each attempt failed. Every failure was recorded as `remote_workspace_operation_failed` / `export_interrupted`, so nothing named the full disk. 3. The task went to `blocked`. The only copy of the commit was in the retained sandbox. **Expected behavior** The checkpoint does not need free sandbox disk when the provider can stream. A workspace export that cannot fit says that the sandbox disk is full. **Steps to reproduce** 1. Run a native agent with a provider whose checkpoint has exclusions (for example ACPX) in a small remote sandbox. 2. In the turn, fill the sandbox disk (for example `pnpm install` of a large repository, or `fallocate` of the free space) and finish the task. 3. The checkpoint fails with `runner_remote_checkpoint_snapshot_failed`, and every workspace export attempt fails with a generic code. **Paperclip version or commit** Observed on `410b0e27015`. This change is based on `5955dd4eb4c`. ## What Changed - Synchronize the shared-principal response-recovery regression with its saved comments and empty final deadline. Cover an in-flight discovery invalidation. Keep every authority lock, exact publication, error, and no-retry assertion. This changes test synchronization only. - `syncRemoteRunnerDirectoryOut`: when the in-sandbox checkpoint copy fails with a full-disk tar diagnostic, remove the partial copy and pass the exclusions to the provider sync-out (`exclude` on the directory mapping). The Daytona sync-out streams the directory without sandbox disk when it cannot write its archive. Other copy failures and timeouts still throw `runner_remote_checkpoint_snapshot_failed`. They now also remove the partial copy. - `remote-workspace-transport.ts`: the helper install and the capture scripts report `remote_workspace_sandbox_disk_full: ...` when Node reports `ENOSPC` or `EDQUOT`, or a Git or tar child reports "No space left on device". `command()` passes only that exact message through, like the existing unfinished-Git-operation message. - `native-workspace-finalizer.ts`: a remote save that fails with that message adds it to the failed operation's stderr. The failure code, the retry budget, and the recovery flow do not change. ## Verification - Current-head infrastructure interruption: the [Build job](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125923826158) completed the actual build and production bundle boot, then the runner received a shutdown signal during the clean public Grok npm-install check. GitHub marked that step cancelled. The exact host-stop cause was not retained. One bounded rerun of only the failed Build and dependent `ci / verify` jobs passed on the unchanged head. [Build attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125931400745) completed successfully, including the clean Grok npm-install check. No product code, assertion or workflow changed for this interruption. - CI failure retained: commit `6c5937fcae` failed [chat shard 1/3](https://github.com/paperclipai/paperclip/actions/runs/41634638570/job/125913957217) because the shared-principal fixture saw zero saved comments after `worker.ready` and zero active work. The other chat child passed. The exact case passed once locally and then 30 times on the same Node 24.22.2 version as CI; those passes alone do not explain the CI interleaving. - Controlled RED/GREEN: invalidate the real database discovery once while its read is in flight. The worker correctly retains a future reconstruction deadline, but the old initial-pass/active-count barrier reproduces the same zero-versus-one comment assertion. Waiting for both saved comments and final `active=0`/`nextWakeAt=null` passes. Six real-PostgreSQL controls pass on final commit `a60e5629e8`, including endpoint independence, source revocation, and external lock failure/recovery. No sleep, timeout increase, weakened assertion, or product scheduling/authorization change was added. - A local server typecheck first lacked the checkout's built Runner declarations. After the declared Runner TypeScript build, the server typecheck passes on `a60e5629e8`. Final-head remote verification is complete: 54 successful checks, 3 expected skips, no failures or pending checks, and successful `ci / verify` and `ci / e2e` aggregates. Greptile is 5/5 on this exact commit with no unresolved review threads. - Merge preparation: rebased onto master `b22ef22d56` and kept both test-import additions. Added the checkpoint fallback and capacity-failure contract to `doc/execution-semantics.md`. On rebased commit `6c5937fcae`, 47 focused checkpoint, transport, finalization and retained warm-projection tests passed across three files. The name filter skipped 1,040 unrelated cases. Map and whitespace checks passed. The final-head CI and Greptile results are recorded above. - `npx vitest run server/src/services/native-runtime/native-session-executor.test.ts server/src/services/native-runtime/native-workspace-finalizer.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts` passes (1,152 tests). - `npx vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts` passes (30 tests). - New tests: - Checkpoint, with real files: a full-disk snapshot leaves a partial copy. The provider then gets the exclusions, the session files are kept, the package caches are left out, and the partial copy is removed. - Checkpoint: a permission failure and a timeout still fail, remove the partial copy, and do not call the sync-out. - Transport: the real capture scripts run with an `ENOSPC` preload, for the helper install and for the archive write. Both report the full-disk message and leave no partial attempt. A message with extra text still collapses to `remote_workspace_operation_failed`. - Finalizer: the operation stderr names the full disk and keeps `workspace_sync_out_failed`. Other failures keep the bare code. - With the source change removed, the 3 checkpoint cases and the 2 transport ENOSPC cases fail. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Suites that refer to the changed error codes (`explicit-native-continuation`, `native-recovery-notice-work`, `native-run-finalizer-telemetry`, `native-workspace-export-retry`, `remote-workspaces.integration`) pass. In `chat-channels.integration`, 2 cases fail locally on unmodified `master` too (a bundled skill path outside the checkout, and a chat run timing assertion). They are not related to this change. - Manual recovery of the observed task: the retained sandbox was resumed with free space, file preservation was requested again, and the saved workspace contained the agent's commit. ## Risks - The fallback runs only after the snapshot fails with a full-disk diagnostic. The normal path does not change. - Provider exclusions are not anchored to the directory root. In the fallback, a nested entry that has an excluded name (for example a nested `tmp` or `config.toml` in a Codex home) is also left out. The fallback never copies more than the snapshot, so credentials stay out of the checkpoint. - The fallback depends on the provider honoring `exclude`, which the sync protocol requires. The Daytona and Kubernetes providers honor it. - A workspace export on a full disk still fails. This change only names the cause. A full fix needs one of these: free-space reserve for the export, removal of re-creatable caches before the export, or a capture that streams its archive. Each of these changes behavior, so this pull request does not include one. - No migration. ## Model Used - OpenAI Codex, GPT-6 family, assisted with merge preparation, conflict resolution, documentation, test synchronization and verification. The harness does not expose the exact serving model ID or context-window size. - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). It did the root-cause analysis from run events, tool output, and sandbox inspection, wrote the code and tests, and ran local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com>