0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-09 16:02:36 +01:00

Files

T

DottaandPaperclip c6573962f8 fix(workspaces): recover full sandboxes through read-only export (#19465 )

## Thinking Path

> - Paperclip preserves agent work across failed runs and server
restarts.
> - A remote workspace must have a durable save before its source can be
released.
> - Portable capture normally writes a Git mirror and a full archive
inside the sandbox.
> - A full sandbox can prevent capture before the provider's streaming
download starts.
> - Retrying that write cannot create the required free space.
> - This change transfers a checked standalone source and seals its
archive on the controller.
> - The source remains retained unless the existing durable save
completes.

## Linked Issues or Issue Description

Related: #18644 reduced temporary capture space. #18172 added the
provider's disk-free directory transfer. This repair connects portable
capture to that transfer when the archive itself cannot be created.

**What happened?**

Workspace capture returns `remote_workspace_sandbox_disk_full` after
provider work has finished. File recovery repeats the same sandbox
write.

**Expected behavior**

Recover a stable standalone workspace without requiring another full
archive on the source disk. Preserve staged changes, untracked work,
confined links and the original source.

**Steps to reproduce**

1. Create a standalone Git repository with a commit, staged changes and
an untracked result.
2. Use the existing real-command transport fixture. Inject ENOSPC into
the sandbox helper write or archive write.
3. Run portable capture. Before this change it rejects the helper
failure.
4. With the change, restore the saved archive after removing the fixture
source. Check the commit, staged content, untracked result and Git
connectivity.

## What Changed

- Attempt one controller-side capture after an exact sandbox disk-full
failure.
- Read a source inventory without installing a helper. Use the existing
provider bulk transfer with confined links and runtime exclusions.
- Compare the transferred inventory with source fingerprints before and
after sealing.
- Reject copied Git pointers, alternates and administrative symlinks.
Parse config as data without includes; allow only portable keys before
controller Git normalization.
- Preserve socket exclusions. Retain the source and remove incomplete
controller staging after failure.
- Update execution semantics and focused regression coverage. No
database migration or new execution authority.

## Verification

- TDD red on base `2f35eb1080`: the new full-sandbox test failed with
`remote_workspace_sandbox_disk_full` before production edits.
- Green: 202 transport and archive tests passed before the additional
Git configuration guards. At head `8541086cb5`, all 116 transport tests
and the PostgreSQL durability acceptance pass. The final `e24e173ec9`
exclusion guard passes those 116 cases plus its new Git-history
regression across the full run and focused rerun. Environment: macOS
arm64, Node 26.4.0; real Node/Git commands and filesystem transfer.
- Review regressions were also red before their fixes: an internal Git
config symlink bypassed the check, and `core.attributesFile` could
select a controller path. Git administrative symlinks and nonportable
config now fail before normalization. The focused safety selection
passes 11 cases.
- Cases cover helper and archive ENOSPC, commit/staged/untracked
continuity after source removal, confined links, sockets, changed
source, corrupt or extra copied files, unsafe Git metadata, retained
diagnostic text and failed download.
- Real PostgreSQL acceptance passes: normal workspace save recovers one
sandbox ENOSPC, commits the encrypted snapshot, clears its writer, loses
the original allocation, then independently reloads the saved result
twice. The first acceptance attempt used an incorrect expected status
(`available`); the documented state is `committed`. No production change
was needed for that correction.
- Review regressions for untracked Python/Node environments and
excessive Git roots failed before their fixes. Dependency exclusions now
match normal capture. Controller parsing starts only after
whole-tree/repository budgets and layout checks; refspec/settings
budgets and a shared two-minute verification/normalization deadline
bound further work. The excessive-root test verifies zero Git
subprocesses.
- Real provider-boundary red/green: Daytona direct tar and
capacity-fallback PTY transfers first dropped `._result.txt`. The new
optional mapping contract preserves all user names and excludes only
current transfer scratch. Kubernetes implements the same contract.
Explicit caller exclusions and archive validation remain in force.
- Standalone Linux qualification on Node 24.22.2/Debian bookworm: fresh
shared/SDK build; Daytona and Kubernetes typecheck and build; Daytona
file-sync/capacity/diagnostics 68/68 pass; Kubernetes file-sync 31/31
pass, including the actual `/proc` case. Qualification was refreshed
again on the final `be6f7feacc` source tree: current shared/SDK rebuilt;
all 815 staged source/config/manifest hashes match the delivered tree
before and after testing. An initial container run as root bypassed a
read-only file test; the same source passes as uid 1000. Installing zstd
enabled eight initially skipped cases.
- A further review regression proved `.git/pyvenv.cfg` could exclude Git
history as a dependency. The inventory now handles `.git` before
dependency filtering, matching normal capture. The full capture/restore
test verifies the original commit and saved work. The first green
attempt incorrectly compared command-result object identity; correcting
the test to compare stdout passed with no production change.
- Further red/green regressions cover read-only private-copy cleanup, a
regular file replaced by a FIFO before open, case aliases of Git
administrative metadata, and permission preservation under restrictive
umask. Nonblocking no-follow file handles verify identity before and
after reading. Cleanup changes only private controller directories.
Provider extraction preserves ordinary permissions, refuses special mode
bits before writing, and does not adopt source ownership, ACLs,
attributes or flags. Final macOS Daytona tests pass 65/65; the final
Linux qualification above was repeated on these exact provider sources.
- Focused strict TypeScript compilation of the inventory and archive
modules passes.
- The first sandboxed archive run failed three Unix socket fixtures with
EPERM. The permitted run outside that filesystem sandbox passed all
tests.
- Initial full local server typecheck could not resolve the unbuilt
Runner package and produced cascading errors. Exact-head `e24e173ec9` CI
typecheck, build, Runner and browser checks pass. That CI attempt failed
two server shards: the unchanged retained-cleanup backlog test timed out
at 15 seconds and passes locally unchanged; ten cache-reclaim fixture
oracles failed because directory fallback consumed their sealed-archive
counters/faults. Final commit `caed4e4f83` isolates the intended
unavailable-directory boundary while preserving archive counts, recovery
status, integrity and no-second-reclamation assertions. Those ten
failures reproduced locally before the fixture edit. The full affected
file passes 394/395; its sole other HTTP socket-hang-up case passes
unchanged in an isolated rerun. This is combined coverage of all 395
cases, not a single complete green run. The final branch integrates
master `53645a0104` at `98ea07225592930cbb0d18bcf250bc64f6bb13b4`,
preserving upstream workspace phase timings. After integration: 117
transport cases, the PostgreSQL durability acceptance, the existing
object-storage/timing regression, and 12 affected cache cases pass. The
expanded recovery file now has 421 cases. Final-head
`be6f7feacc45c1e32a26a2417e973f1e3752fd8d` CI passes all 55 applicable
checks (three skipped, one neutral review), including the expanded
recovery suite, typecheck, builds and Runner coverage. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42251345554).
- Provider exclusion regressions were red through four real Linux
transfer paths: Daytona direct archive, direct safe-link repair,
capacity PTY repair, and Kubernetes streaming. A root socket and nested
regular file share a basename; literal glob-character files/subtrees
also survive. Preserve mode now anchors tar exclusions as literal
root-relative paths and escapes `find -path` pruning. Default callers
keep glob semantics. Receipt-count regressions were also red before
correcting preserved-output counting. All 31 Kubernetes and 68 Daytona
tests, builds and typechecks pass on the final source with 815 matching
inputs.
- The final semantics documentation explicitly records special mode
bits, the stricter config allowlist (including innocuous unlisted keys),
and provider qualification limited to Daytona/Kubernetes; these address
the optional current-head review observations without broadening
controller Git authority.
- No Daytona credential is available in this session. Provider transfer
is a local filesystem boundary in these tests. Hosted task recovery and
rollout remain unverified.

## Risks

- Recovery reads the source repeatedly and transfers the standalone
tree. It can take longer than normal capture.
- Linked worktrees, initialized submodules, Git administrative symlinks,
nonportable Git configuration, special mode bits and alternate object
stores remain retained if the normal capture cannot finish. This
fallback does not follow their pointers onto the controller.
- A provider without usable directory export, a changed source, or
insufficient controller space cannot produce a receipt. The original
source stays retained.
- The server, SDK and affected provider plugins must be built from this
revision together. An older provider can fail inventory verification on
names it omits; it cannot publish an incomplete save. No merge,
publication or deployment is included.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, code editing, shell
tools and automated tests. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-06 11:00:32 -06:00

src

fix(workspaces): recover full sandboxes through read-only export (#19465 )

2026-11-06 11:00:32 -06:00

CHANGELOG.md

fix: bound workspace Git scans (#11572 )

2026-08-17 22:11:30 -05:00

package.json

feat: configure company agent observability (#16874 )

2026-10-19 15:36:04 -05:00

README.md

fix(workspaces): size archive deadlines and preserve resume budgets (#19364 )

2026-11-05 16:09:32 -08:00

tsconfig.json

…

README.md

@paperclipai/adapter-utils

Shared utilities for Paperclip adapters: process spawning, environment injection, sandbox/SSH transport, workspace sync, and the round-trip helpers that move code between the local execution-workspace cwd and wherever the agent actually runs.

For the adapter-author guide see docs/adapters/creating-an-adapter.md and the in-repo notes at packages/adapters/AUTHORING.md.

Sandbox bridge deadlines

Command-managed bridge control operations (queue reads, input delivery, setup, and cleanup) have a host-enforced deadline of at most 30 seconds per shell command. A shorter configured timeout still applies. These operations cannot inherit the agent run's hours-long lifetime or wait forever on a provider that ignores its timeout. Long-lived agent session commands keep their own limits.

A control timeout reports a transport failure. It does not prove that the remote command stopped and does not replay an uncertain write. The process session closes failed input delivery and uses its existing shutdown path; normal execution settlement must still verify termination.

Native archive transfer deadlines

Native syncIn and syncOut calls accept an optional minimumTimeoutMs budget. The server validates this budget and extends the worker RPC deadline. Providers can honor this field for file transfers without changing their configuration. Daytona uses the larger of this minimum and its configured timeout. Other providers keep their existing behavior. The minimum is limited to 30 minutes and does not change sandbox lifetimes, stored settings or the timeout for other commands.

The remote-workspace archive path uses the verified capture size for downloads and the local archive size for uploads. Large archives get one second per MiB plus 60 seconds, capped at 30 minutes. Small archives keep the existing provider timeout when that calculation is no greater than five minutes. Archive size, checksum, confinement and host space checks still apply.

An exact retained-sandbox resume keeps its recorded provider timeout and RPC overhead. It uses the normal one-minute fallback only when no timeout was recorded. A five-minute cold restore must not be cut off by that fallback. Resume still targets the recorded allocation and does not acquire a replacement.

No-remote-git contract

The local execution-workspace cwd is the only persistence boundary across runs. No adapter may depend on a git remote for cross-run state.

Adapters that run the agent on a different host should use the SSH round-trip helpers in src/ssh.ts:

prepareWorkspaceForSshExecution({ spec, localDir, remoteDir }) — bundles the local cwd (tracked files, dirty edits, untracked additions, and the git history needed to reconstruct it) to remoteDir before the run starts. Runs with no git remote configured.
restoreWorkspaceFromSshExecution({ spec, localDir, remoteDir, ... }) — syncs the remote cwd back into localDir after the run, including any new commits the agent created. Also runs with no git remote configured.

prepareRemoteManagedRuntime in src/remote-managed-runtime.ts wraps both calls for adapters that want a per-run remote workspace and an automatic restoreWorkspace() finally hook.

The invariant is pinned by the no-remote-git contract case in src/ssh-fixture.test.ts, which asserts that a remote-only commit propagates to the local worktree through the prepare → restore round-trip with no git remote configured at any point. Do not regress that test.