mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandPaperclip
2f47d0fadb
## Thinking Path
> - Paperclip preserves agent work when its control plane changes.
> - Workspace imports capture and verify portable filesystem archives.
> - The archive function uses asynchronous I/O but hashes and handles
manifests on the caller's JavaScript thread.
> - Concurrent imports therefore share one CPU execution context for
that work.
> - This pull request runs the same archive function in bounded native
Node workers.
> - The parent keeps all database, storage, source locks and publication
decisions.
## Linked Issues or Issue Description
**What happened?**
Concurrent administrative workspace imports can spend many minutes
capturing archives while the owning Node process uses about one CPU
core. Promise concurrency does not move hashing and manifest work off
that process's JavaScript thread.
**Expected behavior**
Independent archive calls use separate CPU execution contexts. At most
eight archive workers run in one process. Cancellation must wait for
file cleanup and actual worker exit before the caller releases its
locks.
**Steps to reproduce**
Run several workspace captures in one process. The existing wrappers
call the same archive function on the parent thread. The regression
holds eight real worker threads at a CPU barrier while the parent
remains responsive, then cancels a queued ninth call without spawning
it.
**Deployment mode**
All server callers of the portable capture and restore wrappers. Related
import work: https://github.com/paperclipai/paperclip/pull/20311 and
https://github.com/paperclipai/paperclip/pull/20394.
## What Changed
- Run the existing self-contained archive function in native workers.
Its algorithm and format stay byte-identical.
- Limit active workers to eight per process, with a 1 GiB old-generation
heap limit each. Workers stop when their owned call finishes.
- Forward cancellation to the worker's AbortController. Wait for actual
exit before returning, throwing or releasing an active slot. Cancel
queued calls before they spawn.
- Own capture scratch in a private parent-created worker temporary root.
Remove it after actual exit, including abnormal exit.
- Keep direct archive algorithm fault tests on the canonical
implementation. Add real-worker byte, manifest, restore, CPU barrier and
cancellation tests.
## Verification
- Four real-worker regressions passed: forced-exit capture scratch
cleanup; exact capture bytes/manifest and restored files/modes/links;
eight CPU barriers plus queued abort; abort cleanup joined through
actual worker exit.
- Canonical workspaceArchive declaration/body hash is unchanged from the
base:
`d6cffeab112b817fab8f65a3c0d8ecdc8d23589d37bf44ae30fe9e53213d54f4`.
- The combined archive run passed 114 cases. Three Unix-socket cases
failed because the sandbox denied listen with EPERM; a focused retry
with socket permission passed all three. The initial separate worker run
passed all three cases; the new head passed all four, including
forced-exit cleanup. All server layered typechecks and module boundary
checks passed.
- The initial ac4bc5760d source passed a synthetic smoke on staging’s
Linux Node v24.22.2 through the installed tsx loader: direct/worker byte
and digest parity, worker restore of three files plus a symlink, and
private fixture cleanup. It made no database, object-storage or provider
calls.
- Greptile reviewed final head
`3a51fe67ff17a56b2ddd7877c89734ba60afd18e` at 5/5 with no actionable
issues. All applicable final-head CI passed, including aggregate verify,
typecheck, build, workspace consumer suites and all server shards. The
second review remains queued. No measured migration speedup claim is
made yet.
## Risks
Each active worker adds a bounded V8 heap and native overhead. Eight
workers can use more memory and I/O bandwidth than the former single
JavaScript thread. The existing 64 MiB manifest limit and streaming file
buffers stay in place. Worker failures refuse the call and cannot
publish authority. The canonical restore staging path stays unchanged;
this PR does not claim cleanup after every arbitrary process crash.
Cooperative cancellation retains the existing cleanup path; the parent
waits for actual worker exit. No database, key, object-storage or
provider behavior changes.
## Model Used
OpenAI Codex, based on GPT-6, with tool use and code execution. The
exact deployment identifier and context-window size are not exposed in
this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>