mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
dfd2a6efcab84bc19106a2dbf720eb53cb2282d3
36
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
e01fe7f84a |
perf(workspaces): reuse duplicate cohort source archives (#20574)
Capture identical live sources once within a migration cohort and reuse the completed archive for per-workspace encryption and upload. Bound shared scratch to one capture key per root and retain existing source, journal, and publication guards. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bf0220d34a |
perf(workspaces): migrate with one archive and no readback (#20554)
## Thinking Path > - Paperclip coordinates agents and keeps their workspace files. > - A legacy workspace must move from a local volume to object storage. > - The importer creates an archive, downloads and restores it, and captures the source again. > - Parent imports also download and recapture children that already have committed snapshots. > - This repeated work makes the one-time migration slow. > - This change copies each source file once and trusts the upload acknowledgement. > - Existing source locks and atomic publication still own the migration. ## Linked Issues or Issue Description **What existing behavior does this improve?** The one-time migration of legacy local and Git workspaces to durable snapshots. **Current behavior** Each workspace is captured twice and restored as a test during migration. Parent imports also read back already imported child snapshots. Capture hashes source files before writing them and scans them again afterward. **Proposed behavior** Build one archive from one source-file read, encrypt it, upload it, and publish its reference. Trust the upload acknowledgement. Reuse journaled uploads after an interrupted process. Retain committed child snapshots by database reference. **Reason and benefit** Remove repeated source reads, temporary copies, complete downloads, and recursive child verification. The tests assert one capture, one upload, and zero migration downloads. **Breaking changes** The archive format stays compatible. File hashes required by current readers are computed inline during the copy. There is no preliminary file-hashing pass. Existing readers can cold-restore the result. Related work: #19817 and #20447. ## What Changed - Add single-pass capture that fills fixed-length checksum fields in the existing archive header after copying. - Remove migration readback, test extraction, fresh-source recapture, and Git integrity scans. - Compute the encrypted archive checksum while writing it. - Reuse acknowledged and journaled uploads without rebuilding or downloading them. - Retain already imported children through their database bindings. Remove the recursive payload verification cache. - Keep company checks, source and writer locks, recovery bindings, journal identities, cancellation, and atomic publication. - Bind archived repository intent paths to the operation without reconstructing an archive to check them. - Update the migration contract and the affected tests. ## Verification - Node 24.22.2 and pnpm 9.15.4 on macOS arm64. - At |
||
|
DottaandPaperclip
|
0934b834c4 |
perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path > - Paperclip manages agent work and preserves its workspace files. > - Online migration captures local files and publishes remote workspace authority. > - Each capture verifies encrypted readback, restores the archive and compares the current source. > - A cohort kept every completed parent payload until the whole cohort published. > - A larger cohort could retain many complete copies while later members still worked. > - This pull request releases only finished parent payloads before a capture worker starts another member. > - Nested-child proof, final fresh checks and atomic publication keep their existing owners. ## Linked Issues or Issue Description Refs: #20418, #20437, #20441. **What happened?** A cohort retained each member's capture archive, verified archive, comparison archive and restored tree until atomic publication. The capture worker limit did not limit that retained parent data. **Expected behavior** Release completed parent payloads once their consumers finish. Keep files that a later publication check still needs. **Steps to reproduce** Run the added held-peer regression against the previous owner. It completes one parent, holds a later member and finds all four completed parent paths still present. ## What Changed - Release three named parent archives and the restored tree after a successful online cohort capture, before that worker takes another member. - Keep parent scratch roots and all nested-child verification areas. Keep every final fresh child and publication check. - Retain payloads for archived repository intents. Use the existing writable-tree cleanup for read-only restored directories. - Add real filesystem regressions for early release, retained child proof, late child mutation and the final archived-intent path check. Update the source-fence guide. This branch incorporates the final #20441 owner and both distinct saved-relative-path regressions. Its scratch-specific production change is nine added lines and one removed line. Single imports and saved journal/archive formats stay unchanged. ## Verification - Canonical Node24 preflight passed. The Runner build came from the content cache. After the current-master merge, the preflight checked 1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin SDK prerequisites passed. - Genuine RED on the unchanged #20441 owner: one held-peer case failed at four retained payload paths; the archived-intent control passed. The run took21.14s, with221 unrelated cases intentionally unselected. The first sandbox-restricted setup attempt hit DNS errors and was stopped; the normal network-qualified preflight then passed. - Initial treatment: four selected PostgreSQL/filesystem cases passed in38.61s, with219 intentionally unselected. The held peer observes released parent payloads and retained child areas. A later child mutation still refuses the whole cohort and publishes no authority or activity. - The archived-intent control observes all parent payloads at the actual final saved-relative-path check. The changed path refuses publication and keeps task/repository state intact. This exercises its actual singleton owner; archived-only sources do not enter the online cohort API. - After a normal merge of current master and the final #20441 test, the five affected cases and server TypeScript passed. No full local matrix was repeated. Initial CI failed the inherited recovered-upload assertion because the new retention test leaked its archive and filesystem spies. That case passed alone but failed in the adjacent three-case sequence (two pass, one fail); scoped `finally` cleanup made the same sequence pass three of three in17.15s, with191 unrelated cases intentionally unselected. No upstream assertion or production code changed. Server TypeScript passed again. All required [CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511) passed on final normal-merge head `604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and one neutral; no pending or failing checks). The previously failing server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with no open threads. The merge resolves the duplicate #20441 squash insertion; both tested owner and test files are byte-for-byte identical to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior head received [Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885); its archived-cohort coverage warning was [answered using the actual public admission boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017). - `git diff --check` passed. No hosted cleanup, migration, provider action or deployment was performed. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication' pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reduces retained parent data. It does not establish a total disk bound. Active capture/encryption/readback files and retained child areas still need measured capacity. - Cleanup failure uses the existing worker abort and peer-join path before publication. Original source files, uploaded objects and journals remain intact. - Archived repository intents still need their restored payload at publication. Those payloads stay retained. No capture limit, budget or integrity guard changes. - Saved operations and archive identities carry forward unchanged. Resume still uses the existing full source and authenticated object checks. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0991d9bda1 |
fix(workspaces): protect migration sources and add guarded recovery service (#20443)
## Thinking Path > - Paperclip manages agents and their task workspaces. > - Online migration saves files before it publishes remote workspace authority. > - A saved migration journal can survive the importing process. > - Git cache expiration did not count that journal as a consumer. > - Expiration could remove a source and change its fingerprint while its import was incomplete. > - This PR protects unfinished imports and exposes a guarded internal service transition for an already affected cohort. > - New imports can preserve the current sources while all old recovery data stays retained. ## Linked Issues or Issue Description **What happened?** Git cache expiration could archive and remove a workspace after an import process ended but before its saved cohort finished. The saved source fingerprint then differed from the current archived workspace. Exact cohort resume correctly refused that changed source. Reopening the workspace cannot recreate the old metadata or fingerprint. **Expected behavior** An unfinished migration remains a durable consumer after process loss. If an older server already expired some sources, one canonical transition can fail the exact unpublished cohort without losing its snapshots, objects, source history, or recovery obligations. New attempts use the current verified sources. **Steps to reproduce** 1. Start a cohort import of eight Git worktrees with local changes. 2. Interrupt it after four snapshot uploads. 3. Let an older server expire two idle source worktrees. 4. Resume the saved cohort. It refuses the changed archived sources. Related: #20429 preserves exact cohort identity when independent companion imports change only the scope proof. #20424 preserves prepared recovery state for archived sources. This PR handles completed source expiration, not a scope-only change. Public searches found related expiration work and no matching compensation owner. ## What Changed - Count saved migration markers and unfinished migration journals as Git expiration consumers. Keep failed migration retention holds as consumers too. - Expose `supersedeExpiredRemoteWorkspaceMigrationCohort` as an internal service API for a privileged operator with a real database handle and reviewed company/cohort inputs. It is called directly by the PostgreSQL/Git test and reviewed one-time administrative qualification. It adds no CLI mode or HTTP route. The existing CLI still refuses a damaged cohort; this PR does not claim automatic CLI recovery. - Require an exact unpublished cohort of up to eight members. It requires completed expiration and an unexpired restore recipe for each missing member. The other members must retain their saved source fingerprints. - Recheck current source and recovery state under source, lifecycle, finalizer, and row locks. One transaction fails all old journals, writes audit records, and removes only their exact matched active markers. - Keep old cohort proofs, operations, snapshots, and object references. Genuine failure receipts use the existing failed-save retention hold. This transition does not report migration success. - Reuse the existing bounded source-lock queue when `sourceLockWaitMs` is set. Keep fast refusal by default. Cancellation targets only this waiter, and current checks still run after admission. - Record predecessor IDs on new attempts and reserve a snapshot generation above the existing ledger maximum. New success does not retire the old hold. - Document the owning transition and retention rules. ## Verification - The two new journal-consumer cases failed under the old expiration predicate and passed under the new predicate. - The full new recovery case passed in 21.12 seconds with real PostgreSQL locks, Git repositories, encrypted local object storage, and archive restores. It covers four retained uploads, two real expirations, unchanged old resume refusal, authority change refusal, transaction rollback, a held native finalizer, idempotence, eight new imports, dirty/untracked file restores, and a retention pass 30 days later with no object deletion. - Server TypeScript and module-boundary checks passed. - On `7cec8c915acfad2d08874eaf26abaa852f560219`, the queued-admission regression failed before forwarding the option (24.07 seconds), then passed (22.25 seconds). It verifies a real held project lock, cancellation that preserves the holder and all journals, release into full compensation, retention, and eight restored fresh imports. Current server TypeScript and module/diff checks passed. The only production follow-up changes are the optional queue input and its existing-owner call. - The earlier `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c` CI run had one Runner support failure: the unchanged credential-free Python/Pi calibration exceeded its outer 25-second subprocess deadline. That job passed 6,926 other tests and skipped four. The timeout is retained as a nonpass. The changed logical head requires fresh CI and review; no test deadline was raised. - The affected cohort, Git expiration, and eligibility suites passed all 148 cases in 228.41 seconds on `aa589f3f447e36d0277c32f8b44cbe5e84a673be`. Merged current master `a9d0946e64`; the only conflict retained both independent added test blocks. The complete merged cohort suite passed all 30 cases in 70.00 seconds on `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c`, including upstream larger-cohort controls. Final-head server TypeScript, incremental preflight, module-boundary, and diff checks passed. Unchanged expiration/eligibility evidence remains valid. Fresh `809ab704e619812d8f1bf38cf5de9c65893861cc` CI and review are pending. - On `809ab704e619812d8f1bf38cf5de9c65893861cc`, the complete recovery fixture passed in 21.38 seconds after adding one valid archived recipe with no restore point. It refuses with `migration_supersession_expiration_unverified` and keeps all workspaces, journals, and snapshots unchanged. The same case then verifies queue cancellation/release, full compensation, retained history, fresh imports, and restored bytes. Server TypeScript passed. This follow-up changes only the test and service-scope documentation; product bytes remain identical to qualified `7cec8c915acfad2d08874eaf26abaa852f560219`. - Earlier fixture attempts did not pass. They exposed the selection of a held source and an incorrect expected refusal string. The fixture now selects eligible sources and asserts the canonical authority refusal. Production guards were not relaxed. - Live qualification of the exact owning product bytes completed in 230.419 seconds. All eight old operations became failed with auditable supersession lineage. All eight old snapshot rows remained byte-for-byte identical, and their object references remain retained. The existing source-lock queue waited 1.151 seconds for a normal finalizer handoff. The three complete source/recovery passes took about 218 seconds including their transaction, which explains why the earlier 180-second total budget was insufficient for that measured workload. No source proof or publication guard was skipped. Compensation made no provider or object-storage calls. - Earlier live qualifications remain nonpasses: the 180-second attempt did not finish; its instrumented successor completed source checks and refused a busy source lock before mutation. The successful bounded-queue attempt retained those failure records. Fresh archived/online imports are a separate subsequent step and are not claimed complete here. - The local checks do not claim browser acceptance, app deployment, fleet enrollment, or a completed zero-downtime cutover. ## Risks - This owner mutates migration bookkeeping. It preserves all-or-nothing row comparisons and current source/recovery locks. A changed journal, authority, writer, finalizer, expired recipe, or partial transaction refuses the transition. - Historical snapshots and objects remain pinned after new success. This intentionally retains storage. Release requires a separate reviewed action. - An older importer cannot use the successor generation contract. Use the containing canonical owner for recovery. Normal app delivery must also carry the expiration guard; a private maintenance graph does not patch a running server's cleanup worker. - The migration owner crosses 1,000 lines. The bounded compensation transition remains beside its private source inspection, fingerprint, fence, reservation, and publication contracts. Moving it now would require exporting or duplicating those safety owners. No unrelated refactor or new framework is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source editing, Git, and local execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
29a66d679e |
fix(workspaces): verify archived repository intents before publication (#20441)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration moves local files into durable, authenticated snapshots. > - Archived Git workspaces can retain an unmaterialized repository request after their local tree is removed. > - Admission cannot prove filesystem absence from the removed tree. > - This pull request checks strict request candidates against the private restored Git tree and the authenticated snapshot readback. > - Publication rechecks current repository rows without changing the request or task history. ## Linked Issues or Issue Description **What happened?** An archived workspace with an authenticated restoration recipe and an unresolved, unmaterialized repository request fails migration with `repository_preparation_unresolved`. Its original directory no longer exists. **Expected behavior** Migration can preserve the request when the actual private restored tree proves its target absent and every parent safe. It must refuse partial materialization, symlinks, or changed request state. **Steps to reproduce** 1. Archive a Git workspace with a valid saved recipe and a pending repository request. 2. Remove its original local workspace directory. 3. Run canonical workspace migration. The previous implementation refuses before it can inspect the restored tree. **Paperclip version or commit** Reproduced on `1ca3e8ea2a759e511ab2d2659339c6c770a7a2ba`. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:repository_preparation_unresolved` Refs #20416, #20424, #20436. These fixes retain their source, recovery, and ownership guards. ## What Changed - Separate strict repository request metadata from filesystem absence proof. - Admit only strict candidates backed by a canonical archived Git recipe to private materialization. - Verify target absence and safe parents before a new upload and after authenticated archive restore. - Lock and recheck current repository rows before authority publication, including recovered uploaded archives. - Preserve previous-release recipes, request rows, task selection, history, provider references, and unresolved recovery obligations. No persisted schema or public option changes. ## Verification - Genuine baseline RED: both pending and needs-repair archived requests fail on unchanged master. - GREEN: 27 focused real-PostgreSQL/FS/Git tests pass. Includes partial/symlink refusal before upload, request mutation during upload, recovered-upload readback, and existing native F/source checks. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'archived repository intent|unmaterialized repository request|unresolved repository request|pending repository|unselected archived recipe|archived prepared source|competing native F owner'`. - After rebasing onto current master, all six new cases pass on `0b6854789acc067a44e697acebdc04e51b9afe88`. Server layered typecheck, module boundaries, and `git diff --check` pass. A review follow-up adds one pending-path mutation regression (1/1 real-PG pass) without changing product code. Fresh required CI runs on final head `2a30a4b444108488ac8b51d0c46c340ee15227b9`; fresh Greptile is 5/5. Server7 had an unchanged trace-parent assertion failure; that same shard passed on the previous product-identical head and the linked-source sibling PR. The original failure remains recorded. The single targeted server7 retry passed. Final head has 56 successful checks, 4 intentional skips, and 1 neutral review status. ## Risks - Metadata admission is only a candidate check. A candidate still fails unless the private authenticated filesystem proves absence. - Publication keeps repository rows locked while it checks their current paths against the owned restored tree. A real pending-relativePath mutation now tests this final check. Existing workspace/source/native locks remain in place. - Absence is proved only in SAVED data reconstructed from the retained recipe and snapshot. This does not prove that a request never materialized before archival. Existing Git restore points can exclude ignored `.paperclip-repositories` paths outside recorded checkouts. This PR does not change that preexisting coverage or recover bytes that were never saved. - The original recipe, restore point and checkout references remain unchanged. No files are recovered into or removed from the original directory. No request is marked ready or restored. Partial target files present in reconstructed saved data remain blocked and retain their state. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
07f6e8ff75 |
perf(workspaces): group same-root cohorts with bounded workers (#20437)
## Thinking Path > - Paperclip preserves agent work when its control-plane server is replaced. > - Online migration imports local workspaces into authenticated snapshots. > - Several workspace records can point to one physical directory. > - Shared child verification avoids repeated work inside one held cohort fence. > - The eight-member limit splits 30 identical-root aliases into four cohorts and repeats the child passes. > - This pull request admits up to 32 members only for one canonical physical root, while keeping each work stage bounded to eight peers. > - The benefit is fewer repeated child checks while capture and publication guards stay in place. ## Linked Issues or Issue Description Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found shared verification and restart fixes, but no larger same-root cohort owner. The shared verifier and archived-source changes are merged into this branch from current master. **Current behavior** Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and 6. Each cohort performs an initial and final local child-validation pass. The same children are checked eight times in total. **Proposed behavior** Admit one cohort with nine to 32 members only when every canonical resolved physical root is exactly equal. Keep the existing complete source-scope equality and admission guards. Keep mixed-leaf cohorts limited to eight members. Bound source reads, reservations, recovery checks, final readers, replay and cleanup to eight joined peers. Capture concurrency stays at most eight. **Reason and benefit** For N distinct canonical covered children, the 30-alias case changes 8N local child checks to 2N, a 75% reduction in repeated child validation. If N is 68, that is 544 checks versus 136. Thirty parent captures, uploads and restores remain required. No total wall-clock improvement is claimed. **Breaking changes** Saved cohort fields and exact membership remain unchanged. Older tools that enforce the eight-member bound refuse larger cohorts. Use the newer tool to resume them; never split or regroup their saved membership. ## What Changed - Raise the canonical service, saved-cohort and maintenance-request member bounds to 32. Keep the capture-worker bound at eight and single-workspace modes at one. - Require one exact canonical physical root for more than eight members. Keep the existing full source-scope comparison, leaf rule and P/I/L/F guards. - Use one private stage mapper with batches of at most eight. Join every started peer before an error can leave the stage. Stop future batches after a failure or cancellation. - Preserve the existing cancellation and pending-snapshot ordering for stages of at most eight members. Check cancellation around extended batches. - Keep all saved cohort, member, operation and snapshot identities, proof refresh and atomic publication checks. - Bound cleanup without skipping later scratch cleanup after an earlier cleanup error. - Extend the existing PostgreSQL fixture with ten real aliases, partial-upload resume, cold restoration, replay and mixed-root refusal. Extend the request boundary test. - Document old-tool refusal, work counts and the retained-scratch capacity bound. ## Verification - Canonical Node24 worktree preflight and Plugin SDK prerequisites passed. After merging current master, preflight passed again with unchanged cached build inputs. - Genuine RED on the old bound: both real ten-alias import/resume cases and the 32-member request reject the eight-member limit (3 failed; 47 not selected; 18.23 seconds). - First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias case performs exactly four local child captures for two children: one initial and one final pass. Actual reader, transaction and upload maxima are eight. All ten archives cold-restore after the original source is parked; replay retains the same cohort. - The partial-upload case keeps all ten saved operation IDs and the existing snapshot IDs. More than eight members across different canonical leaf roots refuse before any journal or storage work. The request parser accepts 32 members and eight workers, and refuses 33 members or nine workers. - The first expanded corpus had 55 passes and one real compatibility failure: a new eager cancellation check skipped the old pending snapshot boundary. The mapper now retains existing cancellation ordering for at most eight peers. That failure remains recorded as a nonpass. - Final sources include current master at `18697b8d21`, with the verifier, archived-source and UI repairs. The selected four-file corpus passes 47 cases in 78.88 seconds, with 191 intentionally not selected. The full maintenance suite separately passes 23 cases in 10.21 seconds. Nine cases overlap; the final union is 61 distinct passing checks. - Final coverage includes all 27 cohort cases, all 23 maintenance cases, all six actual old-format compatibility cases and five archived prepared-source controls. Existing source mutation, finalizer contention, actual fence loss, cancellation peer joins, journal compare-and-swap and outer-commit rollback controls pass. - Final server TypeScript, module boundaries and whitespace checks pass. Only the resource-risk documentation changed after that test run; runtime and fixture bytes stayed unchanged. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/maintenance/online-workspace-migration.test.ts server/src/__tests__/online-workspace-migration-compatibility.test.ts`; archived controls use the existing `remote-workspace-migration.test.ts` cases named `unselected archived recipe` and `archived prepared source`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial source review was 4/5 and requested a control for the extended reader-batch stop boundary. The test-only follow-up `ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member failure and cancellation cases. They hold a first-batch peer, verify that the source fence stays held, and prove members nine and ten never start preflight. Both pass in 10.87 seconds with no new journal, snapshot or upload effects. Runtime source hashes are unchanged; the local union is now 63 distinct checks. - The first stop-test attempt observed the two controlled readers before all SQL-backed peers had entered the inspection callback. Its assertion was too early. The corrected test checks the held first-batch readers at that point, then requires all eight to join before completion. Both cases pass; the earlier attempt remains a nonpass. - Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664) and [aggregate verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765) passed. Full rollup: 56 successes, four intentional skips and one advisory neutral result. Classic Snyk passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693) both reviewed that exact head at 5/5. The one review finding is addressed and resolved; no open findings remain. Older partial CI and the initial 4/5 review remain nonfinal evidence. Hosted acceptance and measured wall-clock savings remain pending. ## Risks - Completed parent captures retain their scratch until the atomic publication completes. A 32-member cohort can retain up to four times the parent scratch of an eight-member cohort. Check actual parent capture, verification and restore sizes against current target filesystem capacity before selecting a larger cohort. Keep the existing resource guards. - Larger member arrays must not raise execution fanout. Every reader/reservation/final stage is bounded to eight and joins current peers before failure. Capture workers remain at most eight; child-verification workers remain two. - More than eight members require the exact same canonical physical root and complete source scope. A different root, new writer, changed source, recovery owner or journal still refuses through the existing owners. - Shared verification is limited to the held invocation fence. The separate fresh final pass remains required after all captures join. Different child sets and restart attempts do not share a local-byte proof. - The old saved membership and archive identities remain authoritative. Older tools fail closed on the larger bound. No schema, archive or provider format changes are included. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4608744977 |
fix(workspaces): preserve recovery for verified archived Git sources (#20436)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved Git files and pending native recovery. > - The archived-Git owner validates saved recipes, exact cleanup receipts, and surviving owned branches. > - Online recovery preservation currently accepts only the recipe variant. > - A valid cleanup receipt or owned branch therefore cannot migrate while an old native reference is still prepared. > - This pull request admits every verified archived-Git variant through the same recovery and finalization locks. > - Tasks stay done and keep their selected workspace, history, and pending provider obligations. ## Linked Issues or Issue Description Refs #20424, #20298, and #20418. The related cleanup work in #19812 does not cover this admission gap. **What happened?** Canonical inspection accepts the exact old commit of an archived Git workspace through a valid branch-deletion receipt or an owned surviving branch. Migration still reports `remote_copyback_unresolved` when a terminal native run retains a prepared reference. A read-only staging inventory reproduced this sole blocker for four such workspaces. Their original Git objects and canonical source proofs already exist. **Expected behavior** Preserve the verified Git baseline and every typed native/provider recovery obligation through the existing migration owner. Keep the task status, selection, history, archive state, and recovery records unchanged. Do not reopen the task or substitute the current default branch. **Steps to reproduce** 1. Save a clean archived Git workspace with an authentic cleanup receipt or a surviving owned branch, without a restore recipe. 2. Retain a terminal prepared native reference and its exact provider allocation and recovery action. 3. Remove the old task worktree. 4. Call `migrateRemoteWorkspace` online. **Paperclip version or commit** Reproduced on merged #20424, `915edcff49bb2cbacfbaee604b41e082e0397109`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Change three conditions to accept a non-null canonical archived-Git source instead of requiring its recipe field. - Use the same typed recovery inspector, sorted native finalization locks, fresh reinspection, and source-task publication locks for all accepted variants. - Reuse the existing Git source validator, private scratch materializer, encrypted snapshot restore, journal, and recovery markers. No source-proof schema or provider protocol changes. - Extend the existing real fixture for cleanup receipts and retained branches. Verify exact old Git commits and unchanged task/recovery state. Refuse receipt revocation and branch movement during upload. ## Verification - Genuine RED on merged #20424: both new source-kind cases fail with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - GREEN: nine focused PostgreSQL/Git/encrypted-storage cases pass. They include both new variants, two actual upload-time source mutations, the existing recipe interrupted-journal resume, reference/provider mutations, and a competing native finalization owner. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'no-recipe archived|unselected archived recipe|archived prepared source|competing native F owner'`. - All nine focused cases, server typecheck layers, module boundaries, and whitespace checks pass on exact published head `03065f97b31846a405a8911c30640ef62374e4a4`, after the clean rebase onto merged #20418. Required CI is now clean on this exact head:56 SUCCESS and4 expected SKIPPED checks, with no pending or failed checks. Fresh Greptile is5/5 with no actionable findings or unresolved threads. - Previous-release state: existing cleanup receipts, owned branches, prepared v3 references, and journal shapes carry forward unchanged. Saved snapshots and source identities retain their existing owners. Unrecognized or changed sources still refuse by the existing named errors. Migration does not declare remote files recovered or provider allocations stopped. - Hosted work in this PR was read-only source inspection and inventory. No staging import, source mutation, lifecycle reopen, or provider call was performed by this PR author. ## Risks The existing archived-Git validator now admits its other verified variants to recovery preservation. Incorrect source proof could publish incomplete authority. This PR keeps its exact commit/receipt/branch checks and full authenticated capture/readback. It also keeps all native ownership, reference, allocation, and publication checks. Missing or changed sources remain blocked, and pending remote data retains its original recovery obligation. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
18697b8d21 |
perf(workspaces): share cohort child-source verification (#20418)
## Thinking Path > - Paperclip must preserve agent workspace bytes during online migration. > - A cohort can contain several aliases of the same physical source. > - The importer verifies imported children before capture and again before publication. > - The immutable-object cache avoids repeated downloads but still recaptures local child bytes for every alias. > - This pull request shares the initial proof within the held cohort fence and checks fresh bytes once more after every capture joins. > - The benefit is less repeated local archive work while child mutations still block the whole cohort. > - Required browser CI twice reproduced an independent stale composer-mode bug. A focused regression proves that refreshed persisted task mode was ignored after mount. The state repair follows task data when there is no local override. Explicit local mode choices retain their own generation so a completed send cannot clear the next draft’s choice. ## Linked Issues or Issue Description Refs: #20311, #20387, #20401, #20404. **Current behavior** Two aliases recapture the same imported child's local bytes four times, even with one immutable object read. A larger alias cohort repeats this multi-GB work for each member. **Proposed behavior** Share the successful initial verification for the same physical root, Git roots and exact recorded child set or discovery mode. After all captures join, run a separate fresh final verification for each root and recorded child set. Keep every member's database and publication checks. **Reason and benefit** The local PostgreSQL fixture proves that two aliases now perform two child recaptures instead of four. Hosted timing improvement has not been measured. ## What Changed - Add invocation-local initial and final proof maps to the existing cohort child verifier. - Retain the existing canonical source-owner check before and after shared results. - Defer cohort final nested checks until all captures finish, and join every final reader before publication or fence release. - Keep singleton verification, saved journal schemas, snapshot IDs, archive bytes and per-member publication checks unchanged. - Extend the existing real PostgreSQL alias fixture with a child mutation during the last parent upload. - Repair the required CI planning-mode race with a nullable local override: absent override follows task data; a user selection belongs to the draft. A successful send or confirmed resend/receipt clears it only if no newer choice exists. Failed saves preserve the mode and restored message. Queued-edit backup retains whether the mode was explicit. Leave the browser assertion and timeouts unchanged. ## Verification - Canonical Node24 worktree preflight and plugin SDK prerequisites passed. Server TypeScript passed. - Genuine unchanged-source PostgreSQL RED: two aliases performed four local child recaptures instead of two (one failed, 22 intentionally not selected). GREEN: five source-sharing, last-upload child mutation, capture/cancellation join and actual outer-commit rollback controls passed (18 intentionally not selected; 66.53s). The actual fence-loss control also passed (one passed, 22 intentionally not selected; 38.13s). Both wrappers completed cleanup. - Migration source and tree bytes remain identical to locally qualified `466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair the composer and its tests. - Required planning browser CI originally failed, then failed its one unchanged-head retry, because the planning chip remained absent after opening-question dismissal. Those failures remain recorded ([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985), [retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)). The final-head identical case passes in 30.5s, with all 34 shard cases passing ([final CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)). Assertions and timeouts were not relaxed. - Composer regressions reproduced stale loaded mode, optimistic-save rollback, overwritten next-draft choices, and stale mode after confirmed resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI TypeScript and token gates pass. The original planning browser case also passed locally (15.5s; 26.9s including setup). - The first final-head serialized-suite attempt was automatically cancelled at its unchanged 20-minute job limit. Grouped Turbo output did not expose an individual failing assertion; cause remains unknown ([retained attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)). One same-head diagnostic retry was a cache miss and executed all 419 heartbeat recovery tests successfully in 376.94s on the identical CI checkout ([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)). It is not classified as infrastructure or a known flake. - Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61 checks terminal (56 success, four skipped, one neutral); required CI is green ([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)). Fresh Greptile is 5/5 with every review thread resolved ([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)). Focused commands: ```sh node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished" pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx pnpm --filter @paperclipai/server exec tsc --noEmit pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates ``` ## Risks - Reuse is limited to one invocation under the same complete source and finalizer fences. Different physical roots, Git roots, recorded child sets and discovery mode have different keys. No local-byte proof is stored durably or reused after restart. - Initial and final maps are separate. A child change during any parent upload must fail the fresh final verification before the atomic publication transaction starts. - Final readers all finish before an error can release the source fence. Per-member database child rows, source fingerprints, journals, keys and publication locks retain their existing checks. - This reduces repeated verification for aliases. It does not change capture concurrency, upload policy or immutable checkpoint validation. Hosted speedup remains unverified. - The CI repair must distinguish task data from explicit draft intent. Only a successful send with the same choice generation clears its override; a later draft selection and failed-save rollback retain it. Regression tests cover these transitions. It adds no styling, request, or persistence changes. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
915edcff49 |
fix(workspaces): preserve archived native recovery during migration (#20424)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved files and pending recovery work. > - An archived Git workspace can have an authenticated restore recipe and a terminal native run with pending remote files. > - Its task can select a different workspace. Reopening the archived workspace would change that task. > - The current migration owner refuses this source because it has no live local directory. > - This pull request uses the existing online recovery preservation owner for this exact archived recipe case. > - Migration retains the old task, provider allocation, recovery action, and native sync reference. ## Linked Issues or Issue Description Refs #20298, #20394, and #20418. Related cleanup work in #19812 does not cover this migration case. **What happened?** A historical archived Git workspace with a saved restore recipe and a terminal prepared native sync reference cannot migrate online. Admission reports `remote_copyback_unresolved` and `workspace_recovery_unresolved`. The old task can already select another workspace, so reopening it is not a safe migration path. **Expected behavior** Authenticate the archived files into private scratch. Preserve all typed native and provider recovery obligations in the existing migration journal and markers. Keep the original task selection, history, archive state, and restore flags unchanged. **Steps to reproduce** 1. Archive a Git worktree with staged, unstaged, and untracked files in a saved recipe. 2. Retain a terminal failed native run, its prepared sync reference, and its matching remote allocation and recovery action. 3. Remove the old local worktree and select a different workspace on the task. 4. Call `migrateRemoteWorkspace` without `writersStopped`. **Paperclip version or commit** Reproduced on `b2a7d322ed0bafda04a0b78ce14e4f25c8d5d170`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Permit the existing online recovery owner to inspect recipe-backed archived Git sources. - Acquire the existing native finalization locks for those sources. Reinspect their exact proofs after lock acquisition and before publication. - Lock related source-task ancestry before publication. - Reuse the existing private Git restore, authenticated archive, journal, marker, and recovery-source contracts. Add no schema or provider calls. - Leave active sources and archived sources without a valid recipe on their existing refusal paths. - Include the shared UI correction from #20418. A nullable draft override follows fresh task data when the user has no local choice. Explicit choices increment a generation. Normal and recovered successful sends clear only their own choice; failed sends and newer choices remain intact. ## Verification - Unchanged master RED: the new historical-source fixture fails with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - Five focused real PostgreSQL, filesystem, Git, and encrypted-storage tests pass on Node 24.22.2. The success case loses an upload reply, resumes the same operation and snapshot, restores the staged/unstaged/untracked bytes, and verifies unchanged task history, lease, recovery action, finalizer, and recipe flags. - Recipe, prepared reference, and provider allocation changes during the actual upload each refuse publication. A competing native finalization owner refuses before capture. - Migration command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'unselected archived recipe|archived prepared source|competing native F owner'`. - All server typecheck layers, module boundary checks, and whitespace checks pass for the migration revision. The migration source bytes remain unchanged through the shared UI follow-ups. - Initial CI run 42727051535 exposed an existing planning-mode browser failure: shard 3/8 passed 33 cases but the refreshed task mode did not appear. The same issue occurred twice on #20418. A real local regression proved that initial cached data could leave the composer in its old mode. This path does not invoke migration. - Review then exposed genuine defects in the initial shared UI repair: an optimistic save rollback could reset a failed draft, and an earlier send could replace a newer draft choice. Recovered delivery also left a consumed choice pinned. All three have real RED→GREEN controls. The final shared fix uses an explicit nullable override, a choice generation, and one attempt-bound settlement owner for normal and recovered success, with no value-equality inference or durable payload changes. - Exact shared UI source now passes all 145 composer tests (2.88s), the original desktop/mobile planning browser case (15.5s; assertions and timeouts unchanged), token gates, and the canonical UI typecheck. The test harness uses the full persisted `IssueWorkMode` type, including `skill_test`, to fix the observed CI typing failure. - These UI results are reused from the identical shared source in #20418 through `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`. The unchanged planning browser pass is on the explicit-state model before the later settlement-helper extraction. The final receipt/resend controls, full composer suite, typecheck, and token gates include that extraction. They do not claim a local browser run on this entire combined branch. Fresh required CI and review qualify this PR head. - Previous-release state: existing recipe, v3 prepared reference, and journal shapes carry forward unchanged. A saved partial operation resumes with the same IDs. Migration does not certify remote file recovery or provider termination. Live provider recovery and hosted migration remain separate acceptance steps. ## Risks The change admits one more source class into an existing preservation path. An incorrect proof could publish incomplete file authority. The canonical recipe validation, full authenticated restore, native finalization locks, source reinspection, and publication checks remain mandatory. Pending remote data stays pending and retains its original allocation and reference. The shared UI repair updates only an unchanged draft mode; its regression checks preserve a local user choice. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3040448da5 |
fix(workspaces): refresh saved cohort source proofs on resume (#20429)
## Thinking Path > - Paperclip preserves agent work when a control-plane server is replaced. > - Online workspace imports save exact cohort, operation and snapshot identities for restart. > - A cohort proof also includes the complete current companion source scope. > - An independent companion import removes that companion from the local scope. > - The saved cohort then fails resume even when every saved member is unchanged. > - This pull request rechecks the current scope under fresh locks and refreshes its proof for all members in one transaction. > - The benefit is restartable online migration while independent work progresses. ## Linked Issues or Issue Description Refs: #19873, #20394, #20418. Related PR search found the original cohort owner, recovery-proof refresh and separate child-verification reuse. This fix addresses cohort scope freshness. **What happened?** An interrupted cohort rejects resume with `workspace_cohort_membership_changed` after a nonmember companion is migrated. The saved source proof includes companion workspace and task IDs. The exact saved members and their fingerprints can remain unchanged. **Expected behavior** The same saved cohort must resume after fresh admission. It must keep all member, operation and snapshot IDs. Changed members, active owners, changed journals and lost source locks must still refuse publication. **Steps to reproduce** 1. Start a two-member cohort and interrupt the second upload after the first archive is saved. 2. Import an archived nonmember companion through the existing canonical migration owner. 3. Resume the same two-member cohort. The original implementation refuses before it acquires the source fence. **Paperclip version or commit** Reproduced on master `3a5572c9e5`. **Deployment mode** Online administrative workspace migration with PostgreSQL and authenticated snapshot storage. ## What Changed - Admit a changed aggregate scope proof only when every saved member has a running journal and its current fingerprint matches both saved fingerprints. - Keep partial reservations on the original strict aggregate-proof path. Keep exact cohort and operation membership checks. - Run the existing source, lifecycle, recovery and finalizer admission before refresh. Lock member rows, inspect the current scope again, and compare every journal in one transaction. - Change only `cohortMigration.sourceProof`. Preserve all other metadata, identities, archive and recovery records. Abort the complete refresh on cancellation, a changed journal or a lost fence. - Extend the existing real PostgreSQL fixture with partial upload, canonical companion migration, refusal controls, transaction rollback, cold archive restore and replay. - Document the proof lifetime and resume boundary. ## Verification - Canonical Node24 worktree preflight passed. Plugin SDK build prerequisites passed. - Genuine RED on unchanged product: the real partial-upload and companion-import sequence fails with `workspace_cohort_membership_changed` (1 failed; 22 intentionally not selected; 10.44 seconds). - The identical positive passes after the fix (1 passed; 22 intentionally not selected; 9.87 seconds). - The complete focused cohort file passes: 23 tests in 33.33 seconds. Existing controls include changed member fingerprints, foreign and completed journals, active operations, peer cancellation, actual source-fence connection loss and outer-commit failure. - The new sequence also checks a held source lock, a held finalizer lock, cancellation before refresh, and an injected second-member update failure. The failed update rolls back every proof change. Both saved archives restore tracked and untracked work, and replay uses the same cohort. - After one final read-only preflight assertion, that complete new sequence passes again in 14.52 seconds with 22 existing cases intentionally not selected. Server TypeScript passes on the final files. Module boundaries and `git diff --check` pass. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts`; final case selection: `-t 'same uploaded cohort'`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial local setup attempts remain nonpasses: sandbox PostgreSQL could not bind localhost, then the new archived companion fixture used a noncanonical repository folder. The corrected fixture uses the existing managed repository path. Neither setup failure is counted as the product RED. - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/20429#issuecomment-6755862789) is 5/5 on exact head `6c640c4667b136e5416c1375f491729cf4bb9b49`, with zero inline threads. Required final-head CI is running. Hosted acceptance is not claimed by this PR. ## Risks - The old aggregate proof contains only a hash. Refresh therefore requires durable fingerprints for every saved member and repeats all current admission guards. A partially reserved cohort cannot use this path. - A stale proof must never hide a changed member or a new writer. Current source inspection, P/I/L/F locks, locked member rows, complete journal compare-and-swap and the existing capture and publication checks remain authoritative. - The all-member update must stay atomic. A database fault or cancellation rolls it back. The original cohort, operation and snapshot IDs remain fixed. - The change adds one bounded transaction only when a fully saved cohort's scope proof changes. It adds no recurring work, schema, provider, archive format, concurrency or deployment changes. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
88186fe578 |
fix(workspaces): preserve unmaterialized repository intent during migration (#20416)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration preserves saved files when a web server becomes replaceable. > - A repository request records intent before any checkout exists. > - Online migration currently requires that request to finish, even when preparation needs credentials and has saved no repository files. > - This pull request preserves that database intent while migrating the existing workspace files. > - Normal admission still owns repository acquisition and authorization after restore. ## Linked Issues or Issue Description **What happened?** Online workspace migration refuses a pending repository request, or a request marked `needs_repair` with `repository_preparation_failed`, even when its contained checkout path does not exist. This can force an unrelated agent turn and credential acquisition before existing files can be preserved. **Expected behavior** Preserve an absent, unpinned repository request unchanged. Migrate the existing files under the normal source locks. Prepare the repository only on a later authorized admission. **Steps to reproduce** 1. Create a task-scoped local workspace with saved files and an absent repository checkout. 2. Keep its repository request pending, or let canonical preparation fail before checkout publication. 3. Attempt online workspace migration. Before this fix, it refuses with `repository_preparation_unresolved`. **Paperclip version or commit** Reproduced against master `8004ad069e`. **Deployment mode** Self-hosted control plane with PostgreSQL and local legacy workspace files. The regression uses real local PostgreSQL and independent filesystem restore. Related migration work: #20371. No duplicate open unmaterialized-request fix was found in the GitHub search. ## What Changed - Accept unchanged pending requests and canonical preparation failures only when no pin or branch exists and the contained checkout path is absent. - Require a proven online source or the existing offline writer-stop contract. - Leave companion workspace requests under their existing strict blocker. Classify the current workspace request with its filesystem check. - Extend existing migration and cold-admission fixtures. Repository rows retain their catalog, requested ref, failure state, timestamps, and identity. ## Verification - RED: the real PostgreSQL/filesystem regression produced two `repository_preparation_unresolved` failures for online pending and failed requests. Offline pending passed. Duration: 24.95 seconds. - Local server TypeScript check passed with Node 24 and pnpm 9.15.4. - GREEN on final head `eeaea6134a56397ca8c12ed8f7009a19c41d79b1`: 18 focused cases passed across the existing migration and cold-admission suites in 77.48 seconds. This covers offline pending intent, online pending/failed intent, active preparation and partial checkout refusals, strict companion requests, publication races, real archive restore, and later normal repository preparation. - The first GREEN attempt reached successful cold preparation, then failed one inherited assertion that expected the old failure code after success. The assertion now expects the canonical owner to clear it. The migration row-equality checks had already passed; no production change was needed for this assertion failure. - Final-head [PR workflow 42723571639](https://github.com/paperclipai/paperclip/actions/runs/42723571639) completed SUCCESS. The [final `ci / verify` gate](https://github.com/paperclipai/paperclip/actions/runs/42723571639/job/129531238991) passed at 2026-11-10 00:08:04 UTC. The full rollup contains 55 successful checks and 4 intentional skips, with no pending or failed checks. Greptile reviewed exact head `eeaea6134a56397ca8c12ed8f7009a19c41d79b1` at 23:52:42 UTC with 5/5, a successful check, and no inline threads. The optional `pnpm verify --explain` selection was invalidated when the assertion commit landed during selection; no verification pass is claimed from it. Focused local tests and required CI supply the verification scope. - The first local test invocation could not resolve the plugin SDK before its local build. It ran no tests. The SDK was then built and the RED regression ran successfully as a failure proof. ## Risks - The failed-request exception is limited to `repository_preparation_failed`. Unknown failures remain blocked. - Active preparation, a pin, a branch, a partial checkout, unsafe paths, and unresolved companion requests still block migration. - Migration does not edit repository intent or grant repository/task access. A restored request can still fail normal preparation if its current authorization or source is unavailable. - Offline migration retains its existing writer-stop and maintenance checks. Online migration retains its source, task, lifecycle, and publication fences. - This patch changes internal migration admission. No schema, public API, user command, or documentation contract changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and tool use. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bf818280f2 |
perf(workspaces): reserve cohort members concurrently (#20401)
## Thinking Path > - Paperclip controls agent work and preserves its workspaces. > - The online importer moves up to eight shared-source leaves under one source fence. > - Each member must have a durable reservation before capture starts. > - Serial reservations repeat the same validation cost before any archive work. > - This pull request reserves members in parallel through their existing independent transactions. > - Every reservation finishes before an error can release the fence or start capture. ## Linked Issues or Issue Description Refs: #19873, #20387, #20394. **Current behavior** A measured six-member cohort spent about 110 seconds in serial reservations. Each reservation keeps its existing source, task, recovery and journal checks. **Proposed behavior** Start the bounded member reservations together. Join all outcomes, retain member order, and refuse capture if any member fails. Keep the exact partial-journal resume contract. **Reason and benefit** This removes serial metadata work before capture. Hosted speedup has not been measured for this change. ## What Changed - Replace only the cohort reservation caller loop with `Promise.allSettled`. - Preserve the existing per-member transaction, preassigned operation ID, source fingerprint and all source fences. - Extend the real PostgreSQL failure fixture to require overlapping reservations and complete peer closure before fence release. - Document the unchanged partial-journal and atomic publication contract. ## Verification - Canonical cached worktree preflight and plugin SDK prerequisites passed on Node 24.22.2. - The unchanged production baseline failed the new reservation overlap assertion: one reader entered instead of two. The canonical PostgreSQL wrapper cleaned up. The other 21 tests were intentionally not selected. - The first baseline command failed before PostgreSQL because root-level `tsx` resolution was absent. The corrected command uses the checkout's explicit `cli/node_modules/tsx/dist/loader.mjs`. - The four focused PostgreSQL controls passed before the final rebase: overlap/join failure, exact partial-journal resume, parallel capture with both cold restores, and atomic outer-commit rollback. Eighteen tests were intentionally not selected. Server TypeScript passed. - The final branch is based on merged #20394 at `302ada90dfda0e8fbcd8a514d506d310c9843bcb`. The two directly affected PostgreSQL reservation/resume controls passed again on final head `06d436e45705da3b616a2b91725960f6e0ea52c0` (2 passed, 20 intentionally not selected; 100.56 seconds). Server TypeScript passed again. The canonical wrapper completed cleanup. - The separate Ponytail review remains 4/5. Its nonblocking fan-out warning was resolved as the documented phase contract; no source change was required. Optional multi-error aggregation is outside this change. - Fresh Greptile review is 5/5 on `06d436e45705da3b616a2b91725960f6e0ea52c0`, with no actionable findings or unresolved threads: https://github.com/paperclipai/paperclip/pull/20401#issuecomment-6754025480. - Exact-head CI is complete: 56 success, 4 skipped and 1 neutral, with no failed or pending checks. Aggregate `ci / verify` passed: https://github.com/paperclipai/paperclip/actions/runs/42718439355. - Reproduce the final focused cases on Node 24: `node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "joins concurrent reservation validation readers|resumes the exact membership after reservation stops halfway"`. Run `pnpm --filter @paperclipai/server exec tsc --noEmit` for TypeScript. ## Risks - API phase contract: `concurrency` limits capture workers. Preflight, reservation and post-finalizer validation fan out over the exact selected member set, with the canonical maximum of eight. The original DB pool bounds connections. `concurrency: 1` does not promise one metadata query or transaction at a time. - Up to eight reservations can use independent pool connections at once. Members that pass can leave durable journals when a peer fails. The existing exact-member resume path retains those journals; this change does not retry automatically. - Capture and publication remain blocked until every reservation settles. Publication stays serial in one transaction. Saved journal schemas, singleton behavior and recovery proof checks do not change. - This does not change concurrency inside the publication transaction. One PostgreSQL connection serializes that SQL, and changing its row-lock order has no measured benefit here. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
302ada90df |
fix(workspace): refresh source admission proofs on import resume (#20394)
## Thinking Path > - Paperclip preserves agent work and recovery obligations across server replacement. > - Online workspace imports keep durable operation and snapshot journals for interrupted work. > - A recovery admission fingerprint hashes complete run, finalizer, and lease rows. > - Accounting updates can change that hash while every saved recovery reference remains identical. > - Resume currently treats that expired admission observation as immutable source identity and refuses the existing journal. > - This pull request refreshes only that observation after exact identity checks and finalization fencing, preserving the original archive and journal. ## Linked Issues or Issue Description **What happened?** An interrupted import whose saved recovery sources have no changing online recovery hold can fail on its next attempt with `remote_workspace_migration_source_changed`. A change to a run's accounting fields changes `onlineProof.fingerprint`, even when all eight saved recovery records have the same typed references and ownership fields. **Expected behavior** For source-only admission drift, resume must compare every saved recovery source field exactly, except the supported ephemeral admission fingerprint. It must inspect the current source again, acquire the existing finalization fence, and retain strict full-proof checks through publication. It must keep the original operation, snapshot, archive, nested-child scope, and cohort identity. **Steps to reproduce** 1. Save a journal for an online import with eight recovery sources and no active online recovery holds. Interrupt after real authenticated restore of its uploaded snapshot. 2. Change only accounting fields on source 6's saved run. 3. Resume the same journal. The unchanged importer refuses it. The fixed importer preserves the original identities and commits after fresh guarded inspection. **Paperclip version or commit** Reproduced on master `9c634e42e449e91f3935cf59036981852f94a65e`. **Deployment mode** Online administrative imports with PostgreSQL. Related preservation work: https://github.com/paperclipai/paperclip/pull/20311 and https://github.com/paperclipai/paperclip/pull/20370. The independent readonly parallelism work in https://github.com/paperclipai/paperclip/pull/20387 changes different loops. ## What Changed - Compare the complete `legacyRecoverySources` records at resume, neutralizing only a valid fingerprint under the supported proof schema. Preserve every other field and refuse unsupported proof versions by name. - Reuse the current canonical full proofs. Persist them with an operation metadata compare-and-swap only after the existing finalization lock and strict reinspection succeed. - Keep strict full-proof checks within each attempt and before publication. Preserve all journal, snapshot, archive, nested-child, and cohort identities. - Add focused real PostgreSQL regressions in the existing online-source fixture, including exactly eight saved sources and repeat-apply idempotence. ## Verification - Genuine baseline RED: singleton and cohort resume both fail after accounting-only changes on the unchanged product. - `pnpm exec vitest run server/src/services/remote-workspace-online-source.test.ts -t 'eight saved recovery|changed saved recovery|recovery accounting changes|saved recovery proof'`: 8 passed. - Controls refuse changed valid references, a matched run/lease allocation change, competing ownership, and unsupported proof versions. They also prove that bookkeeping changes within an admitted attempt still refuse and a busy finalization fence prevents proof refresh. - Successful resumes preserve the operation/snapshot IDs, cipher SHA, manifest digest, byte size, nested scope, and cohort. Repeat apply creates no duplicate operation or snapshot. - Server layered typecheck, module boundaries, and whitespace checks passed. All applicable exact-head CI passed (including typecheck, build, all test shards and aggregate verify); hosted acceptance remains owned by the deployment workstream. Greptile re-reviewed the narrowed scope at 5/5. Ponytail scored 4/5, recommending a regression for the unchanged hold-bearing refusal; the maintainer explicitly kept that separate behavior outside this source-only fix. ## Risks This changes only source-proof resume admission. `onlineRecoveryHolds` and their action/intent fingerprints remain compared exactly, as on the baseline: a hold-bearing journal whose hold proof changes still refuses. Supporting that separate case would require preserving and validating the hold owner’s complete intent facts; this PR deliberately retains that safety check. Bookkeeping that changes during an active attempt still causes the existing strict publication checks to refuse that attempt. Older unsupported proof shapes remain refused. The metadata update uses the existing company/workspace/operation identity, running/unfinished status, and metadata compare-and-swap. No schema, provider, or storage format changes are included. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ec75547a2e |
perf(workspaces): validate cohort members concurrently (#20387)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration preserves each member under shared source locks. > - A cohort must validate every member before it reserves or captures state. > - Two read phases checked each member in sequence. > - An eight-member operation spent about eight minutes in those phases before capture. > - This pull request runs the bounded readers together and joins all results. > - Each source check, lock, and write order stays in place. ## Linked Issues or Issue Description **What happened?** An eight-member cohort spent 4 minutes 13 seconds in preflight, then 2 minutes 20 seconds in serial reservation, then 4 minutes 10 seconds in the read checks after finalizer locks. Capture started only after these phases. The two read phases invoked the same checks in sequence for each member. **Expected behavior** Run these read checks together within the existing eight-member limit. Wait for every started reader before throwing or releasing the source or finalizer fence. Keep reservation and publication serial. **Steps to reproduce** 1. Import a shared-source cohort with several members. 2. Delay one member's preflight or recovery inspection. 3. Observe that the old loop does not start the next reader until that inspection ends. 4. Fail one reader while another is still running. The fence must remain held until both readers finish. **Paperclip version or commit** Reproduced baseline: `facd409073c3c149421fd8f5af86c571fc28f294`. The branch now includes `9c634e42e449e91f3935cf59036981852f94a65e` and its merged queue-owner change. The queue hunks are unchanged. **Deployment mode** Server with PostgreSQL and online workspace migration. Related: [cohort migration in #19873](https://github.com/paperclipai/paperclip/pull/19873), [indexed workspace lookups in #20372](https://github.com/paperclipai/paperclip/pull/20372), and [queued source-lock admission in #20370](https://github.com/paperclipai/paperclip/pull/20370). The duplicate search found no open change for these two read phases. ## What Changed - Run cohort preflight member checks with `Promise.allSettled`. The existing input limit bounds this phase to eight readers. - Run the member checks after finalizer lock acquisition the same way. - Throw the first rejected result in the original member order after every reader has joined. - Add one fault fixture for both phases. It checks real source/finalizer locks, peer joining, retained local data, and the absence of capture or publication. ## Verification - Against the unchanged baseline, the two new phase cases failed because only one reader started instead of two. These were behavioral failures, not setup failures. - Focused real-PostgreSQL cohort checks selected 13 cases. The first run passed 10 and failed three existing controls. The two new reader-join cases passed. Eight unrelated cases were not selected. - The first run's existing capture overlap check observed one active uploader instead of two. Two existing fence/commit controls reached their 15-second test limits. Server typechecking ran at the same time. This does not establish the cause of those failures. - One retry ran only those same three controls after typechecking finished. All three passed with unchanged source, assertions, and deadlines. Eighteen cases were not selected. The earlier failures remain accounted for; they are not classified as infrastructure faults or flakes. - `pnpm --filter @paperclipai/server typecheck` and `git diff --check` passed. - These focused counts were recorded before the non-overlapping queue-owner rebase. On final head `06158d905c4716bf44b1d028f96c706ac60e0e11`, both new reader-fence cases passed again. Nineteen cases were not selected in this recheck. - Focused commands: - `pnpm --filter @paperclipai/server exec vitest run src/services/remote-workspace-cohort-migration.test.ts -t 'joins|captures two shared-Git|before reservation|source fingerprints|member journal|outer commit|fence connection' --maxWorkers=1` - Retry only: `pnpm --filter @paperclipai/server exec vitest run src/services/remote-workspace-cohort-migration.test.ts -t 'captures two shared-Git|actual outer commit|actual fence connection' --maxWorkers=1` - Checks used Node 24.22.2, pnpm 9.15.4, and disposable PostgreSQL 18.1. No hosted operation used this patch. End-to-end improvement is unverified. - Public commands, input limits, journal state, and recovery contracts do not change. No command documentation update is needed. ## Risks - Concurrent reads increase database and filesystem load while source locks are held. The existing cohort input limit bounds each phase to eight readers. Initial source discovery already uses the same bound. - A failed phase waits for started peers. This retains the fence until their work ends. Existing query and caller limits remain in place. - Serial reservation and publication remain. Every freshness, fingerprint, recovery, and publication check still runs. - The local tests prove overlap and joining. They do not establish a hosted speedup. Database contention can limit the benefit. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime model ID and context window are not exposed in this session. Used reasoning, code editing, shell tools, and real PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9c634e42e4 |
fix(workspace): bound queued source-lock admission for imports (#20370)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing local workspaces need durable imports before a volume-free deployment. > - Imports must acquire the same source locks as workspace writers. > - Try-only admission can repeatedly lose to background maintenance, even when each writer finishes. > - This pull request adds an optional bounded wait to the canonical migration APIs. > - Operators can join the source-lock queue while all existing preservation checks remain active. ## Linked Issues or Issue Description **What happened?** An online import can repeatedly return `workspace_source_busy` while background workspace maintenance reacquires its project source lock. The import does not join the PostgreSQL lock queue. **Expected behavior** An administrative import can wait for a finite writer to finish. One wait budget covers the complete source scope. Timeout or cancellation must release acquired locks before the import reserves a journal. **Steps to reproduce** 1. Hold the source lock for an eligible local workspace. 2. Start an online import while the writer holds that lock. 3. Release the writer. Before this change, the import has already refused admission. With `sourceLockWaitMs`, the import acquires the source scope and commits normally. **Paperclip version or commit** Reproduced on master `0e3fdebea8257bdff11573081c76b306f12db230`. **Deployment mode** External PostgreSQL with an online workspace import. Related preservation work: https://github.com/paperclipai/paperclip/pull/20311. ## What Changed - Add optional `sourceLockWaitMs` to singleton and cohort migration APIs. Omitted options keep current admission behavior. - Queue the sorted source keys with one decreasing wait budget. Abort cancels only the dedicated backend identified by PID and backend start time. - Use an independent source transaction for queued singleton imports. Keep lifecycle checks, fresh source checks, journal ownership, and publication after commit. - Document the canonical API option. Add real PostgreSQL importer regressions for release, timeout, abort, and partial-key cleanup. - Correct the inherited task-case lookup summary so the existing search-discovery regression passes without changing its assertion or search algorithm. ## Verification - Baseline RED: the new singleton regression fails on the unchanged product because the import returns busy without queueing. - `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'queued|bounds the whole source queue'`: 20 passed, including five new cases. - `pnpm exec vitest run server/src/services/isolated-remote-workspace-source-lock.test.ts`: 4 passed. - Server layer typecheck, `node scripts/check-module-boundaries.mjs`, and `git diff --check` passed. - Tests used Node 24.22.2 and the existing embedded PostgreSQL fixture. Hosted rollout verification remains pending. - CI build, typecheck, browser suites, all serialized suites, and the other general shards passed. The final server shard failed only the inherited `case-history-api-discovery.test.ts:14` search-ranking assertion. Its unchanged local regression reproduces on the base and passes 3/3 with a separate one-line route-summary correction. The shared one-line summary correction is included in follow-up commit `8163ffa9d4728134676b84c8c242a5bf62c802b5`; the unchanged case-discovery suite passes 3/3 locally. The original CI attempt remains failed: https://github.com/paperclipai/paperclip/actions/runs/42704666536/job/129458605783. ## Risks Queued admission can hold earlier source keys while it waits for a later key. The caller controls the finite budget. The abort path opens a short-lived dedicated cancellation connection. If cancellation cannot connect, PostgreSQL's remaining lock timeout still bounds the wait. This option must be called at the root of administrative work; an external scope that already holds the same source locks can cause self-contention. The existing pre-held journal-resume path must keep its omitted-option behavior. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
1b7fe33432 |
fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
965ca91dd0 |
Capture legacy workspace cohorts under one source fence (#19873)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace migration preserves local files as authenticated portable
snapshots.
> - Workspaces that share Git storage require one complete source fence.
> - Independent imports contend on that fence and cannot safely run as a
batch.
> - This change adds an opt-in cohort owner that captures members
concurrently and publishes them atomically.
> - Existing source, recovery, snapshot and publication checks remain
authoritative.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The administrative remote-workspace migration owner for workspaces in
one shared physical source component. Related changes: #19688, #19854
and #19855.
**Current behavior**
Each independent import acquires the complete source fence. Parallel
independent calls therefore contend instead of overlapping archive work.
A caller cannot use a surrounding transaction and assume that a
successful savepoint confirms the outer commit.
**Proposed behavior**
An explicit `migrateRemoteWorkspaceCohort` call selects one to eight
leaves from one proven component. It owns the full
source/lifecycle/copyback fence, uses the original pool for bounded
archive work, validates every member, publishes serially, and returns
success only after the outer commit.
**Reason and benefit**
This supplies a correctness-gated path for a controlled two-member
hosted qualification. It makes no wall-clock speedup claim. No running
app call site or scheduler selects cohorts automatically.
**Breaking changes**
The existing single-import API retains its guards and rejects a journal
owned by a cohort. New cohort journals require the same complete member
set on resume.
## What Changed
- Extract the existing reservation, capture, validation and publication
stages for reuse by both owners.
- Add an opt-in cohort API with default concurrency two and a maximum of
eight. Every member must share the same complete physical/Git/alias
component; selected roots must be leaves relative to each other.
- Persist exact cohort and operation membership in the existing journal.
Compare all selected fingerprints with the aggregate proof before the
first reservation under the complete fence, and recheck each member when
reserving its journal. Refuse foreign journals, incomplete or mixed
replay sets, lost fences and changed sources. Keep all authorities and
activity events unpublished if the outer commit fails.
- Share the merged nested-child verifier inside one complete fence.
Queue its prepare calls, retain its two-child bound and authenticated
immutable proof, and recheck local bytes and database bindings on every
pass.
- Document the API, exact-membership resume protocol, resource bounds
and required hosted qualification.
## Verification
- Integration head `fce1896010`: 32/32 PostgreSQL/filesystem cases
across the cohort owner, the complete merged nested-verifier suite, and
five existing parent-import cases.
- Final source-proof repair: 19/19 cohort-owner cases pass, including
real first- and last-member metadata changes before reservation,
changed-source partial-resume refusal, mixed local/imported refusal with
storage forbidden, cancelled/failed/finished admission, a late finished
timestamp, complete fences, queued child-proof reuse, cold restore,
cancellation, commit failure and exact replay. All local database checks
use the canonical owned wrapper, one Vitest worker and Node 24.22.2 on
macOS arm64/PostgreSQL 18.1.
- Final rebased head is based on
`fbe689938c3bf8c5849d2cd56b9da90029db9389`, including current
source-owner, database-schema and Runner contract changes. The 19-case
owner suite, full server layered source/tests/test-tree typechecks,
module boundaries and diff checks are rerun on that base; required CI
supplies the native Runner qualification.
- Selection-only `node scripts/local-verify.mjs --base
|
||
|
DottaandPaperclip
|
54b83b15a3 |
Reuse authenticated nested workspace verification (#19854)
## Thinking Path > - Paperclip preserves the work that its agents produce. > - Portable workspace migration can save nested workspaces before their parent. > - The parent must prove that a committed child snapshot covers any files it omits. > - The existing owner downloads and restores the same immutable child archive before and after parent capture. > - The fresh local-file checks are required, but the repeated immutable archive check is avoidable within one live source fence. > - This pull request reuses only an authenticated archive proof with an unchanged reference and provider identity. > - It also bounds independent child branches at two workers while retaining every fresh local and Git check. ## Linked Issues or Issue Description Refs #19805 and #19688. A duplicate search found no separate fix for repeated child archive verification. **What existing behavior does this improve?** Online Git parent migration after nested workspaces have committed portable snapshots. **Subsystem affected** server/ — workspace migration and nested snapshot coverage. **Current behavior** Each parent verification pass downloads, authenticates and restores every child archive, then captures the child's current local tree. The second pass repeats the immutable download and restore. Independent child branches also run in sequence. A large parent can exhaust its existing budget before parent capture begins. **Proposed behavior** One private verifier belongs to the existing live source-owner transaction. It can reuse a completed archive proof only when the full snapshot reference, operation, child closure, ETag and byte length still match. Missing or weak ETags and non-cancellation HEAD errors keep full verification. Caller cancellation still stops work. A changed identity forces fresh authenticated readback. The owner still captures current local bytes and Git state in both passes. **Reason and benefit** Remove duplicate immutable work without treating an old local-file check as current authority. Two independent topmost branches can overlap. Recursive descendants stay serial within those branches, so the bound does not grow with depth. ## What Changed - Add a private per-fence verifier to the existing tree owner. It clears its proof map at the end of the import and refuses a different owner transaction. - Reuse only completed ciphertext, GCM, plaintext, and canonical restore verification with a matching fresh provider identity. - Retain all current database, operation, recursive coverage, local-file, Git and manifest checks on each pass. - Bound independent branches at two. Abort queued work on failure and await active peers before scratch cleanup or fence release. - Use the verifier only in the existing online Git importer. Keep the offline full-verification path and all capture deadlines unchanged. - Document resource admission, fallback behavior and the hosted acceptance limit. ## Verification - Genuine baseline RED through the real parent importer: child GET counts 2 + 2, canonical restores 4, maximum overlapping readbacks 1. The four fresh local child captures already pass. - Exact positive GREEN: child GET counts 1 + 1, canonical restores 2, maximum overlapping readbacks 2. All four fresh local child captures remain. - The fixture uses real encrypted archives and the canonical S3 provider's local HTTP HEAD parsing with actual payload-derived ETags. It uses fake fixture credentials only. - Final focused suite: **13/13 PASS**, 148.36 seconds, at `eaf17d3340`. This includes recursive max-two completion, failure/cancellation joining, identity/ref/content changes, same-fence alias reuse and stale-owner refusal. - Both affected existing migration suites: **256/256 PASS**, 201.84 seconds, at initial head `eaf17d3340`. - The HEAD-failure review repair has genuine two-case RED evidence and three passing failure/cancellation controls. Full layered source/test-tree typecheck and module boundaries pass at repaired head `763648adad`. Final exact-head controls: **21/21 PASS**, 86.63 seconds (all 16 new controls plus five existing parent cases). - The first extended recursive fixture correctly failed because it created a Git branch after taking an earlier snapshot. The fixture now creates all branches before any import; the source-change guard remains intact. - Exact-head CI: 56 success, four expected skips, and one neutral Paperclip Review. Required gates pass; Greptile is fresh 5/5. Its optional aggregate-handler cleanup recommendation is resolved with the intentional fail-closed rationale. - Local verification is macOS arm64 / Node26.4.0. The observed hosted parent still needs to complete within its unchanged budget after normal delivery. No hosted timing improvement is claimed here. ## Risks - Two simultaneous child verifications can use more I/O, memory and scratch space. Operators must count logical verifications, even in one process. - HEAD availability and provider identity are required for proof reuse. Providers without a usable ETag perform the full check. - Reuse lasts only for one live transaction. A retry, changed reference, changed provider identity, process exit or owner change requires fresh verification. - Faster verification does not waive changed child bytes or guarantee a specific hosted completion time. No provider turn, authority marker, source deletion or larger timeout is introduced. ## Model Used OpenAI GPT-6 through Codex, with repository and code-execution tools. The exact serving model revision and context-window size are not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c92bc9e5ec |
perf: reuse authenticated workspace upload readback (#19811)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace import saves local work to an encrypted immutable snapshot before it changes storage authority. > - The snapshot uploader reads the whole object back to verify its ciphertext. > - The importer then downloads that same object again for authenticated decryption and restore. > - Both reads are required by the current call sequence, but they need not transfer the object twice. > - This PR lets that caller retain the authenticated upload readback in private scratch. > - The importer still restores the full archive, recaptures the source, checks ownership, and publishes through the existing CAS. ## Linked Issues or Issue Description Refs #19642, #19805, and #19810 for the existing workspace import and preservation owners. **What happened?** A fresh workspace import performs a PUT and two full GETs of the same encrypted snapshot. The first GET verifies and discards the object. The next GET decrypts it for restore. Already-uploaded and lost-reply recovery paths also repeat the GET. **Expected behavior** Use one authenticated readback for the exact final winning reference. Preserve the default upload behavior and returned reference. Keep every restore, source, recovery, ownership, and publication check. **Steps to reproduce** 1. Capture a local source through the real workspace archive codec. 2. Upload it through the real snapshot ledger and filesystem object backend. 3. Load and restore its archive. Count the object GETs. 4. The three baseline cases each make two GETs. The opt-in path now makes one and restores the same bytes after the original source is removed. **Paperclip version or commit** Baseline `585c885168`. The first PR head was `5b18121609`; the follow-up repairs the review findings without changing the storage optimization. ## What Changed - Add optional `verifiedArchivePath` to the existing upload owner. The default path and `WorkspaceSnapshotRef` return contract stay unchanged. - Authenticate each readback with the existing ciphertext, GCM, key-binding, and plaintext checks. - Bind retained files to the full winning reference. Recheck private file ownership and plaintext SHA/size before atomic publication to a new caller-owned path. - Clean all invocation scratch on success or failure. Do not overwrite caller files or reuse state across calls. - Use this path in POSIX workspace import. Keep full extraction, the second source capture, closed alias/source/F/writer/task checks, and CAS publication. Windows keeps the existing download path and does not bypass private-file checks. - Cover real cold restore, fresh/replay/lost-reply GET counts, late winners, private-path rejection, local tampering, and GCM failure. ## Verification - Genuine baseline RED: three real PostgreSQL/crypto cases each report two GETs where the new contract requires one. - GREEN: all 30 snapshot storage cases, one actual online migration case, and seven focused migration controls pass; 38 total. The other 232 cases in those files were unselected locally. - Review RED: four obsolete second-GET fault triggers and one uploaded-journal retry case failed before repair. The triggers now fire on the retained readback and assert that they fired. The uploaded-journal case interrupts after its real ledger commit and proves recovery and cold Git restore through the same journal. - A focused platform-selection control proves Windows retains the original two-GET path. This is not native Windows acceptance. - Strict server TypeScript, module boundaries, and `git diff --check` pass. - Local command from `server/`: `node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-online-source.test.ts src/services/remote-workspace-migration.test.ts --testNamePattern 'remote workspace snapshot storage|imports an alias with unrelated|removes plaintext verification files|rechecks a service that becomes|rechecks a writer that appears|preserves a local edit|keeps the existing download path|missing-base-uploaded-retry' --hookTimeout 90000 --maxWorkers 1 --no-file-parallelism --fsModuleCache`. - The codec timing control used an exact earlier owner and a synthetic 72.9 MB archive. Discarding final-capture output saved only 3%; this PR keeps that capture unchanged. - This removes one full object GET. It does not claim a measured hosted wall-clock improvement. - Fresh Greptile review is 5/5 on `0933a0a29a`, with no open findings. Exact-head CI is complete and green: 56 successful checks, four intended skips, and one neutral result. The second review returned Ship (4/5); its optional test-maintainability suggestion is deferred. No source change or CI restart was made for that suggestion. ## Risks - A late upload can win with a different reference. The retained file must match all fields of that final reference; otherwise the owner authenticates the winner. - A private file can change before relay. Ownership, no-follow reads, stable file identity, plaintext SHA/size, and no-overwrite publication protect that boundary. - Authentication may overlap the encrypted upload scratch. Peak local disk use can grow by up to one ciphertext file. Data remains streaming; no whole-object memory buffer is added. - Opt-in callers must supply a canonical private directory owned by the process. Existing callers that omit the option retain their behavior. - No archive format, database schema, provider lifecycle, writer protocol, admission, or background polling changes. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use, and local code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
585c885168 |
fix: preserve prepared sources held by provider artifact failures (#19810)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration archives local files before it changes their storage authority. > - A failed provider session can leave an active board-owned recovery hold over a prepared local source. > - The existing migration owner accepts finalized sources and workspace-export failures. > - It rejects a settled provider-artifact failure even when the prepared source and recovery allocation have complete bindings. > - This PR permits that exact preservation case under the existing source and finalizer locks. > - The remote recovery hold remains active. The archive preserves local bytes and the existing recovery recipe. ## Linked Issues or Issue Description Refs #19642 and #19805 for the existing online migration and source-closure owners. **What happened?** Online workspace migration rejects a prepared source held by a terminal `runner_remote_provider_artifact_incompatible` recovery. This also blocks another local workspace that shares the Git source. The provider session had executed work. A pre-execution exemption would be unsafe. **Expected behavior** Preserve the local source through the existing immutable archive owner when the board recovery, task version, terminal run, finalizer, and declared allocation all match. Keep the remote recovery action and authority unchanged. **Steps to reproduce** 1. Create a prepared native source and a failed run with provider session and tool events. 2. Set its finalizer to an unowned terminal artifact failure. Add the matching active board recovery and task status version. 3. Remove the local descriptor and inspect or migrate a shared local Git alias. 4. Before this change, the source proof is rejected. The regression fixture reproduces this failure. **Paperclip version or commit** Reproduced on master `1674902b4f`. The final branch is based on `5a92d53f7d`. ## What Changed - Accept only the matching active board-owned artifact failure with a current task block, no live execution authority, an exact producer fingerprint, and a declared prepared-source allocation. - Include the task version and complete existing source/recovery evidence in publication revalidation. - Lock the proven recovery tasks in sorted company scope before publication checks. A held descendant task produces the existing busy error. - Add a real PostgreSQL regression with 124 tool executions, a missing descriptor, shared Git aliases, immutable download, and cold restore of the local archive. - Add stale, foreign, owned, controlled, malformed, and publication-race rejection cases. - Document that this archive preserves local bytes and the recovery recipe. It does not prove remote export or release the remote hold. ## Verification - Genuine baseline RED: the positive migration regression rejects the source before the production fix. - Focused treatment: 12/12 PostgreSQL cases pass. - Review-race RED: both a selected source and a Git companion published while a descendant recovery task was locked. The corrected tests reject both cases. - Two affected owner suites after the lock repair: 91/91 pass in 23.18 seconds. - Strict server TypeScript check passes after the lock repair. - Module boundaries and `git diff --check` pass. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/42367152081) passes: 56 successful checks, four conditional skips, and one neutral review check. No checks remain pending or failed. - Greptile and Paperclip reviews are both 5/5 on `0a6e6c3ece`, with no open findings. - The first descendant fixture attempt hit the immutable task/privacy guard before it reached the race. The corrected fixture creates that task binding at admission and produces the two genuine REDs described above. ## Risks - A broad classification could let a migration move local authority without a complete recovery recipe. The new branch requires the exact failure, producer, task version, source allocation, and released execution owners. - Evidence can change during upload. Existing locks and full publication revalidation reject those races. - The remote allocation may contain changes absent from the local source. This PR preserves only local bytes and the durable recovery route. It does not mark remote work exported, finalized, or safe to discard. - No schema, provider lifecycle, admission, or polling behavior changes. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use, and local code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d5c4f7dd02 |
fix: preserve shared Git sources during online workspace migration (#19688)
## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. **What happened?** Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. **Expected behavior** Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. **Steps to reproduce** 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. **Paperclip version or commit** This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. **Deployment mode** Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7ac99718a4 |
fix(workspaces): fence legacy sources for online migration (#19642)
## Thinking Path > - Paperclip coordinates AI agents and preserves their saved work. > - Remote workspace storage replaces legacy host directories with durable snapshots. > - Several legacy workspace records can name one physical project directory. > - A task lock alone cannot exclude a new task that selects that directory. > - Terminal recovery records can retain valid sources after physical writers settle. > - This change fences the source through its real preparation and lifecycle owners. > - Idle ordinary sources can then migrate while unrelated remote work continues. ## Linked Issues or Issue Description **What happened?** Online migration rejects ordinary legacy directories when several workspace records name the same path. A source inspection found 45 aliases across three project roots with no root Git marker or previously imported nested child. The instance also had unrelated running remote work. Requiring the whole instance to drain prevented useful migration of these idle sources. Terminal native sources and superseded finalization attempts also remain blocked after their exact physical and durable owners settle. Their files, provider allocations and recovery records must remain available through the change of file authority. **Expected behavior** Migrate an idle ordinary source under a derived physical source fence. Let unrelated object-backed turns continue. Preserve provider and recovery obligations. Keep unknown writers, shared Git stores and uncertain source state blocked. **Steps to reproduce** 1. Create several local workspace records for one managed project directory. 2. Give the aliases different source tasks. Leave their local writers idle. 3. Keep a valid remote workspace turn running on another source. 4. Inspect or apply migration to an alias. The old owner returns `overlapping_workspace_requires_maintenance`. 5. Retain a terminal native prepared source with a null coordinator owner, or an old finalize attempt with an exact later committed receipt. The old online path keeps these sources blocked. **Paperclip version or commit** Current tested head: `7478c817f09f7eed396922c0f7831af07ae6ff14`, based on master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. Related recovery work: #19131. **Deployment mode** Hosted native Runner with remote workspace storage. Qualification uses disposable local PostgreSQL databases, actual archive/storage owners and synthetic source trees. The source observations were read-only. ## What Changed - Retain the existing company/task source keys and add one physical managed-project key. Heartbeat source selection, materialization, reopen and lifecycle owners share it. Configured aliases of canonical task directories retain their original task key. - Fence runtime provision and each configured service CWD through both startup owners. Check all unsettled service CWDs for physical source overlap, even when workspace IDs differ. Runs with no services keep their filesystem-free path. - Derive a bounded ordinary alias closure. Lock every alias task and lifecycle key. Reject foreign companies, changed membership, unproven descendants, included Git metadata and other unresolved writer obligations. - Commit remote authority on the connection that holds source exclusion. Refresh a stale selected local workspace after admission if migration published remote authority. Publish activity after commit. - Admit exact terminal native recovery sources under the existing physical finalizer keys. Bind descriptors, allocations and owner checkpoints to journaled fingerprints. Recheck them before capture and publication. - Add a separate company-scoped online entry for exact superseded finalization receipts. Keep the offline entry's deployment prerequisites and keep inventory read-only. - Document the proof and the remaining Git/common-store extension. The original files stay in place. No new scheduler, deployment action or database migration is added. ## Verification - Review found that the real reopen path could hold lifecycle exclusion while waiting for a project source. The new actual nested-reopen regression failed before the repair: lifecycle lock count was 1 while another connection held the project fence. The repair acquires declared and physical project sources before reconstruction and lifecycle. The contention case, ordinary-fetch restore and runtime nested restore passed. The lifecycle-worker restore first completed in 16.64 seconds but exceeded its unchanged 15-second body limit; a bounded warm rerun passed at that same limit. Both wrappers exited and completed cleanup. The source type graph, module-boundary check and diff check pass. Final 7478 CI passed: 58 successful jobs, three skipped, one neutral and a successful Snyk context. Fresh Greptile is 5/5 for exact7478, with no unresolved threads. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42318150314/job/128240631573). - Full CI on `e4283b3452` passed the other server, build and type checks. One exposure owner case failed: two execution workspaces received one preview pair after source admission serialized their starts. The first local seven-case run reproduced that failure; adding the owner argument alone did not change the hash payload. The completed repair binds run-scoped reuse to its execution-workspace owner. All seven existing real-process/fake-broker cases then passed in 13.81 seconds, with wrapper exit 0 and cleanup complete. Public scope IDs and explicit project/agent sharing are unchanged. The final7478 head passed all required CI and fresh review gates. - The service/source/restore owner canary passed 25 cases across four suites before the final no-service fast-path guard (41.33 seconds). This includes actual project service processes with a different project ID and no execution-workspace ID. Their startup waits for source ownership, migration stays blocked while they run, and the source becomes eligible after their verified stop. The final three-case runtime canary passed at the final source inputs (18.70 seconds). It covers that fast path and both service-CWD variants. Both wrappers exited 0 and completed cleanup. - The boundary-repair canary passed 9 cases across four suites at `7c18f1c170`. It covers the real rehome/conversion gateway, project command provisioning, future task selection, stale imported authority, foreign-company binding, and existing standalone filesystem callers. The shared PostgreSQL wrapper exited 0 and completed cleanup. Duration: 94.15 seconds. - The P1 diagnostic passed 46 cases across five suites at the pre-rebase P1 revision. It covers the actual importer with an export warning created by `recordNativeFinalizationFailure`, strict prepared-source/action bindings, finalizer locks, source/lifecycle races, unrelated remote execution, and bare Git with external alternates. It uses the repository shared PostgreSQL wrapper, `--fsModuleCache`, and a local-only 90-second hook allowance. Duration: 245.44 seconds. No product timeout changed. - The canonical retained-directory compatibility check passed 22 cases across two suites. Missing archived task directories still import from authenticated restore points; they do not need a present-directory source proof. - Both initial P1 regressions were observed before the fix. The real finalizer warning made the importer ineligible. The old archive owner accepted a nested bare repository borrowing external objects. The new cases pass and preserve the original warning, prepared descriptor, allocations and files. - Prior attempts remain non-passes: the first 23-case canary passed on an earlier head; its exact-head rerun hit 30-second database setup/cleanup limits. A read of the owned PostgreSQL server found a template clone waiting on `DataFileExtend` for 58 seconds. A broader owner run was intentionally interrupted during slow existing heartbeat setup. One diagnostic invocation used an invalid CLI option and executed no tests. A later review canary passed 21 cases and failed the new real restore contention case: cwd-only admission missed the declared project of an external worktree. Its fixture also omitted the required file-repository authorization callback and produced an unhandled rejection after the failed assertion. The repaired run passed 22 cases, exited 0 and completed cleanup. These earlier failures are not reported as passes. - The recovery owner passed 37 PostgreSQL cases, including 14 new and 23 existing finalized-recovery cases. The operation reconciliation owner passed 40 cases. The initial archive run passed 103 cases and found a missing closed diagnostic entry; the corrected diagnostic and expanded Git rejection cases pass in the P1 run. - The full server type graph passes with the Runner source exports mapped locally because this checkout has no built Runner declarations. The latest module-boundary check and `git diff --check` pass. Standard package build/typecheck and broad final-head regression coverage remain required in CI. - The initial CI exposed missing admission scopes for standalone filesystem APIs and a changed foreign-company error. The latest repair keeps those APIs independent of a database and holds physical fences at their actual database-backed mutation owners. The branch also includes the existing master fix for unchanged Runner default-model assertions. New review findings and new-head CI are being checked. - The local canary command and bounded source selection are documented in `doc/plans/2026-11-06-online-ordinary-source-fence.md`. Source observations only read data. A fresh bounded metadata probe found 254 pending legacy records and zero unsettled local-process services. No live import, hold, cancellation, provider action or source filesystem write was performed for this change. ## Risks - Run-scoped reuse now includes execution-workspace ownership in its private key. Explicit project/agent sharing keeps its previous identity. - A missing writer owner could make a source proof unsafe. Admission derives physical aliases, includes future project selection and retains all existing source/lifecycle/finalizer locks. Unknown and changed state fail closed. - Cross-company physical alias checks use bounded minimal-path pages. Lexical-prefix filtering cannot safely replace them because a symlink can hide a foreign alias. Unknown paths or more than 10,240 foreign records leave the source explicitly unproven. More than 1,000 same-company local records also fail closed. - The project key can serialize local preparation within one project. A first project waits; additional project acquisition fails on contention to avoid a lock cycle. Remote authority keeps its remote preparation path. - Recovery preservation does not prove provider termination or successful file recovery. Existing provider allocations, checkpoints, task intent and recovery actions remain authoritative. - This slice does not finish shared Git migration. Git administrative/common-object/alternates closure and exact nested imported-child coverage remain the next owned extension. No global writers-stopped shortcut is used online. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
6cafcaf89b |
fix(workspaces): recover unpublished archived Git uploads (#19459)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace imports preserve existing files before the server changes workspace authority. > - Archived Git imports reconstruct saved files and Git state in a private repository. > - A failed upload can leave a pending snapshot journal with no object in storage. > - A later reconstruction can have different Git administrative bytes for the same saved logical source. > - This pull request lets that import record a new unpublished capture after it proves that prior objects are absent. > - The benefit is safe retry through the original import journal while source checks and atomic publication remain in place. ## Linked Issues or Issue Description Refs #19396. Refs #19409. A snapshot upload can fail before object persistence. The caller then removes its private archive. A retry reconstructs the same saved Git workspace. If its administrative bytes differ, the snapshot owner rejects the retry with `remote_workspace_snapshot_operation_content_changed`. The pending journal remains blocked even though no snapshot object or remote authority was published. ## What Changed - Add an optional unpublished recapture capability to snapshot storage. Bind it to the original source operation. Existing callers keep the exact manifest check. - Enable this capability only for archived Git reconstruction. The migration owner keeps the source recipe, task binding, ownership, and publication checks. - Verify every prior object before new capture admission. Unknown or corrupt objects stop the retry. Reuse an available prior object, including an object that appears while the replacement upload runs. - Compare the observed upload journal under the snapshot row lock before appending an attempt. Keep network I/O outside that lock. Preserve all prior attempt metadata. - Keep the first atomic publication as the winner. A losing uploader verifies and returns that saved reference. - Add real PostgreSQL, filesystem, and Git regression tests. Document the retry behavior. ## Verification - Baseline `70d23612dd9176cae8ee6b2b6dcf2f6eade9a42a`: the new real Git fixture fails with `remote_workspace_snapshot_operation_content_changed`. It fails the first upload before persistence, removes the old source cache, and advances unrelated remote refs. The saved task commit and restore point stay fixed. - Final frozen source: 20 tests pass in a fresh process. This includes the real Git regression and all snapshot storage cases. Tests cover prior object recovery, corrupt and unknown objects, normal caller mismatch rejection, operation binding, stale journal admission, and late upload publication in both orders. - Reproduce the final focused check from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-migration.test.ts -t 'failed-upload-retry|snapshot storage' --reporter=dot ``` - An earlier six-suite run had 201 passing cases and one failing late-object case. That run started before the final prior-object recheck was added. The failing case passes in the fresh final run above. This earlier run is not a clean full-family result for the final source. - That run skipped 16 lifecycle cases because `PAPERCLIP_REMOTE_WORKSPACE_LIFECYCLE_TEST_DATABASE_URL` was unset. Migration, storage, reservation, and concurrency tests used real disposable embedded PostgreSQL. - `git diff --check` passes. Full repository typecheck passed before rebasing. Full test, build, and current-head CI results are pending. ## Risks - The optional capability permits different physical captures. The migration owner must prove that the logical source and current ownership are unchanged. Ordinary mutable workspace uploads keep the previous mismatch guard. - An old upload can complete after the last object check. The first authenticated atomic publication wins. A late upload cannot replace its object or snapshot reference. Every candidate remains in the retention journal. - Unknown storage results and journal changes fail closed. They can require another canonical retry. This change adds no retry loop and changes no database schema. ## Model Used OpenAI Codex, GPT-6 family. The worker used reasoning, tool use, code editing, and local test execution. The worker runtime did not expose an exact model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c0214826b3 |
fix(workspaces): preserve pending repository requests during import (#19439)
## Thinking Path > - Paperclip manages agent work and its saved files. > - An ephemeral web server needs remote workspace authority before the old volume can detach. > - A task can have a repository request that has never prepared a checkout. > - The offline importer currently treats that pending request as unfinished file publication. > - This blocks migration even when the target does not exist and there is no saved commit, branch or failure. > - This pull request preserves that request in the database and imports the existing files separately. > - The next authorized sandbox preparation can acquire the repository through its existing owner. ## Linked Issues or Issue Description **What happened?** On Core `253bf071f0a496554b61fff897ad3f11afaf8766`, offline migration of a volume-backed workspace fails with `repository_preparation_unresolved` for an untouched pending repository request. The requested directory has never existed. The request has no pinned commit, branch or failure. **Expected behavior** Preserve the pending request and all existing files. Do not label the absent checkout as part of a saved archive. Keep partial publication and active preparation blocked. **Steps to reproduce** Create a local task workspace. Add an unmaterialized pending repository request. Run the canonical offline importer after stopping writers. The unchanged importer rejects the request before preserving the workspace. Related migration work: #19429. A GitHub search found no duplicate pending-repository import fix or matching public issue. ## What Changed - Admit an untouched pending request only at the existing offline writer fence and after proving its target is absent. - Keep the database request unchanged and exclude it from materialized snapshot descriptors. - Recheck file absence and source identity before publication. Reject a request that becomes ready during capture. - Cover cold filesystem loss, real sandbox-side Git acquisition, and eleven blocked states. Document the preservation contract. ## Verification - Red: the new real PostgreSQL/filesystem regression fails on the unchanged importer with `repository_preparation_unresolved`. - Green: both affected migration/lifecycle suites pass, **236 tests**. The cold test deletes the predecessor directory, restores saved bytes, and acquires the pending repository through the full admission owner, checking ready state and the pinned commit. Real Git runs against a local HTTP origin through an exact-URL redirect at the sandbox transport boundary. It does not claim live Daytona or GitHub qualification. - Full workspace typecheck and build pass at the final head. Both affected suites pass **236 tests**, including the full admission owner and eleven blocked states. All final-head CI checks pass: **54 success, 2 skipped, 1 neutral**. Greptile is **5/5**, with no unresolved review threads. - The full local test command failed in `recovers lost inline and check responses without duplicate GitHub publications` in the chat integration suite. That exact case passes in a fresh isolated invocation. Its cause remains unclassified; this is not a full-local-suite pass claim. The original failure is retained alongside the isolated result. - No candidate code has been applied to the serving instance. ## Risks - The exception applies only to offline migration and a pending request with no target, commit, branch or failure. Unreadable paths, symlinks, partial directories and active/repair states retain their blockers. - Requests retain normal repository authorization and acquisition behavior. This change neither fetches repositories during migration nor marks requests ready. - A changed materialized source aborts publication and leaves original files intact. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool use. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
92452524f4 |
fix(server): recover authenticated empty retention checkpoints (#19417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral server enrollment must preserve existing task files. > - Archived task directories can depend on encrypted retention checkpoints. > - A missing object blocks migration even when its authenticated contents were exactly empty. > - Metadata counters alone cannot prove that no saved files existed. > - This change verifies the exact candidate against the original encrypted key envelope before normal migration publication. > - The benefit is lossless migration of this bounded case without resetting task history or weakening ordinary restore. ## Linked Issues or Issue Description **What happened?** Migration stops at an archived empty directory when its retained encrypted object is missing. The original encrypted key envelope still authenticates the payload hash, byte length, company, workspace, checkpoint ID, and generation. **Expected behavior** Administrative migration can reconstruct those exact original bytes when storage confirms absence and every authenticated check passes. It must keep missing nonempty, expired, corrupt, or ambiguous sources blocked. **Steps to reproduce** 1. Capture and encrypt an empty task directory. 2. Retain its reference and encrypted key envelope. 3. Remove the old directory and storage object. 4. Run the normal remote workspace migration. **Paperclip version or commit** The regression fails before this change at `205ffcb606`. Related preservation work is in #19396, #19402, and #19409. No duplicate open fix was found. **Deployment mode** Administrative migration to external workspace storage. Tests use real encryption, real filesystem storage, and disposable PostgreSQL. ## What Changed - Let the encryption owner verify a candidate against the existing scoped encrypted key envelope. Normal ciphertext validation keeps its previous order and checks. - Share the task-directory encoder with an exact empty payload constructor. - Add a default-off loader option for confirmed missing empty checkpoints. Only the administrative migration caller enables it. - Keep the original checkpoint reference and object unchanged. Publish the recovered bytes through the existing snapshot and source-fence owners. - Reject forged counters, wrong identities, missing nonempty data, present corruption, unreadable present objects, unknown absence, and expiry. ## Verification - Final affected run: all 238 tests pass across five crypto, snapshot, retention, and migration suites. Full repository typecheck and build pass; server typecheck and full build also pass after the cancellation repair. - RED then GREEN: exact missing-empty recovery and cancellation of a stalled HEAD through the real storage-service boundary. Cancellation retains the caller's original abort reason. - Disposable PostgreSQL migration publishes a cold-restorable empty archive. Archived status, task status, task binding, original reference, and the absent original source stay unchanged. Replay keeps one canonical snapshot. - Negative cases reject forged empty counters, an authenticated directory or file, every foreign identity field, expiry, present corruption, unreadable present objects, unknown absence, and missing nonempty data. Ordinary restore keeps its previous failure behavior. - Initial verification retained one new fixture-path failure after 145 passes, and a typecheck rejected the fixture's widened schema literal. Correct the fixture to use the returned staging path and its explicit reference type. The final complete affected run passes. - The new HEAD request runs only after a failed read with the administrative option enabled. It uses the existing cancellation contract and adds no polling or background work. - Final-head full CI passes; review is 5/5 with no open findings. This PR does not claim deployed enrollment or a live migration result. ## Risks Recovery requires the original valid encrypted key envelope and an exact format match. A format change or unavailable key continues to block migration. Existing ciphertext hash, size, authentication, and ordinary restore behavior remain enforced. This change has no schema, provider-session, API, or UI contract changes. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
da34bcd48d |
fix(workspaces): resume archived imports from their saved upload (#19396)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspaces let a replacement web server restore saved task files. > - Legacy workspace imports keep a journal and an encrypted upload receipt across interruption. > - An archived Git import can rebuild a missing source from its saved task commit and edits. > - A fresh clone can also contain newer remote refs. Those refs change the archive manifest even when the task files are unchanged. > - This pull request recovers the exact authenticated upload for an interrupted archived Git import. > - The original journal, source binding, task binding, and execution checks still control publication. ## Linked Issues or Issue Description **What happened?** An archived Git workspace import stopped after its upload persisted. A retry rebuilt the source from the remote repository. New remote refs changed the archive manifest, so the original upload receipt rejected the retry with `remote_workspace_snapshot_operation_content_changed`. **Expected behavior** The retry must recover its original saved upload and complete the same import. It must keep the saved task commit, staged edits, unstaged edits, lifecycle state, and task binding. A changed source or task binding must still block publication. **Steps to reproduce** 1. Archive a Git task with staged and unstaged edits, save its restoration recipe, and remove its local repository. 2. Start the workspace import and lose the upload reply after object persistence, or interrupt its independent restore after the upload is acknowledged. 3. Advance the remote default branch and add an unrelated branch. 4. Retry the original import. Before this change, both interruption cases fail with the same content-change error. **Paperclip version or commit** Reproduced on master at `d8b0eae68772a3bd2bb4a8163b5ce95764b97cff`. **Deployment mode** Cloud workspace migration with external PostgreSQL and durable encrypted object storage. The regression tests use real disposable PostgreSQL, an HTTP Git remote, and encrypted filesystem-backed object storage. Related merged work: #18826 and #19138. No duplicate open fix was found. ## What Changed - Add scoped recovery of an acknowledged snapshot or a pending upload whose object persisted. Authenticate the archive before returning it. Recovery alone does not publish the snapshot. - Resume archived Git imports from those exact saved bytes. Keep the original operation and all publication checks. - Lock and recheck each saved nested source’s company, project, and repository identity in the final publication transaction. - Keep mutable local sources on the existing capture and content-comparison path. - Add regressions for both interruption points, unrelated remote changes, source revocation during recovery, and local edits after interruption. - Document the retry behavior. ## Verification - Before the repair: both new archived retry regressions fail with `remote_workspace_snapshot_operation_content_changed`; the mutable-source control passes. - After the repair: all 204 tests in the migration, snapshot storage, archive, and archive environment suites pass. The suites use real PostgreSQL and encrypted storage. - Server package typecheck and full repository typecheck pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. - An added source-revocation test initially expected a later expiry error. The existing publication fingerprint rejected the change first. The test now requires that earlier `remote_workspace_migration_publication_changed` error and still checks that authority was not published. - The review finding was reproduced before the safety repair: changing a saved nested repository URL after upload still allowed publication. The new regression requires rejection, and unchanged approved nested sources still restore. - Full repository typecheck and build pass on the final head. The full local test command stopped at one Slack manager authorization fixture with `socket hang up`; the exact case then passed in isolation on the same head, and all three CI chat shards passed. The original local failure is retained. - The first CI server shard run had 4,724 passes and one runtime-process readiness failure in an unchanged fixture. The exact fixture passed in isolation. One unchanged rerun of that shard passed. The first failure remains recorded in the workflow history. - The earlier candidate’s full local test run was stopped after the confirmed review finding; that interrupted run is not reported as a pass. - All CI gates, including `ci / verify` and `ci / e2e`, pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. Greptile is 5/5 on that head, with no remaining findings or unresolved review threads. ## Risks Recovery applies to archived Git sources, whose saved recipe and lifecycle checks still run before publication. A source change during recovery must keep the snapshot unpublished. Missing or corrupt saved objects still fail authentication or retain the existing import obligation. Mutable local sources still reject changed contents. This change adds no database migration and changes no saved archive format. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dbbde744d2 |
fix(workspaces): preserve finalized recovery holds through offline import (#19180)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Offline workspace import must save files before local volume
removal.
> - A task recovery hold can remain after native file finalization
succeeds.
> - The old importer rejects that hold even when its file source is
exact and no controller owns the run.
> - Clearing the hold would discard the task's recovery obligation.
> - This pull request saves files and carries each exact recovery source
forward without resolving the hold.
> - Accepted-result finalizers keep their existing retry path.
## Linked Issues or Issue Description
Refs #19144, #18943, #19038. This PR builds on merged #19144.
**What happened?**
Offline import reports `workspace_recovery_unresolved` for terminal
native runs with finalized workspace references and active task recovery
holds. Some holds also name an exact historical file allocation.
**Expected behavior**
Save the workspace durably. Keep task warnings, recovery rows, results,
coordinator phases and task pointers unchanged. Retain exact future file
recovery and accepted-result finalization retries.
**Steps to reproduce**
1. Retain a local task workspace and an active recovery hold for a
finalized native run.
2. Stop all application writers through offline maintenance.
3. Attempt workspace import.
4. Observe that the old importer rejects the hold.
## What Changed
- Add an owner module that checks the exact terminal native run,
finalized reference, released ownership and task relationship.
- Require the normal finalizer's successful company/run/task receipt for
an accepted-result retry. Support historical receipts without a
workspace foreign key.
- Validate recorded file allocations with the existing recovery-source
owner.
- Journal and recheck hold evidence. Lock recovery, coordinator, run and
allocation rows before publication.
- Give finalized file sources a separate migration marker. Keep legacy
execution-replay suppression out of accepted-result retry paths.
- Use the owning parser for older finalized references that omit
optional fields.
- Redirect future file recovery to immutable snapshot preservation. Keep
provider-stop and source checks intact.
- Verify native copyback or snapshot receipts for each exact allocation.
Require proof for every source, including completed mixed repairs.
- Retain existing successful copyback receipts. Keep a historical
folder's obligation on its real source workspace.
- Retain the finalized migration marker through the execution blocker
projection. Discover finalized child-task sources from their exact
ancestor workspace reference.
- Document the receipt and lock contracts.
## Verification
- Fresh local run at head
|
||
|
DottaandPaperclip
|
f5f5184f23 |
fix(workspaces): preserve retained branches and private commit sources (#19147)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must preserve existing task files before
volume removal.
> - An old private checkout can be absent while its recorded branch
still exists.
> - A deleted branch can also retain its exact commit in the original
private object source.
> - The importer currently rejects these surviving sources.
> - This pull request saves the verified committed tree and records its
exact source proof.
> - It keeps task state, branch ownership and recovery obligations
unchanged.
## Linked Issues or Issue Description
Refs #19138, #19144, #18826.
**What happened?**
Existing Git workspace records can have missing checkout folders and
retained branch objects. The importer returns
`local_source_unsafe_or_missing`. A successful deletion receipt can also
have a valid private object source that the importer overlooks in favor
of a missing or changed project cache.
**Expected behavior**
Save the surviving committed tree from the exact recorded branch or
deletion receipt. Reject competing owners, live worktree registrations,
changed branches and invalid source identities. Do not use the current
default branch as a substitute.
**Steps to reproduce**
1. Retain an isolated workspace branch in its managed project cache.
2. Remove its clean checkout but keep the branch and original workspace
record.
3. Attempt workspace migration.
4. Alternatively, retain a deleted commit in its recorded private source
while its shared source is unavailable.
## What Changed
- Pin a retained branch in its verified company and project Git cache.
- Require the recorded private path, no competing workspace owner and no
surviving Git worktree registration or index authority.
- For a user-owned branch, require its exact successful unforced removal
receipt after archival. Preserve the branch ownership metadata.
- Prefer a verified recorded private object source for successful
deletion receipts when that source still exists.
- Recheck branch and receipt proof before publication. Preserve
compatibility with existing migration journals.
- Allow missing active records only through the existing offline writer
fence. Keep their status unchanged.
- Add byte-restoration and rejected-authority tests. Document the source
limits.
- Revalidate and honor interrupted journals' original Git source path.
Reject detached registrations and retained orphan index data. Reuse the
cleanup producer command formatter for paths containing spaces.
## Verification
- 172 focused migration, concurrency and isolated-source tests pass with
real PostgreSQL and Git repositories.
- The previous importer fails the retained-branch and offline-active
restoration cases. The previous source selection also fails the
private-cache and clean user-owned receipt cases.
- Tests remove the old home and restore the pinned commit and task file
bytes. New shared default-branch work is excluded.
- Tests reject competing owners, registered worktrees, missing branch
ownership, deleted branches, changed branch tips, forced removal and
revoked receipts.
- Read-only re-inventory of 79 actual failed source records makes 78
eligible with committed owner overlays. One older local test repository
remains blocked. This is source-overlay qualification, not full
released-image acceptance.
- A separate bounded archive replay completed six imports before its
25-minute deadline. That timing failure remains open; it does not
establish complete migration.
- Local server typecheck has no changed-file diagnostics. Stale linked
declarations prevent a clean local whole-server result. Full clean CI
remains required.
- Serving enrollment and empty-home recovery remain separate acceptance
gates.
- Latest rebased head
|
||
|
DottaandPaperclip
|
c8013d06ad |
fix(workspaces): preserve task-only warnings during offline migration (#19144)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must save existing workspaces before volume
removal.
> - Offline migration checks file ownership and recovery authority.
> - The same gate also rejects task assignment and configuration
warnings.
> - Those warnings do not require the old filesystem.
> - This pull request preserves known task-only warnings while it saves
files.
> - File recovery and execution obligations keep their existing
blockers.
## Linked Issues or Issue Description
Refs #18943, #19038, #19138.
**What happened?**
Offline workspace migration refuses tasks with assignment,
configuration, disposition or deliberate-wait warnings. It returns
`workspace_recovery_unresolved` even when the warning has no file or
execution recovery authority.
**Expected behavior**
Save local workspace bytes while retaining the task warning unchanged.
Continue to block unknown recovery kinds, file sources, export failures
and execution recovery work.
**Steps to reproduce**
1. Retain a local workspace on a task with an active assignment or
configuration warning.
2. Stop all app writers through the offline maintenance entry.
3. Attempt workspace migration.
## What Changed
- Admit four known task-only warning kinds during offline file
migration.
- Keep file recovery markers, system recovery, native failure markers,
export-failure causes and unknown kinds blocked unless their exact
legacy source is already preserved.
- Keep the existing live-migration recovery exclusion.
- Recheck warning evidence before publication. Do not resolve warnings
or change their owners.
- Add restoration, rejection and publication-race tests. Update the
migration plan.
## Verification
- 112 focused migration and concurrency tests pass with real PostgreSQL
and filesystem archives.
- The four task-only restoration cases fail against the previous
importer with `workspace_recovery_unresolved`.
- Tests delete the old home, restore its saved bytes and compare each
warning row unchanged.
- File and execution markers, unknown kinds and export failures remain
blocked.
- A task warning that gains file recovery during upload prevents
publication.
- Local server typecheck has no changed-file diagnostics. Stale linked
dependency declarations prevent a clean local whole-server result. Clean
full CI remains required before merge.
- The PR was rebased onto the merged Git preservation fix after
resolving a plan-document conflict. The rebased head passed full PR CI
(41943153422). Current head
|
||
|
DottaandPaperclip
|
de9f263454 |
fix(workspaces): prevent preservation lock cycles (#18918)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces save files before a later task turn can use them. > - Admission, preparation and preservation share task, run and workspace rows. > - Snapshot inserts also lock their foreign-key parents. > - Different acquisition orders could form a PostgreSQL deadlock and delay saved-file recovery. > - This pull request orders those runtime parents and refuses busy legacy publication owners. > - Files and ownership evidence remain available without replaying provider work. ## Linked Issues or Issue Description **What happened?** Snapshot reservation and publication could hold a workspace while waiting for a task or run. Other workspace owners could hold that parent while waiting for the workspace. Review also found reverse preparation, provisioning and legacy-import paths. Real PostgreSQL owner-pair tests reproduce `40P01`. **Expected behavior** Concurrent owners must complete or retain their normal ownership refusal. A lock cycle must not interrupt file preservation. **Steps to reproduce** 1. Prepare a remote workspace with retained files in disposable PostgreSQL. 2. Schedule actual save and reserve, preserve, prepare or provision functions on opposing rows. 3. Release the scheduling barrier. 4. Unfixed source deadlocks. Fixed source saves the files or returns its normal ownership hold. The tests also schedule an actual legacy import beside an arriving reservation and a stale native status commit. **Paperclip version or commit** The original three save regressions fail on unchanged master `ce7766dc11c4515190d48d395f26a90bba900f8a`. Reverse preparation, provisioning and imported no-op controls fail on published `2d13b3f`. The additional import-publication and native-status owner pairs also have causal PostgreSQL RED/GREEN proof. **Deployment mode** Source checkout with PostgreSQL and real filesystem commands. Snapshot encryption and reload use local object storage. No hosted storage, live model or original-task recovery claim. Related: Refs #18908, whose preservation acceptance exposed this separate fault. This repair has no code dependency on that PR. Refs #17846 for separate legacy terminalization locks. Refs #18746 for the documented admission order. Repository search found no existing repair for this owner graph. ## What Changed - Lock company, current task and run before the runtime workspace. Recheck the run binding and pin save-operation attribution. - Apply that order to reserve, prepare, provision and both save transactions. New shared-workspace provisioning receipts use the current task. - Lock import source parents before reservation and publication. Recheck the source binding and retain the publication fingerprint. - Use full-strength `NOWAIT` locks for legacy coordinator, run and allocation rows. Only PostgreSQL `55P03` becomes `remote_workspace_migration_blocked:execution_rows_busy`. The publication transaction rolls back. Other errors propagate. - Preserve writer, epoch, snapshot, foreign-key, Stop and owner checks. Keep command, capture and upload I/O outside row-lock transactions. - Add actual-owner concurrency, unknown-effect, eventual retry and rollback controls. Document each owning transaction and its foreign-key edges. ## Verification - Node 24.22.2 and pnpm 9.15.4: all 107 save-concurrency, migration-concurrency and migration tests pass. - All 81 enabled disposable-PostgreSQL integration and lifecycle controls pass on the final head. Both opt-in database variables point to a dedicated local test database; no skipped case is counted as qualification. - Actual preparation/provision/import no-op peers reproduce three deadlocks on the prior source. The fixed owners preserve and reload files. The real provisioning command runs once; a lost response remains an unknown effect and is never replayed. - Actual legacy publication versus reserve and stale native status commit also reproduce deadlocks before repair. Busy run/lease controls preserve the journal, archive and files, then finish from the same journal after release. - A real publication constraint failure rolls back earlier coordinator, profile and snapshot writes. Its original database error propagates. - Immutable same-company and foreign-task guards, writer/epoch loss and uploaded-object retention pass. Four real shared/inherited reuse cases fail old receipt attribution and pass current-task attribution. - The original spontaneous issue-foreign-key failure did not retain its peer statement. These controlled owner pairs prove supported defects; they do not identify that historical peer. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-save-concurrency.test.ts server/src/services/remote-workspace-migration-concurrency.test.ts server/src/services/remote-workspace-migration.test.ts`. Enable the separate lifecycle suites only with their documented disposable test database variables. - Full recursive typecheck and `pnpm build` pass on production commit `dacf6278abd3257c407a297c6db8872e76d85f61`, using Node 24.22.2 and pnpm 9.15.4. Current head `4c1d7769e6cfa9536a67c4e3b536cef155f19966` changes only the canonical PostgreSQL host-support guards. All 18 changed concurrency cases executed and passed with zero skips; their server test-layer typecheck passes. All 57 current-head check contexts are terminal and clean: 53 passed, two skipped and two neutral. Required `ci / verify` and `ci / e2e` pass. Greptile and Paperclip both reviewed exact current head at 5/5 with no actionable findings. There are no open review threads. - Composition with #18908 includes this exact parent history. Its production-composed head passed 724 affected tests, all 81 enabled integration/lifecycle controls, recursive typecheck and build on Node 24. Its final test-only guard head passed the 18 changed concurrency cases and their test-layer typecheck. This is local server/PostgreSQL/native-backend-boundary evidence, not a hosted model or storage claim. - The required broad local Vitest attempt stopped at an unchanged chat publication-batch test with `socket hang up`. The earlier CI workspace shard timed out in the unchanged CLI chunked-ZIP test. Both failures remain recorded; neither is reported as a pass or hidden by timeout changes. The chat case passes one bounded focused head/base control, so the original full-suite socket failure was not reproduced by that control. ## Risks - Short transactions take parent-row locks earlier. Competing writers can wait for those transactions. - Administrative import now refuses a busy native owner. It retains its journal and archive for a later explicit retry. - Native owners have different lock orders. The import's nonblocking requests avoid a new global native lock policy. - Missing bindings and uncertain command effects remain holds. The repair does not infer termination, release an executor or replay external work. - No schema, foreign-key or timeout change. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this agent session. The agent used reasoning, terminal tools, code edits, PostgreSQL and filesystem tests. A separate Codex agent reviewed the production ownership graph. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84aa989bf9 |
Preserve native sessions through legacy workspace enrollment (#18943)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
82dd3bb702 |
Preserve unstarted queued turns during offline enrollment (#18887)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
0ab3ce7a03 |
fix(workspaces): preserve legacy recovery across storage migration (#18841)
## Thinking Path > - Paperclip manages durable tasks and agent work through the control plane. > - An ephemeral web server needs existing workspace files in shared storage. > - Failed historical runs can still own recoverable sandbox files or unfinished coordinator state. > - Moving local files must preserve those obligations without replaying the old agent turn. > - Migration journals the exact source and fences old copyback; the recovery owner saves verified sandbox files separately. > - New user input can continue only after matching preservation receipts and independent execution-stop checks. > - Accepted results retain their finalization owner and must have its durable workspace receipt before storage moves. ## Linked Issues or Issue Description Builds on merged #18826 and #18835. **What happened?** Existing workspaces with failed native copyback could not migrate to remote storage. Simply changing their storage authority would let later recovery target a retired local directory, lose child-task attachments, or leave a recovered task unable to accept a new turn. **Expected behavior** Preserve local files, exact sandbox provenance, task bindings, checkpoints, and unresolved history. Recover surviving sandbox files into immutable task-scoped attachments after migration, and let a new authorized user turn use those files without replaying the old execution. **Steps to reproduce** Create a terminal native run with a prepared copyback descriptor and a retained sandbox. Migrate its local workspace, remove the old host filesystem, recover the sandbox, and submit a new user turn. Also exercise a child task reusing its ancestor's workspace and an expired observed coordinator. ## What Changed - Journal exact legacy run, task, descriptor, and unfinished coordinator bindings during offline migration. Descriptor-free runs retain the complete declared lease/allocation/directory identities; publication locks and revalidates them. - Publish a run-scoped fence that prevents automatic restart or copyback into the retired local authority. - Release only the exact journaled expired coordinator lease under the offline deployment fence and locked revalidation, recording the prior owner in recovery history. Preserve phase, checkpoint, result, and run status. - Save recovered sandbox files through the existing immutable snapshot owner, retaining allocation and provider-stop checks. - Attach recovered files using the current task's identity when a child reuses an ancestor workspace. - Accept matching migration and snapshot receipts for workspace-repair admission and explicit new-user continuation. Changed coordinator bindings and invalid receipts remain blocked. - Require the finalizer's successful workspace_finalize operation before moving a workspace whose accepted result has finalized copyback; the existing owner can then finish assessment/status effects. - Document unresolved workspace-policy, task-ownership, and missing-provenance cases as separate owner actions that block detachment. ## Verification - 569 tests across migration, retained recovery, and explicit continuation passed on |
||
|
DottaandPaperclip
|
2a16dcf629 |
fix(workspaces): preserve shared directory trees during enrollment (#18835)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server needs existing task files in durable object
storage.
> - Historical workspace records can share a physical folder or contain
other task folders.
> - A task-level migration hold cannot fence every alias, and capturing
a parent recursively duplicates nested task data.
> - Offline enrollment imports children first and verifies their durable
coverage before omitting them from a parent archive.
> - Snapshot retention preserves those child versions for as long as a
covering parent depends on them.
> - Existing files, task bindings, and archived states remain intact
through the storage migration.
## Linked Issues or Issue Description
Builds on merged #18826.
**What happened?**
The per-workspace importer could capture a shared project folder while
another workspace record still owned the same path. It also recursively
included runtime-created nested task workspaces in each parent import.
Packaged enrollment rejected existing workspace data instead of invoking
its migration owner.
**Expected behavior**
Require offline maintenance for overlapping paths. Import nested tasks
before their parents, verify every excluded directory against its saved
snapshot, preserve dependencies through retention, and resume
interrupted imports without replacing existing remote authority.
**Steps to reproduce**
Create a project workspace and another task workspace below its
`.paperclip/worktrees/` directory. Try importing either while ordinary
writers are running, then run offline enrollment. Interrupt a parent
import after upload and resume it.
## What Changed
- Add physical-overlap admission checks and an offline entry-point
guard, including global running and queued executions.
- Import pending records in leaf-first pages through the existing
migration owner.
- Verify nested snapshots by downloading, restoring, and comparing the
original source before and after parent upload; check transitive
dependencies.
- Publish parent dependency receipts under child snapshot row locks.
Retention pins those children and revisits them when the parent is
deleted.
- Index imported-child dependency lookups with a partial GIN index; a
PostgreSQL query-plan regression covers 2,500 unrelated snapshots.
- Propagate cancellation through capture and object storage operations
and preserve existing retry journals.
- Wire workspace migration into packaged maintenance and document its
preservation contract.
## Verification
- The original overlap and duplicate-parent regressions failed before
implementation.
- 73 migration and retention tests pass against real PostgreSQL and
files, including interrupted uploads, resumed batches, equal-path
aliases, changed or missing child data, nested descendants, and active
or queued executions.
- Six S3-compatible storage integration tests pass, including migration
through the maintenance owner and restoration of both parent and child
after deleting the original filesystem.
- The standalone process case was initially skipped because the worktree
had no bundle. After rebuilding the server and runtime bundle, that test
passed against PostgreSQL and S3: authenticated ready/export receipts,
preserved decision secret, blocked application routes, and clean SIGTERM
shutdown.
- Server typecheck, server build, runtime bundle build, and
module-boundary checks passed. Full repository CI passed on
|
||
|
DottaandPaperclip
|
2035118526 |
fix(workspaces): preserve archived data during remote migration (#18826)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace storage lets the web server run without a persistent volume. > - Existing deployments also contain archived and historical task workspaces. > - Retention can remove their folders while preserving encrypted restore points or Git reconstruction records. > - The importer rejected those records and only accepted the task's current workspace. > - This pull request imports verified saved data without reopening or rebinding tasks. > - The benefit is preservation of existing work during an explicit storage migration. ## Linked Issues or Issue Description Refs #18403 and #17472. **What happened?** An explicit remote-workspace import rejected archived records, historical task bindings, and removed folders with valid retained data. **Expected behavior** Import the recorded files or exact Git commit. Preserve archived state and the task's current binding. Refuse missing, expired, corrupt, or changed sources. **Steps to reproduce** Archive a task directory with a restore point, remove its folder, and run the remote-workspace import. Repeat with a Git restore point containing an unpushed commit and staged and unstaged edits. The previous importer reports an inactive workspace or missing source. ## What Changed - Preserve lifecycle state and current task binding while importing archived and historical records. Recheck the binding before publication and on resume. - Expose the task-directory owner's validated restore-point reader. Reuse it for ordinary restoration and migration. - Restore archived directories and Git data into private scratch space. Support saved Git recipes and exact legacy branch-deletion receipts. - Recover pruned unpushed Git commits from the encrypted bundle and its explicit prerequisites. - Keep independent snapshot download, restoration, source checks, writer guards, encryption, and atomic publication. - Document the supported import sources and the remaining tenant-enrollment gaps. ## Verification - Before production edits, two new regressions failed on archived status and historical task binding. - Final focused verification: 58 tests pass across remote-workspace migration and isolated task-directory restoration. - Real PostgreSQL, Git repositories, encrypted storage, and restored file bytes cover cold restoration after removing the old folder and source object, pruned unpushed history, index and worktree edits, corrupt/missing/expired sources, binding changes, and source-folder races. - A Git fixture initially lacked runtime ownership metadata. Correcting the fixture made the valid retained-source case pass; no production validation was weakened. - Full repository typecheck and build passed. Full `pnpm test:run` stopped in chat integration shard 2 after two failures: a retry-notice timeout and a subsequent fake-timer error. All four cases in that retry-notice group passed in a separate focused rerun without source changes; the complete suite has not passed locally. All exact-head CI jobs passed. The separate Paperclip Review app check is still queued; all existing review threads are resolved. Server typecheck passed after the review fixes. - Both review findings were reproduced: the real cleanup owner's verified-tip receipt was rejected, and a lost upload reply for a legacy Git source failed with `operation_content_changed`. The fixes accept the owner's scoped receipt metadata and rebuild reconstructed Git indexes from their staged trees. Both regressions now pass, including retained staged and unstaged edits. - Additional coverage restores from a fresh HTTP clone after deleting the original repository, imports a nested local checkout with encrypted settings, and rejects an unapproved nested source without publishing remote authority. - This PR does not claim a completed tenant enrollment or a new zero-downtime deployment proof. ## Risks Imports can fail when old receipts or repository objects are unavailable. They retain their migration journal and leave the original state intact. Operators must resolve pending writers and recovery before importing. Shared physical roots, nested worktrees, and unresolved sandbox recovery require separate migration work. This change does not remove those blockers or alter automatic retention. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task ID or instance-derived details - [x] I have run the focused tests locally and they pass; full typecheck/build passed and the full-suite failure is documented - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5e4cc01073 |
feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |