PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-08 12:33:05 +01:00

Commit Graph

Author	SHA1	Message	Date
Dotta Paperclip Devin Foley	7d1e9b3477	fix(recovery): retain exact warm native export sources (#17916 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task recovery must save unfinished workspace exports before continuation. > - A prepared export can still refer to a sandbox with a warm reuse policy. > - The strict recovery inspector rejects that policy and exhausts source verification. > - Ordinary reuse must not give the same unfinished allocation to another run. > - An existing user Retry can authorize bounded file maintenance after exact ownership checks. > - This change retains that exact source and preserves the original failure record. > - Recovery completes only when files, stop evidence and policy settlement agree. ## Linked Issues or Issue Description What happened? A failed native run has a prepared `keep_running` workspace export. Its exact sandbox lease still has `reuse_by_environment` policy. Native export cleanup can preserve that policy while it stops the sandbox. Generic workspace recovery then rejects the source with `lease_policy_not_retained`. A saved user Retry remains blocked before file export. Expected behavior Keep unfinished exports out of ordinary warm reuse. With current user authority and an unchanged saved Retry, retain the exact source for one bounded maintenance generation. Preserve files and return continuation to normal admission. Keep all ambiguous sources held. Steps to reproduce 1. Prepare a native workspace export with a warm sandbox and `keep_running` disposition. 2. Save a normal board Retry for the failed source run. 3. Persist its first exhausted source-verification failure before file export. 4. Run the normal maintenance worker. Before this change, source policy rejection leaves the saved files unrestored. Refs #17773 for source diagnostics. Related #17839 records a missing preservation obligation during blocked restart. This change handles an existing prepared warm export whose verification already failed. ## What Changed - Exclude unfinished physical allocations from reuse and obsolete-lease cleanup across every historical lease row. Reuse a clean candidate or acquire a fresh sandbox through normal selection. Preserve exact same-run reattachment, workspace-less execution and ordinary task admission checks. - Preserve the newer prepared remote-workspace writer proof for native and legacy runs. Share its binding check between single-source recovery and one batched allocation read. - Keep persisted recovery-action IDs when constructing continuation source checks. Synthetic sources cannot grant snapshot receipt authority. - Permit one audited maintenance adoption for the first exhausted source-verification attempt. Require an unadmitted canonical saved board Retry, current user authority, unchanged assignment and latest source run. - Keep all company, source, allocation, workspace writer, controller and physical finalization ownership checks. The public read-only inspector keeps its strict policy check. - Pin the exact source for file maintenance. Preserve the previous generation and reuse the existing bounded recovery worker. Create no provider turn or new user request. - Restore the warm policy in the same transaction as verified native copyback completion. A failed write or changed settlement keeps a recovery obligation. - When local merge metadata is missing, settle the exact verified snapshot while keeping its unfinished allocation retained. Deliver the saved Retry through normal fresh isolation and attach the verified files. Keep the accepted result as history. - Require native copyback proof before reopening an accepted result's original workspace finalizer. - Validate snapshot receipts through one shared boundary for warm, retained and ephemeral sources. Bind the receipt to the current generation's operation IDs, company, issue, run, recovery action, lease and provider allocation. - Retire fixture input before removing recovery proof, and use the same configured heartbeat for HTTP and worker dispatch. Keep the exact catalog refusal and no-provider-work assertions. - Document the recovery contract in technical documentation. Leave capability maps unchanged for this bug fix. ## Verification - Candidate-selection RED: all six clean-candidate and fresh-allocation controls fail on unchanged `f54e7baf538` because a pending export aborts the whole acquisition. GREEN: the focused suite passes 15/15 cases after candidate filtering. Both provider paths preserve the protected lease and never resume or destroy its allocation. Obsolete fingerprints cannot send that allocation to cleanup. Historical-row fencing, modern native/legacy writers, exact same-run reuse, workspace-less execution and the single-read allocation-history check remain covered. - The existing reusable-runtime selection cases passed 40/40. Driver/capability contract suites passed 5/5. Reusable service controls passed 2/2. After merging current master, the combined lease/provider suites pass 61 cases and the recovery integration selection passes 36 cases, including the upstream inherited-session and resolved-interruption assertions. This gives 97 passing targeted cases on the final source. A duplicate test-app declaration from the automatic merge was removed while retaining the upstream assertions. - The source-history repair has meaningful red/green evidence: three valid retired-handoff cases could not complete under the old blanket history-row rejection. Settlement now uses the same full source inspector as adoption. It accepts exact retired handoffs and still rejects a predecessor that becomes active after export. The 26 focused history, rollback, snapshot and ambiguity cases passed before this last merge. - Server typecheck passes on the final merged source. Module boundaries and `git diff --check` pass. Full local repository tests and a successful current-head full build are not claimed. Current-head CI is complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required gates pass, with 54 successful checks, three expected skips, one neutral Paperclip review check and a successful security status. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370). Greptile reviewed that exact head at 5/5 with no actionable findings. Its check completed successfully. [Ponytail also reviewed the exact head at 5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408). No review threads remain unresolved, and the branch is mergeable. - Earlier local broad runs exposed a dropped recovery-action ID in the merged source projection and deferred test input being revisited after its records were removed. Both were repaired. Later local disk exhaustion caused manifest and saved-file reserve failures; generated caches were removed only from these checkouts. The affected saved-file control passed unchanged after cleanup. The original logs are retained. No disk reserve or timeout was lowered. - On the prior head, the first [full CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530) received simultaneous runner shutdowns. Four jobs were cancelled, and two late jobs have lost-runner annotations with unavailable logs. The Runner job had five earlier ACPX failures whose full assertion report was lost on interruption. Their cause remains unproven. Identical Runner source passed the [#19004 control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725), including all seven enabled tests in that file. - That attempt's [browser shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595) returned to the exact saved task URL, but its conversation was still loading at the unchanged heading deadline. The [artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931) is retained. All five Cloud-auth cases passed locally on the unchanged head and in the [same-source CI control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966). The delay remains unexplained. One rerun of failed/interrupted jobs was submitted after the workflow completed. This new source revision requires fresh CI. - The earlier [original-head workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175) passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5. These results do not establish readiness for the new head. The earlier Superagent scan was neutral after a pinned-base checkout failure; its external analysis remains unverified. - These tests use real PostgreSQL and controlled provider boundaries. Hosted rollout and original-task acceptance remain unverified. ## Risks - Maintenance can resume and stop the exact sandbox for file export. It cannot admit a provider turn, choose another allocation or bypass normal continuation gates. - A saved Retry alone is insufficient. A changed source, newer run, active writer, competing lease, lost user authority or previous adoption keeps the hold. - Policy restoration can fail. Completion and restoration now commit together, so normal recovery remains responsible for a retry. - Snapshot preservation cannot repair missing local merge metadata. Its old allocation remains explicitly retained. A snapshot receipt grants no permission to reuse or destroy that allocation, or to replay the accepted result. - The allocation check uses one projected join per candidate over the existing provider lease index. It adds no polling timer or recurring scan. A fresh fallback can create another sandbox while the unfinished source remains retained; it grants no authority to execute a provider turn. - This change does not establish the cause of a prior provider failure or prove that any specific hosted sandbox still contains its original files. ## Model Used OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs. Both used repository tools and PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>	2026-11-04 14:47:42 -08:00
DottaandPaperclip	367f29478d	fix(recovery): preserve the bound remote writer before continuation (#18803 ) ## Thinking Path > - Paperclip manages agent work and preserves task files. > - A remote workspace records the exact run that owns its unsaved files. > - A later setup failure can become the latest task run without owning those files. > - Retry only inspected that latest run, so the original writer had no file-recovery owner. > - A released reusable source also failed the legacy retention rule. > - This change registers the bound writer, saves its files, and then admits the saved user request once. > - Existing stop, ownership, approval and execution-evidence checks remain effective. ## Linked Issues or Issue Description What happened? A native run prepared a remote workspace and failed before finalization. Its sandbox stopped with a dedicated stop receipt. The workspace stayed dirty and retained that run as its writer. A later setup failure became the latest task run. Retry inspected the newer run and left the original file obligation unregistered. New attempts failed at the writer fence. Registration alone was insufficient. After file recovery, continuation still required the Retry source to be one of the older recovery sources. A newer pre-provider failure did not satisfy that check. Workspace selection could also choose a fresh directory instead of the recovered files. Expected behavior Find the writer through the task's current workspace binding. Register only its exact prepared source. Preserve the released lease and its stop receipts. Save the files before continuation. Independently prove that a newer setup failure never dispatched work. Select the verified saved workspace and admit the Retry once. Steps to reproduce 1. Fail a native run with a prepared remote workspace and a released reusable sandbox. 2. Keep matching dedicated stop receipts and the workspace's dirty writer authority. 3. Record a newer pre-provider setup failure for the same task. 4. Request Retry. Before the fix, no file-maintenance record owns the actual writer. Paperclip version or commit Reproduced before production edits on master `f7ad1dc7ca`. The branch was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`. Deployment mode Authenticated hosted execution with Paperclip Runner and remote workspace storage. Companion: #18798 (now merged upstream) fixes a pre-existing inbox test synchronization race. Its test-only commit `110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with attribution so this PR can run the complete CI matrix. Related: #17800 registers missing preservation for the selected source run. #18678 holds unfinished remote exports. #18718 repairs a prior warm-controller projection failure. #17916 addresses a separate legacy warm-copyback policy case. ## What Changed - Discover the failed writer through the company-scoped task/workspace binding during Retry and authenticated message admission. - Reuse the exact prepared-source validator for released remote workspaces. Registration changes neither the lease policy nor its stop receipts. Physical recovery retains its full allocation checks. - Check a newer failed preparation separately before allowing an older, repaired source to release continuation. Provider events, leases, results and execution ownership remain disqualifying evidence. - Select the bound saved workspace through the continuation's verified repair receipt. - Add PostgreSQL safety cases and an HTTP Retry workflow with real file capture and saved-copy reads. Document the execution contract. - Reuse #18798's completion barrier for the existing inbox cancellation test; preserve its eight-slot limit, rejection and coalescing assertions. ## Verification - RED before production edits: the exact stopped writer has no `workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass on the baseline. - GREEN: the expanded Retry/message/preview selection passes 17 cases. All 627 tests in the five affected recovery suites pass on recovery commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop, ownership, saved messages and duplicate Retry. - The HTTP workflow registers maintenance, keeps execution held before save, reads back the saved file bytes, and admits exactly one continuation with `resume_preserved_workspace`. A registration-only repair first failed after save because it did not reconcile the newer Retry source. The incident-shaped pair passes with an independent active stranded-task action present. - Greptile identified the path where the newer setup failure already owns a reconciliation hold. Its new regression failed before the correction (saved files, zero queued continuations); the contradictory-dispatch control passed. All four HTTP variants pass after the correction, covering both hold shapes and both dispatch-evidence controls. - Repository typecheck and build passed. Server typecheck and build also passed after the recovery changes. Module-boundary and whitespace checks pass. - The full local Vitest command failed in the GitHub fast-response acknowledgement test (`delivery_unknown`: one response write instead of zero). That isolated case passes on this branch and unchanged master; the original full-run cause remains unestablished. A sibling run passed 199 test cases but timed out in workspace-suite database cleanup. Unchanged master also passed all 126 workspace cases and timed out in the same cleanup hook; all five acknowledgement outcomes passed in that baseline comparison. These local failures remain recorded rather than being reported as suite passes. - The first hosted matrix failed the pre-existing inbox cancellation assertion (expected nine computes, saw eight). The same assertion failed in [master CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840) and reproduced locally before applying #18798's existing repair. All six inbox HTTP/database cases pass after that repair. Final head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed test change to the verified recovery code. - [Final hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377) passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci / verify`, `ci / e2e`, typecheck, build, every server/chat/workspace group, Runner checks and all eight browser shards. There are 54 successful check results, two informational neutral results and three skipped results, with no pending or failing results. - Greptile and the Paperclip reviewer both report 5/5 for that final head; no review threads remain open. The Paperclip review check is informational (`neutral`) and records its local-testing limitations. - The file workflow uses real PostgreSQL, subprocess file capture, encrypted saved objects and production recovery code. Provider resume/stop calls are simulated at the environment driver boundary. This does not prove a live Daytona or Codex recovery. No deployment or original-task retry is claimed. ## Risks - File-maintenance discovery must not authorize another executor. Existing writer, coordinator and physical allocation checks still own that decision. - A false bootstrap summary cannot prove that a newer run never executed. The new path also requires its immutable native-adapter claim, unresolved preparation state, absent coordinator/process/session/lease/result evidence, and no dispatch events. - Additional database reads run only during explicit task admission or Retry strategy selection. They are company/task scoped and add no polling loop. The saved-writer query returns one matching snapshot and omits prompt bodies. - No schema, credential, provider profile or deployment setting changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code execution and test tools. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:25:24 -06:00
DottaandPaperclip	5e4cc01073	feat(workspaces): keep remote files off the webserver disk (#18403 ) Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 14:54:01 -05:00

Author

SHA1

Message

Date

7d1e9b3477

fix(recovery): retain exact warm native export sources (#17916 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task recovery must save unfinished workspace exports before
continuation.
> - A prepared export can still refer to a sandbox with a warm reuse
policy.
> - The strict recovery inspector rejects that policy and exhausts
source verification.
> - Ordinary reuse must not give the same unfinished allocation to
another run.
> - An existing user Retry can authorize bounded file maintenance after
exact ownership checks.
> - This change retains that exact source and preserves the original
failure record.
> - Recovery completes only when files, stop evidence and policy
settlement agree.

## Linked Issues or Issue Description

**What happened?**

A failed native run has a prepared `keep_running` workspace export. Its
exact sandbox lease still has `reuse_by_environment` policy. Native
export cleanup can preserve that policy while it stops the sandbox.
Generic workspace recovery then rejects the source with
`lease_policy_not_retained`. A saved user Retry remains blocked before
file export.

**Expected behavior**

Keep unfinished exports out of ordinary warm reuse. With current user
authority and an unchanged saved Retry, retain the exact source for one
bounded maintenance generation. Preserve files and return continuation
to normal admission. Keep all ambiguous sources held.

**Steps to reproduce**

1. Prepare a native workspace export with a warm sandbox and
`keep_running` disposition.
2. Save a normal board Retry for the failed source run.
3. Persist its first exhausted source-verification failure before file
export.
4. Run the normal maintenance worker. Before this change, source policy
rejection leaves the saved files unrestored.

Refs #17773 for source diagnostics. Related #17839 records a missing
preservation obligation during blocked restart. This change handles an
existing prepared warm export whose verification already failed.

## What Changed

- Exclude unfinished physical allocations from reuse and obsolete-lease
cleanup across every historical lease row. Reuse a clean candidate or
acquire a fresh sandbox through normal selection. Preserve exact
same-run reattachment, workspace-less execution and ordinary task
admission checks.
- Preserve the newer prepared remote-workspace writer proof for native
and legacy runs. Share its binding check between single-source recovery
and one batched allocation read.
- Keep persisted recovery-action IDs when constructing continuation
source checks. Synthetic sources cannot grant snapshot receipt
authority.
- Permit one audited maintenance adoption for the first exhausted
source-verification attempt. Require an unadmitted canonical saved board
Retry, current user authority, unchanged assignment and latest source
run.
- Keep all company, source, allocation, workspace writer, controller and
physical finalization ownership checks. The public read-only inspector
keeps its strict policy check.
- Pin the exact source for file maintenance. Preserve the previous
generation and reuse the existing bounded recovery worker. Create no
provider turn or new user request.
- Restore the warm policy in the same transaction as verified native
copyback completion. A failed write or changed settlement keeps a
recovery obligation.
- When local merge metadata is missing, settle the exact verified
snapshot while keeping its unfinished allocation retained. Deliver the
saved Retry through normal fresh isolation and attach the verified
files. Keep the accepted result as history.
- Require native copyback proof before reopening an accepted result's
original workspace finalizer.
- Validate snapshot receipts through one shared boundary for warm,
retained and ephemeral sources. Bind the receipt to the current
generation's operation IDs, company, issue, run, recovery action, lease
and provider allocation.
- Retire fixture input before removing recovery proof, and use the same
configured heartbeat for HTTP and worker dispatch. Keep the exact
catalog refusal and no-provider-work assertions.
- Document the recovery contract in technical documentation. Leave
capability maps unchanged for this bug fix.

## Verification

- Candidate-selection RED: all six clean-candidate and fresh-allocation
controls fail on unchanged `f54e7baf538` because a pending export aborts
the whole acquisition. GREEN: the focused suite passes **15/15 cases**
after candidate filtering. Both provider paths preserve the protected
lease and never resume or destroy its allocation. Obsolete fingerprints
cannot send that allocation to cleanup. Historical-row fencing, modern
native/legacy writers, exact same-run reuse, workspace-less execution
and the single-read allocation-history check remain covered.
- The existing reusable-runtime selection cases passed **40/40**.
Driver/capability contract suites passed **5/5**. Reusable service
controls passed **2/2**. After merging current master, the combined
lease/provider suites pass 61 cases and the recovery integration
selection passes 36 cases, including the upstream inherited-session and
resolved-interruption assertions. This gives 97 passing targeted cases
on the final source. A duplicate test-app declaration from the automatic
merge was removed while retaining the upstream assertions.
- The source-history repair has meaningful red/green evidence: three
valid retired-handoff cases could not complete under the old blanket
history-row rejection. Settlement now uses the same full source
inspector as adoption. It accepts exact retired handoffs and still
rejects a predecessor that becomes active after export. The 26 focused
history, rollback, snapshot and ambiguity cases passed before this last
merge.
- Server typecheck passes on the final merged source. Module boundaries
and `git diff --check` pass. Full local repository tests and a
successful current-head full build are not claimed. Current-head CI is
complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required
gates pass, with 54 successful checks, three expected skips, one neutral
Paperclip review check and a successful security status. [Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370).
Greptile reviewed that exact head at **5/5** with no actionable
findings. Its check completed successfully. [Ponytail also reviewed the
exact head at
5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408).
No review threads remain unresolved, and the branch is mergeable.
- Earlier local broad runs exposed a dropped recovery-action ID in the
merged source projection and deferred test input being revisited after
its records were removed. Both were repaired. Later local disk
exhaustion caused manifest and saved-file reserve failures; generated
caches were removed only from these checkouts. The affected saved-file
control passed unchanged after cleanup. The original logs are retained.
No disk reserve or timeout was lowered.
- On the prior head, the first [full CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530)
received simultaneous runner shutdowns. Four jobs were cancelled, and
two late jobs have lost-runner annotations with unavailable logs. The
Runner job had five earlier ACPX failures whose full assertion report
was lost on interruption. Their cause remains unproven. Identical Runner
source passed the [#19004
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725),
including all seven enabled tests in that file.
- That attempt's [browser
shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595)
returned to the exact saved task URL, but its conversation was still
loading at the unchanged heading deadline. The
[artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931)
is retained. All five Cloud-auth cases passed locally on the unchanged
head and in the [same-source CI
control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966).
The delay remains unexplained. One rerun of failed/interrupted jobs was
submitted after the workflow completed. This new source revision
requires fresh CI.
- The earlier [original-head
workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175)
passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5.
These results do not establish readiness for the new head. The earlier
Superagent scan was neutral after a pinned-base checkout failure; its
external analysis remains unverified.
- These tests use real PostgreSQL and controlled provider boundaries.
Hosted rollout and original-task acceptance remain unverified.

## Risks

- Maintenance can resume and stop the exact sandbox for file export. It
cannot admit a provider turn, choose another allocation or bypass normal
continuation gates.
- A saved Retry alone is insufficient. A changed source, newer run,
active writer, competing lease, lost user authority or previous adoption
keeps the hold.
- Policy restoration can fail. Completion and restoration now commit
together, so normal recovery remains responsible for a retry.
- Snapshot preservation cannot repair missing local merge metadata. Its
old allocation remains explicitly retained. A snapshot receipt grants no
permission to reuse or destroy that allocation, or to replay the
accepted result.
- The allocation check uses one projected join per candidate over the
existing provider lease index. It adds no polling timer or recurring
scan. A fresh fallback can create another sandbox while the unfinished
source remains retained; it grants no authority to execute a provider
turn.
- This change does not establish the cause of a prior provider failure
or prove that any specific hosted sandbox still contains its original
files.

## Model Used

OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs.
Both used repository tools and PostgreSQL verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>

2026-11-04 14:47:42 -08:00

DottaandPaperclip

367f29478d

fix(recovery): preserve the bound remote writer before continuation (#18803 )

## Thinking Path

> - Paperclip manages agent work and preserves task files.
> - A remote workspace records the exact run that owns its unsaved
files.
> - A later setup failure can become the latest task run without owning
those files.
> - Retry only inspected that latest run, so the original writer had no
file-recovery owner.
> - A released reusable source also failed the legacy retention rule.
> - This change registers the bound writer, saves its files, and then
admits the saved user request once.
> - Existing stop, ownership, approval and execution-evidence checks
remain effective.

## Linked Issues or Issue Description

**What happened?**

A native run prepared a remote workspace and failed before finalization.
Its sandbox stopped with a dedicated stop receipt. The workspace stayed
dirty and retained that run as its writer. A later setup failure became
the latest task run. Retry inspected the newer run and left the original
file obligation unregistered. New attempts failed at the writer fence.

Registration alone was insufficient. After file recovery, continuation
still required the Retry source to be one of the older recovery sources.
A newer pre-provider failure did not satisfy that check. Workspace
selection could also choose a fresh directory instead of the recovered
files.

**Expected behavior**

Find the writer through the task's current workspace binding. Register
only its exact prepared source. Preserve the released lease and its stop
receipts. Save the files before continuation. Independently prove that a
newer setup failure never dispatched work. Select the verified saved
workspace and admit the Retry once.

**Steps to reproduce**

1. Fail a native run with a prepared remote workspace and a released
reusable sandbox.
2. Keep matching dedicated stop receipts and the workspace's dirty
writer authority.
3. Record a newer pre-provider setup failure for the same task.
4. Request Retry. Before the fix, no file-maintenance record owns the
actual writer.

**Paperclip version or commit**

Reproduced before production edits on master `f7ad1dc7ca`. The branch
was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`.

**Deployment mode**

Authenticated hosted execution with Paperclip Runner and remote
workspace storage.

Companion: #18798 (now merged upstream) fixes a pre-existing inbox test
synchronization race. Its test-only commit
`110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with
attribution so this PR can run the complete CI matrix.

Related: #17800 registers missing preservation for the selected source
run. #18678 holds unfinished remote exports. #18718 repairs a prior
warm-controller projection failure. #17916 addresses a separate legacy
warm-copyback policy case.

## What Changed

- Discover the failed writer through the company-scoped task/workspace
binding during Retry and authenticated message admission.
- Reuse the exact prepared-source validator for released remote
workspaces. Registration changes neither the lease policy nor its stop
receipts. Physical recovery retains its full allocation checks.
- Check a newer failed preparation separately before allowing an older,
repaired source to release continuation. Provider events, leases,
results and execution ownership remain disqualifying evidence.
- Select the bound saved workspace through the continuation's verified
repair receipt.
- Add PostgreSQL safety cases and an HTTP Retry workflow with real file
capture and saved-copy reads. Document the execution contract.
- Reuse #18798's completion barrier for the existing inbox cancellation
test; preserve its eight-slot limit, rejection and coalescing
assertions.

## Verification

- RED before production edits: the exact stopped writer has no
`workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass
on the baseline.
- GREEN: the expanded Retry/message/preview selection passes 17 cases.
All 627 tests in the five affected recovery suites pass on recovery
commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop,
ownership, saved messages and duplicate Retry.
- The HTTP workflow registers maintenance, keeps execution held before
save, reads back the saved file bytes, and admits exactly one
continuation with `resume_preserved_workspace`. A registration-only
repair first failed after save because it did not reconcile the newer
Retry source. The incident-shaped pair passes with an independent active
stranded-task action present.
- Greptile identified the path where the newer setup failure already
owns a reconciliation hold. Its new regression failed before the
correction (saved files, zero queued continuations); the
contradictory-dispatch control passed. All four HTTP variants pass after
the correction, covering both hold shapes and both dispatch-evidence
controls.
- Repository typecheck and build passed. Server typecheck and build also
passed after the recovery changes. Module-boundary and whitespace checks
pass.
- The full local Vitest command failed in the GitHub fast-response
acknowledgement test (`delivery_unknown`: one response write instead of
zero). That isolated case passes on this branch and unchanged master;
the original full-run cause remains unestablished. A sibling run passed
199 test cases but timed out in workspace-suite database cleanup.
Unchanged master also passed all 126 workspace cases and timed out in
the same cleanup hook; all five acknowledgement outcomes passed in that
baseline comparison. These local failures remain recorded rather than
being reported as suite passes.
- The first hosted matrix failed the pre-existing inbox cancellation
assertion (expected nine computes, saw eight). The same assertion failed
in [master
CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840)
and reproduced locally before applying #18798's existing repair. All six
inbox HTTP/database cases pass after that repair. Final head
`7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed
test change to the verified recovery code.
- [Final hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377)
passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci
/ verify`, `ci / e2e`, typecheck, build, every server/chat/workspace
group, Runner checks and all eight browser shards. There are 54
successful check results, two informational neutral results and three
skipped results, with no pending or failing results.
- Greptile and the Paperclip reviewer both report 5/5 for that final
head; no review threads remain open. The Paperclip review check is
informational (`neutral`) and records its local-testing limitations.
- The file workflow uses real PostgreSQL, subprocess file capture,
encrypted saved objects and production recovery code. Provider
resume/stop calls are simulated at the environment driver boundary. This
does not prove a live Daytona or Codex recovery. No deployment or
original-task retry is claimed.

## Risks

- File-maintenance discovery must not authorize another executor.
Existing writer, coordinator and physical allocation checks still own
that decision.
- A false bootstrap summary cannot prove that a newer run never
executed. The new path also requires its immutable native-adapter claim,
unresolved preparation state, absent
coordinator/process/session/lease/result evidence, and no dispatch
events.
- Additional database reads run only during explicit task admission or
Retry strategy selection. They are company/task scoped and add no
polling loop. The saved-writer query returns one matching snapshot and
omits prompt bodies.
- No schema, credential, provider profile or deployment setting changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution and test tools. The runtime does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-03 11:25:24 -06:00

DottaandPaperclip

5e4cc01073

feat(workspaces): keep remote files off the webserver disk (#18403 )

Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured.

Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion.

Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-10-31 14:54:01 -05:00