mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
0b2b7aa737dfeee4b2e57287306bd5d3060a7e98
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
df89950eb6 |
fix(native): save or set aside a superseded run's unsaved workspace so its task queue moves (#21136)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result. Then a finalizer exports the run's workspace files and applies the result to the task. > - When a newer run already decided the task, the finalizer retires the older result as superseded. Since #21106, this also ends the run. > - If that retirement interrupts a failed workspace export, the run's files stay unsaved. The run also stays the writer of the task's workspace. Nothing records those files for recovery, so nothing ever releases the workspace. > - Every run queued on the task then waits behind a save that never settles. After one hour, each queued run fails setup instead. > - This pull request records those files for the existing automatic workspace recovery. Recovery saves them from the retained sandbox, or, when every bounded attempt fails, continues the task from its last save with the files attached. > - The benefit is that the task's queue moves again by itself, and the run's unsaved work is saved or kept findable. The superseded result is never applied. ## Linked Issues or Issue Description Refs #21106 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. Its workspace export failed once with `remote_workspace_archive_fs_eacces`. The finalizer then retired it as `native_finalization_superseded`, because another run had already decided the task. #21106 ended that run at 14:59:17Z (`failed`). The task still could not move. The next queued run was cancelled with `workspace_busy` ("ended, but its workspace save has not settled"). Its retry re-armed every minute. Four more runs stayed queued. The task showed no recovery action. **Expected behavior** The ended run's unsaved files are saved from its retained sandbox, and the workspace is released. If they cannot be saved within the bounded retries, the task continues from its last committed save, and the files are attached to the task. The queued runs proceed in both cases. The superseded result is never applied. **Steps to reproduce** 1. Accept a native result for a run that writes a remote workspace. 2. Let the workspace export fail once, while a newer run has already committed the task's decision. 3. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is `failed`. 4. See that the workspace still names the run as its dirty writer, and that no retained-file recovery action exists. 5. Queue another run on the task. It defers with `workspace_busy`, and after one hour it fails setup with "The workspace still has an unfinished writer or save operation." **Root cause** `recordRetryableFailure` (`native-run-finalizer.ts`) retires a superseded finalizer without preserving the export. `preserveNativeFailedWorkspace` records a retained-file hold for an accepted result only when the failure code is `native_workspace_sync_out_retry_exhausted`. A `native_finalization_superseded` coordinator never qualifies. `readRemoteWorkspaceWriterHold` keeps an ended dirty writer as the holder until its save settles. Without a hold, no worker ever settles it. **Paperclip version or commit** Staging Core with #21106 (`81d2e4d8d9`). Master `ef2b53b675` has the same behavior. ## What Changed - `native-superseded-workspace-export.ts` (new): identifies a superseded finalizer that failed in the workspace export. It selects ended runs that still hold their workspace with unsaved files and no retained-file hold. It records the hold through `preserveNativeFailedWorkspace`. It checks again only while a just-ended run's sandbox stop receipt is still arriving (at most one hour). - `native-failed-workspace-recovery.ts`: `preserveNativeFailedWorkspace` treats a superseded workspace export like an exhausted copy-back, with the same proofs (run ended, no live controller, accepted result, confirmed sandbox stop-and-retain, source still bound to the workspace). It marks the hold `supersededWorkspaceExport`. - `native-run-projection-work.ts`: the native run projection worker also selects these runs. It loads them at startup and wakes when a finalizer changes. Rows written by older releases are therefore settled after deploy without a person. - `workspace-recovery-fallback-discovery.ts` and `workspace-recovery-fallback.ts`: a marked hold whose bounded re-export is spent (at most 5 retries in 30 minutes) is eligible for the existing continue-from-last-save fallback. That fallback sets aside only this run's workspace epoch and keeps its location on the workspace record. It attaches the readable files to the task unapplied. The task message says Paperclip "could not save" the previous run's unsaved files. - `preserved-result-finalization.ts`: a superseded result is never resumed for application, even after its files are saved. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/services/superseded-workspace-export-recovery.test.ts` (new). The first two tests fail on master and pass on this branch: - "records the files for automatic recovery, saves them from the retained sandbox and frees the workspace". It uses the staging row shape. The projection worker records the hold, and a second pass changes nothing. The workspace waits for that save. The system recovery saves `saved.txt` into a committed snapshot. The workspace becomes free. The finalizer, the result and the task's decision do not change, and the result is not resumed. On master the worker selects nothing (`expected 0 to be 1`). - "continues from the last save with the files attached when every re-export fails". After an exhausted re-export budget, the hold is resolved `restored`. The files are attached (`quarantine.state = captured`). The workspace moves to epoch 2 with no writer, and the abandoned epoch keeps its lease location. The message has no request for a person to fix anything. - "leaves a superseded result alone when it never failed in the workspace export". - Related suites pass: `native-run-projection-work`, `native-finalization-recovery`, `workspace-recovery-fallback`, `native-failed-workspace-recovery`, `workspace-recovery-retry-copyback`. - `tsc --noEmit` for `server`. ## Risks - Low to medium risk. The change acts only on runs whose finalizer is already terminal and superseded, that failed in the workspace export, and that still own a dirty workspace. All existing preservation proofs still apply. - The saved files of a superseded run become the workspace's next saved version. The run's result (the task decision) is still never applied. - When the export can never succeed, the task continues from its last save. Files that cannot be read are not attached. The workspace record keeps the set-aside epoch's location. Saving the readable part of a partly unreadable workspace is not in scope. - Runs that already failed setup before this deploys are not restarted by this change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d2e4d8d9 |
fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result, then a finalizer exports the workspace and applies the result to the task. > - When a newer run already committed the task's decision, the finalizer retires the older result as `native_finalization_superseded`. > - That retirement set the finalizer terminal, but it did not end the run. The run stayed `running` and held the task and every run queued behind it, with no way out except a person. > - A stuck run must never hold a task and its queue forever. > - This pull request ends the run when its finalizer is retired as superseded, and it ends runs that an older release left in that state. > - The benefit is that the task continues with its queued runs by itself. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. The workspace export then failed (`remote_workspace_archive_fs_eacces`). The task's last decision came from another run that was created 42 seconds later but ran earlier. The finalizer judged this run superseded: it set the finalizer to `terminal_failure` with `native_finalization_superseded`. The run stayed `running` with "Recovery needed". Over one hour later it still held the task, and five queued runs waited behind it. A server upgrade at 12:02Z did not change it. **Expected behavior** A run whose finalizer is retired ends. The task keeps the newer run's decision, the retired result is never applied, and the queued runs proceed. **Steps to reproduce** 1. Accept a native result for a run. 2. Commit a status decision on the same task from a run that was created later. 3. Let the older run's finalization fail once, so the finalizer records the failure. 4. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is still `running`. **Root cause** `recordRetryableFailure` (`server/src/services/native-runtime/native-run-finalizer.ts`) projects a terminal run status only when `exhausted && !supersededByNewerRun`. A superseded retirement therefore never ended the run. No other path ends a run whose finalizer is already `terminal_failure`: the reconciler skips `terminal_failure` rows. **Paperclip version or commit** Staging Core `3eca7a6112`. The behavior is the same on master `482063aeca`. ## What Changed - `native-run-finalizer.ts`: a superseded retirement now ends the run with the same status mapping as an exhausted finalization (workspace failure: `failed`; otherwise the provider's terminal state). It does not change the task, does not apply the result, and does not keep the workspace export lease. The run's error says in plain words that a newer run already decided the task. - `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`. It ends a native run that is still `running` while its finalizer is `terminal_failure` / `native_finalization_superseded`. This covers rows that an older release wrote. It locks in the finalizer's order (coordinator, task, run), and re-checks the row under the lock. - `native-run-projection-work.ts`: the registered native run projection worker (it loads at startup, pages through all candidates, and wakes on finalization changes) also selects such runs and ends them with `endSupersededNativeFinalizationRuns`. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/__tests__/native-finalization-recovery.test.ts`, red on master and green on this branch: - "retires an older failed finalizer when a newer run already committed the issue": the retired run now ends (`succeeded`, `finishedAt` set). Master: the run stays `running`. - "ends a run an earlier release left running after retiring its finalizer as superseded": the shape of the staging row (workspace export failed, superseded finalizer, run `running`). The repair ends it as `failed` with the plain message. The task, its decision and the finalizer stay unchanged, and no status decision is written for the run. A second pass changes nothing. - `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at startup, a run an earlier release left running after retiring its finalizer as superseded": the registered worker ends the staging-shaped row at startup and settles its claimed wake. Red without the worker change. - Related suites pass: `native-finalization-recovery`, `native-run-projection-work`, `native-finalization-reconciler-timing`, `native-reconciliation-timing`, `native-workspace-finalizer`, `native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`. - `tsc --noEmit` for `server`. ## Risks - Low risk. The change only ends runs whose finalizer is already terminal and superseded. It never applies a result and never changes the task. - A superseded run whose workspace export failed now releases its sandbox lease through the normal settlement instead of waiting forever. Its result was not applied, because a newer run decided the task. - Not changed here: the superseded check compares run creation times. A run created earlier but executed later is still judged superseded. This PR makes that verdict end the run; it does not change the verdict. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
50fe32684d |
fix(server): wake committed native run projection from durable changes (#17752)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run has separate result acceptance, task decision, run status, and resource settlement steps. > - A task decision can commit before its run status is saved. > - Recovery currently revisits committed decisions through the periodic orphan pass. > - This PR gives the existing run projection repair its own transaction-driven worker. > - Existing records reconstruct unfinished work after restart. Completed history becomes quiet. > - Other completion and resource obligations keep their existing recovery paths. ## Linked Issues or Issue Description Refs #17748 (stack parent: accepted-result commit), #17734 (workspace finalization), and #17717 (retained session cleanup). Related #17614 preserves terminal notifications. This slice is prepared on current parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. **Current behavior** The native recovery pass revisits committed decisions to repair the heartbeat-run status and metadata. A crash between decision and run writes creates real work, but periodic scanning also revisits retained history. **Proposed behavior** Wake projection repair after committed producer changes. Reconstruct mismatches on startup and settings activation. Retry concrete failures through the shared scheduler. Keep the run projection before resource settlement. **Reason and benefit** The repair worker performs no recurring SQL after its work settles. A real decision, effect settlement, or ownership release reactivates it. This is one worker migration; it does not establish host or database suspension. ## What Changed - Extract the existing projection predicate and write into their native-runtime owner. The finalizer keeps initial metadata enrichment and task decision authority. - Discover accepted, committed, mismatched run projections with company, issue and run bindings. Exclude pending status effects and held runner ownership. Read narrow fields instead of whole result payloads. - Use existing transaction hints, scoped delivery scheduling, failure backoff and lifecycle handling. Add no tables or per-feature timers. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant projection quiet; explicit settings and dedicated producer signals keep their existing contract. - Connect finalization, pending effects, lifecycle status writes, issue changes and worktree execution activation. Preserve outer transaction commit and rollback behavior. - Lock the current run before deciding which terminal event and completion time to save. Concurrent live and recovery projection must not announce the same transition twice. - Delegate only already-committed projection from the periodic pass. Its candidate scan uses the same pending predicate to defer remaining consequences and cleanup until projection finishes. - Add real PostgreSQL producer, race and lifecycle tests plus an owner audit. ## Verification - Current local source `a83b9449233a275176e637c765059f414305d7c1` incorporates the parent with a normal merge. All 1,208 parent-only, seven owned-only and ten automatic shared paths are preserved. The only manual conflict keeps new projection fanout inside the ordinary issue/run guard. The complete package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: the original projection-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes with scoped fanout. A real committed result then repairs the run and emits one terminal event before returning to quiet. Production bytes restore exactly in both phases. Twenty cases outside that focused pattern are not counted as passing there. - The complete main command passes **1,830 cases in 19 files**, without skips, in 301.14 seconds. It includes all 21 projection cases, 85 startup cases, required quality review and heartbeat run lifecycle, plus physical-work and result-commit consumers. Separate process recovery passes 410 in 163.58 seconds and external-chat response passes 273 in 53.26 seconds. Total: **2,513 distinct passes in 21 complete files**, with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5` adopts the actual parent squash with this entire tested child tree unchanged. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Seventeen dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only calls use the existing fresh-per-call recovery helper. All temporary imports restore byte-exact and all 21 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One pre-existing opt-in closed Pi snapshot case is skipped because its distribution root is not configured. No live provider qualification is claimed. - The owner audit at `doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains prior failures and their disposition. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable findings. Old scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, deployed suspension, expanded Cloud wake coverage or measured cost savings. ## Risks - Process-local signals cover instrumented application writers. Independent SQL writers and replicas without supported ingress are not newly covered. Durable startup reconstruction remains enabled. - Projection must precede remaining committed resource settlement. The real reconciler test proves the periodic path defers cleanup while projection is pending and resumes it after repair. - Status decisions, external conversation settlement, watchdog resolution, Muse reconciliation and resource release keep their own authority and recovery. A committed result alone is not proof of completed cleanup. - Candidate discovery uses narrow bounded pages. There is no large-fleet performance or deployed sleep claim. Existing Cloud wake coverage and sleep blockers remain unchanged. - Parent #17748 has merged. This PR targets master at `28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |