From 0b0e53029fa572429b892d13fcb72b22decb0d38 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:14:45 -0500 Subject: [PATCH 1/2] fix: recover saved workspace results automatically (#15824) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path - Paperclip preserves accepted agent results before workspace copyback and result commitment. - A replaced controller can leave a physical copyback owner receipt behind. A lost database lock or expired lease cannot prove that its tar/Git children stopped. - Infrastructure recovery should belong to the runtime and deployment harness; task users should not investigate processes or attest shutdown. - New receipts record the exact deployment. The harness independently verifies that deployment fully stopped and signs a short-lived proof bound to the stack, run, owner and deployment. - The existing guarded reconciler then finishes the same accepted result without a new agent turn. Legacy receipts retain the documented administrator API after independent platform verification. ## Linked Issues or Issue Description **What happened?** An accepted native result can remain at `workspace_finalizing` with an abandoned physical copyback owner. Repeated recovery sweeps could replace specific instructions with advice to start new agent work. The first version of this PR exposed a shutdown-evidence form in the task; that asks end users to resolve infrastructure details. **Expected behavior** Automatically verify and recover the exact stopped deployment, preserve the original accepted result and existing approval/monitor waits, and show only a quiet task status. Unknown or live workers must remain fenced. Older receipts without platform identity need independently verified operator assistance through the administrator API. **Steps to reproduce** 1. Persist an accepted native result and a copyback receipt from a prior controller. 2. Replace that deployment before workspace finalization commits. 3. Let the normal finalization reconciler revisit the saved result. 4. With a platform-bound receipt and a fully stopped deployment, recovery should finish automatically. With missing, live, unverifiable or changed ownership, it must not write files or create another provider turn. Related public work: #14402 introduced copyback ownership; #9687 handles dead-run finalize barriers; #15795 handles terminal stale-run cleanup. This change addresses an accepted result whose physical copyback owner prevents commitment. ## What Changed - Record Railway project, environment, service and deployment identity on new physical copyback owners. - Request read-only platform verification from the managed cloud harness. Require an Ed25519 stop receipt with a disjoint purpose, configured signing key, exact stack/run/owner/deployment binding and valid expiry. - Keep live, unknown, malformed, unavailable and legacy owners fenced. Compare the whole prior receipt during ownership takeover, under the existing dedicated advisory lock. - Persist platform proof and audit automatic takeover. Resume normal finalization of the same saved result, preserving task ownership, approved boundaries, monitors and pending confirmations. - Remove process metadata, shutdown evidence, confirmation checkboxes and infrastructure-repair forms from task views. Suppress the duplicate technical owner-recovery card. Show “Restoring saved work…” only from server-confirmed recovery or queued operator recovery; platform identity alone shows the operator-assistance fallback. Remove the status when recovery resolves. - Share current-task, current-running-execution, accepted-result, authoritative-reviewer and newer-execution checks between automatic and operator recovery. Validate after the provider lookup, atomically with the owner claim, and again before file publication; closed or superseded tasks cannot start copyback. - Keep the separately authorized exact-owner API for operator recovery of legacy receipts, with active-lock, live-controller, stale-result and newer-execution rejection. Generic dismissal cannot bypass the copyback fence. - Replace obsolete form stories with production-backed automatic, queued, legacy, completed and mobile examples; retain stable suite IDs. Update the operator runbook and scoped Storybook catalog. ## Verification - Red/green: new platform-proof tests failed before implementation; the database takeover regression failed against the original ownership guard; reduced-UI assertions failed against the original form. All pass after the changes. - Passed: 59 physical-owner/finalization database tests using isolated PostgreSQL, including real reconciler commitment from all three saved-result phases through both automatic and operator recovery. They preserve result identity, exact saved file bytes, pending approval and overdue monitor waits, and create no new run or wake request. Unknown evidence and changed ownership remain fenced. - Passed: 162 focused proof, cloud identity, recovery route, recovery-card and reduced-UI tests. - Passed: full workspace typecheck and build, final server/UI typechecks, token gates, and Storybook build. - Browser inspection: desktop task composition shows only the quiet status; completed example removes it and retains the approval-wait message. Mobile shows the existing bottom navigation with no horizontal overflow. These are fixture UI observations, not live provider acceptance. - A companion harness change implements the actual provider check, signed receipts, server-token authentication and bounded/coalesced provider reads. Its full typecheck and test suite passed: 3,240 passed, 92 skipped, zero failures. This application alone cannot enable platform verification. - The updated full application `pnpm test:run` was stopped after repeated local embedded-Postgres startup failures; it is not claimed green. The same affected recovery suites pass against isolated PostgreSQL. CI passed on final head `9ad5c3529e`: 54 successful checks, two intentional skips, and successful legacy Snyk status. Greptile is 5/5 with no unresolved findings. - Global Storybook catalog validation is green: 225 files, 2,224 stories. Its 17 catalog/index tests pass. Master was merged and the inherited index metadata was reconciled after source review. - Review correction red/green: two UI and two database assertions reproduced misleading progress / missing server recovery state; all pass after server-confirmed state was added. All 153 focused server tests and 68 UI tests pass on the final recovery code, including repeated-sweep idempotence. Full workspace typecheck/build, token gates and Storybook build also pass. - Safety regression red/green: stale-task copyback and publication tests failed against the previous guard. The final suite covers changes during proof lookup, closed/cancelled tasks, replaced execution/checkout, newer and queued legacy runs, reassignment and rejected/unbound results, plus the real reconciler selecting closed or superseded tasks. Files remain untouched and no workspace operation starts. - No staging mutations, deployment, live provider probes or live production recovery were performed. The three historical receipts lack deployment identity and still require independent operator proof. ## Risks Automatic takeover requires the companion harness release and configured runtime-identity keys. Deploying only this application leaves unavailable evidence fenced. Legacy receipts are not retrospectively assigned a deployment identity. An authorized operator can still make an incorrect attestation through the fallback API; its administrator gate and exact-owner safeguards remain. Provider history that is missing or deeper than the bounded lookup defers automatic recovery. Copyback can still fail under its existing retained-workspace policy. No migration, telemetry change or cloud provider credential delivery is required. The companion harness PR also has passing CI and a 5/5 review; neither branch is deployed. ## Model Used OpenAI Codex, GPT-6, with repository inspection, browser verification, tool use and code execution. The exact serving model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Paperclip Co-authored-by: Devin Foley --- doc/native-workspace-finalization-recovery.md | 86 ++++- maps/agent-access-map/recovery.md | 29 +- maps/feature-map/recovery.md | 28 +- maps/storybook-map/INDEX.md | 10 +- maps/storybook-map/REVIEW.md | 78 ++++ maps/storybook-map/catalog.json | 121 ++++-- packages/shared/src/index.ts | 1 + packages/shared/src/validators/index.ts | 1 + packages/shared/src/validators/issue.ts | 8 + .../__tests__/authorization-service.test.ts | 35 ++ ...ue-agent-mutation-ownership-routes.test.ts | 70 +++- server/src/routes/issues.ts | 17 + server/src/routes/openapi.ts | 10 + server/src/services/authorization.ts | 16 + server/src/services/cloud-runtime-identity.ts | 6 +- server/src/services/issue-recovery-actions.ts | 17 +- ...native-workspace-finalization-ownership.ts | 99 ++++- .../native-workspace-finalizer.test.ts | 361 ++++++++++++++++++ .../native-workspace-owner-recovery.ts | 87 +++++ .../native-workspace-platform-proof.test.ts | 34 ++ .../native-workspace-platform-proof.ts | 65 ++++ .../native-workspace-recovery-binding.ts | 36 ++ ui/src/api/issues.ts | 2 + .../IssueRecoveryActionCard.test.tsx | 7 + ui/src/components/IssueRecoveryActionCard.tsx | 1 + .../WorkspaceExportRecovery.test.tsx | 88 ++--- ui/src/components/WorkspaceExportRecovery.tsx | 50 +-- ui/src/pages/IssueDetail.tsx | 8 +- .../01-notice.stories.tsx | 29 ++ .../02-task-journey.stories.tsx | 43 +++ .../fixtures.tsx | 37 ++ 31 files changed, 1306 insertions(+), 174 deletions(-) create mode 100644 server/src/services/native-runtime/native-workspace-owner-recovery.ts create mode 100644 server/src/services/native-runtime/native-workspace-platform-proof.test.ts create mode 100644 server/src/services/native-runtime/native-workspace-platform-proof.ts create mode 100644 server/src/services/native-runtime/native-workspace-recovery-binding.ts create mode 100644 ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx create mode 100644 ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx create mode 100644 ui/storybook/support/workspace-finalization-recovery/fixtures.tsx diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 01201315417..a9a6e6365f1 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -35,6 +35,46 @@ leave tar/Git children alive, so incomplete copyback requires operator stop verification even on the same host. PID reuse is checked against its recorded start time rather than trusted by PID alone. +## Automatic cloud recovery + +New Railway copyback receipts record the exact project, environment, service and +deployment ID. On takeover, the existing finalization reconciler asks the cloud +harness to inspect that deployment. The harness resolves the stack's owned +resources independently and issues a short-lived Ed25519 receipt only when the +exact deployment is `REMOVED` with `deploymentStopped: true`. This covers all +processes in the deployment, including orphaned copyback children. Missing, +truncated, live or unavailable provider evidence leaves the physical fence intact. + +The tenant verifies the configured harness key, disjoint receipt audience, stack, +run, owner token, deployment identity and expiry. It records proof and an activity +entry when replacing only the exact owner under the existing advisory lock. The +normal reconciler resumes the accepted result; it does not create a provider turn +or change approval/monitor waits. A changed owner during verification fails closed. + +Stop proof alone does not authorize old work to overwrite a current task. After +the provider read, automatic recovery uses the operator path's shared binding +checks under short coordinator, issue and run row locks, atomically with the +owner claim. The task must still be open, assigned to the run or its authoritative +review participant, and bound to that running native execution and accepted +result. A different checkout, newer execution or queued/running legacy execution +prevents takeover. Publication checks repeat this validation; row locks are never +held across physical copyback I/O. + +The harness endpoint is `POST /v1/tenant/workspace-worker-stop`. The fleet-wide +server token authorizes a read request, never stop evidence: responses must be +signed by the harness. This endpoint requires the companion cloud harness change. +Deploying only the application keeps unknown owners fenced. No cloud provider +credentials are delivered to tenant applications. + +The task view shows only “Restoring saved work…” after the server confirms recovery +or an operator queues it, +then removes the notice when commitment resolves recovery. It exposes no process +identifiers, shutdown form or manual resume control. Legacy receipts without +platform identity and unsupported platforms say operator assistance is needed; +operators use the documented exact-owner API below after independent stop proof. +The three diagnosed historical receipts do not gain a deployment identity merely +by upgrading. Never infer one from elapsed time or a hostname. + ## Unverified copyback after controller replacement Generic orphan recovery does not turn a missing provider PID into `process_lost` @@ -66,6 +106,48 @@ exact prior controller **and its copyback subprocesses** have stopped. Retain th sandbox, accepted native result, and workspace descriptor. Do not run a new provider turn, delete workspace contents, or relax archive confinement. +For legacy receipts, an authorized company owner, company administrator or +instance administrator records independently verified platform evidence through +the operator API. The task has no shutdown form or resume control. The API is +`POST /api/issues/:id/recovery-actions/resume-workspace-finalization` with: + +```json +{ + "actionId": "", + "runId": "", + "ownerToken": "", + "controllerAndCopybackStopped": true, + "stopEvidence": "" +} +``` + +This is an operator attestation, not an automatic deployment-platform probe. +The separate `runtime:confirm_owner_stop` authorization action limits this +attestation to active company owners or administrators, the trusted local board, +or an instance administrator. Ordinary runtime managers and agents cannot release +the physical owner receipt. Missing role data keeps the client notice read-only. +A +missing advisory connection alone is insufficient. Admission rejects an active +copyback lock, an observably live local controller, a changed owner, a newer run, +or an unbound result. It records the exact owner and operator evidence atomically +with releasing the receipt. The accepted result, workspace descriptor, source +lease, task status, monitors, and approval waits are unchanged. The ordinary +reconciler performs copyback and applies the saved result through the status +arbiter. No provider wake is enqueued. The recovery action remains visible until +finalization commits. Admission supports saved-result copyback from `result_accepted`, +`workspace_finalizing`, and `retryable_failure`; an abandoned retry does not need +a replacement run. The existing retry schedule remains unchanged. +Repeating the same confirmation before reconciliation is +idempotent; it cannot release a newer owner. Generic recovery resolution, including +false-positive dispositions, cannot dismiss this hold or bypass saved-result +finalization. + +Repeated sweeps preserve these copyback-specific instructions, including for +older actions whose recovery budget was already exhausted. They must never +recommend starting a replacement provider run for this hold. + +For older servers without this action, the maintenance fallback follows. + After that platform verification, use a database maintenance transaction to release only the exact receipt shown by the recovery action. Replace the four placeholders with the action's company, run, token, and source issue. The advisory @@ -138,8 +220,8 @@ A board member with runtime management access can complete the saved result: the export failure while preserving all user files. Restore transport access or other provider prerequisites before retrying. Extending the sandbox retention window can be necessary during this repair. -3. In the task's **Workspace export needs repair** notice, describe the repair and - choose **Retry workspace export**. The equivalent board API is +3. Use the operator API to record the repair and request export-only retry. The + task shows operator assistance is needed, without an infrastructure form. The board API is `POST /api/issues/:id/recovery-actions/retry-workspace-export` with `{ "actionId": "", "runId": "", "repairNote": "" }`. 4. Confirm the same run commits, the saved result determines the task outcome, diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index 1b90fa083d4..c7cc042f969 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -129,23 +129,40 @@ Review: `baseline`. API: `present` — representative operation IDs: - `POST /api/issues/{id}/recovery-actions/retry-workspace-export`. +- `POST /api/issues/{id}/recovery-actions/resume-workspace-finalization`. - `POST /api/execution-workspaces/{id}/reconcile-branch`. Agent access: `restricted` — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work. -Policy: Task/run visibility, run ownership and source/action policy; workspace export -retry and some operator controls require board. +Policy: Export retry and branch reconciliation require board runtime management. +Physical-owner stop confirmation uses the separate `runtime:confirm_owner_stop` +action. It requires active company owner/admin membership, trusted local board +access, or instance administration. Agents and ordinary runtime managers are denied. +The exact source task, running native run, accepted result and owner receipt must +match; active copyback locks, live local writers, and newer runs prevent admission. +An authorized administrator must first verify through the deployment platform that +the exact controller and all copyback children stopped, then attest that evidence. +Elapsed time, an expired coordinator lease or a lost advisory connection is not proof. +Only the exact owner receipt is released; normal reconciliation performs export and +commitment without another provider turn or bypassing existing task waits. -Evidence: `source-only` — [implementation](../../server/src/routes/issues.ts#L9478) at -the reviewed commit above. +Evidence: `source-only` — [route](../../server/src/routes/issues.ts), +[administrator authorization](../../server/src/services/authorization.ts), and +[owner admission](../../server/src/services/native-runtime/native-workspace-owner-recovery.ts) +reviewed for this change. Local authorization, route, UI and finalization regressions +cover member/agent denial, exact-owner admission and the real reconciler preserving +monitor and approval waits. This is not live provider-stop proof. -Verification: `not-run` — assert the documented agent denial with an owned test fixture; +Verification: `not-run` for live agent/deployment execution — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review. Gaps: Export retry and branch reconciliation require board runtime:manage at -their HTTP routes. Native `call_api` preserves these denials. +their HTTP routes. The exact-owner operator fallback additionally requires +runtime:confirm_owner_stop. Native call_api preserves these agent denials. +Automatic managed-cloud recovery uses independently verified, signed platform +proof through the finalization reconciler; agents cannot attest owner shutdown. ## `unknown-outcome` diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 3cd71abaffc..7d5a5945d6d 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -119,7 +119,22 @@ Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNoti tests pending/error/gated states and waiting for the real retry acknowledgement. The recovery-card suite covers workspace divergence, reissue, reconcile, and confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx) -covers its separate repair surface. +covers quiet restoration, legacy/operator assistance and resolved states. + +For an abandoned native copyback owner, the harness verifies the exact stopped +deployment and the reconciler restores saved work automatically. Task users see +no process details or shutdown form. The documented operator API handles older +receipts after independent platform verification, through a separate administrator +gate. The [authorization regressions](../../server/src/__tests__/authorization-service.test.ts) +and [route regressions](../../server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts) +cover administrator admission, ordinary-member and agent denial, suspended and +foreign membership, missing role data, and stale cloud admin rows. +The [native workspace finalizer suite](../../server/src/services/native-runtime/native-workspace-finalizer.test.ts) +runs the real reconciler with a valid saved yield from `result_accepted`, +`workspace_finalizing`, and `retryable_failure`. It verifies physical fencing +before verified stop, automatic and operator recovery, successful export and commitment, hold resolution, +unchanged saved results, retained approval/monitor waits, and no new provider wake. +These local regressions do not prove a deployment platform stopped a foreign writer. Manual: reproduce an exhausted disposable action. Click once, verify pending state prevents repeated clicks, and inspect a successful continuation or an @@ -128,6 +143,17 @@ repair, record the compared branches/revisions and preserve uncommitted work; confirm the repair's actual result before resuming. Check the denied and stale action paths. A changed card label is insufficient proof of workspace integrity. +For an abandoned native controller, follow the [stop-verification procedure](../../doc/native-workspace-finalization-recovery.md) +with a disposable deployment. A new platform-bound owner recovers automatically +only after the harness signs proof that the exact deployment fully stopped. +Legacy owners require independently verified stop evidence through the operator +API; the task has no shutdown form. Confirm +the original accepted result commits, the recovery hold resolves, saved files +remain intact, and existing approval waits or monitor checks remain. No new agent +turn should start. Repeat with a retry-phase owner and an ordinary runtime manager; +the first can resume through automatic proof or authorized operator confirmation, +while the second cannot use the operator API. + ### `post-stop-message` Automated: [legacy continuation](../../server/src/services/recovery/legacy-continuation.test.ts), diff --git a/maps/storybook-map/INDEX.md b/maps/storybook-map/INDEX.md index ec3a4c08b5a..cc36bca0cc8 100644 --- a/maps/storybook-map/INDEX.md +++ b/maps/storybook-map/INDEX.md @@ -4,7 +4,7 @@ [Contract and maintenance](README.md) · [Review findings](REVIEW.md) -223 story files; 2215 indexed stories. Counts describe inventory, not passed tests or complete feature coverage. +225 story files; 2224 indexed stories. Counts describe inventory, not passed tests or complete feature coverage. Status describes product alignment; implementation describes the rendered subject; boundary describes simulated data. Sources and limitations are in catalog.json. Related capabilities cross-list one canonical story file. @@ -335,10 +335,12 @@ Stopped work, workspace repair, bounded continuation, and reconnect. | [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | current / production / fixtures | 9 / 3 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | -| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | +| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite. | | [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 8 / 4 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 2 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Master reconciliation inspected assignment-history recipient suggestions and manual-selection retention; existing stories do not establish the new assignment-history behavior. Expanded upstream verification recipes were inspected; the existing fixture scenarios do not prove the added backend handoff, workspace, recovery or authorization contracts. | +| [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) | `recovery/workspace`, `recovery/source-action` | current / production / fixtures | 6 / 2 | Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests. | +| [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) | `recovery/workspace`, `recovery/source-action` | unreviewed / mixed / fixtures | 3 / 0 | Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance. | ## Apps, channels, and extensions @@ -667,9 +669,9 @@ References are navigation links, not proof of parity. An absent link means no ca - `execution-environments/images`: No cataloged story. - `recovery/diagnosis`: [Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards](../../ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx) (unreviewed), [Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab](../../ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice](../../ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page](../../ui/storybook/stories/secrets/credential-access/02-task-page.stories.tsx) (unreviewed), [Projects and execution/Recovery/Task chat/Recovery notice](../../ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Tasks/Execution recovery](../../ui/storybook/stories/recovery/execution-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue/Blocked notice](../../ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff](../../ui/storybook/stories/tasks/successful-run-handoff.stories.tsx) (unreviewed) - `recovery/bounded-continuation`: [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) (unreviewed) -- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed) +- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed) - `recovery/exhaustion`: No cataloged story. -- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed) +- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed) - `recovery/unknown-outcome`: No cataloged story. - `recovery/user-continuation`: No cataloged story. - `recovery/reconnect`: [Projects and execution/Recovery/App/Connection recovery](../../ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx) (unreviewed) diff --git a/maps/storybook-map/REVIEW.md b/maps/storybook-map/REVIEW.md index b1ab397fee0..e3575ea0ee7 100644 --- a/maps/storybook-map/REVIEW.md +++ b/maps/storybook-map/REVIEW.md @@ -274,6 +274,35 @@ Reviewed the same twelve workspace-recipe-dependent entries after extending the Reviewed the twelve unchanged workspace recipe subjects and fixtures after correcting implicit default selection. Task creation and controller classification now share configured-policy, primary, creation-time and ID precedence. Explicit choices and retained denial markers still suppress controller hooks. Project strategy fixtures show configuration, while runtime status and file/browser/artifact/service fixtures do not execute source scripts or prove this backend ordering. Refreshed only their reviewed recipe fingerprints and retained all classifications and stable IDs. No browser render, play, screenshot, Storybook build or live runtime acceptance was performed. +## Workspace finalization UI refinement — 2026-10-13 + +Source base: `f07efa2802`, followed by scoped edits to the production recovery notice and its two story suites. Reviewed `recovery/workspace` and `recovery/source-action`; stable story IDs and administrator admission remain unchanged. The notice leads with saved work, discloses controller metadata, explains evidence/confirmation requirements, and distinguishes queue acknowledgement from completion. A stale controller error preserves the note and clears confirmation so the operator must verify again. The task composition uses viewer identity for human bubbles and omits a routine status badge. + +MacOS in-app browser inspection passed for the task journey at desktop and 390px mobile widths: controller disclosure with Enter, evidence without confirmation remaining disabled, explicit confirmation enabling submission, and a simulated queued result in place. The `owner-changed` story's play assertions passed; the friendly error was visible, the note remained, confirmation cleared and the action was disabled. The `long-controller-name` story rendered at 390px without horizontal overflow. The mobile task shell showed its bottom navigation; evidence text computed to 16px and Tab reached the enabled action. No physical mobile keyboard, Linux screenshot baseline, live deployment stop, real API admission or provider copyback acceptance is claimed. Other story states received source review, not a full browser execution pass. + +Refreshed the two reviewed catalog entries only. Global catalog validation already reported unrelated source-fingerprint drift before this work; those entries require their own review. + +## Automatic workspace recovery revision — 2026-10-13 + +Supersedes the task shutdown form reviewed above. The production task now renders +only a quiet restoration status or an honest operator-assistance fallback. It +contains no process metadata, evidence field, confirmation checkbox or resume +button. The task thread omits the duplicate technical recovery card. Obsolete +form stories and their simulated API fixture were removed; suite IDs remain +stable. The composed task journey includes a saved state without the notice. + +Observed in the macOS in-app browser: automatic status in the production shell, +completed example without the status, retained approval-wait message, and mobile +bottom navigation with equal content and viewport widths (no horizontal overflow). +These are fixture UI observations, not live provider shutdown or task commitment. +The production component test verifies removal on a resolved action update. The +real reconciler's database tests separately preserve accepted-result identity, +monitor and approval waits, and create no new run or wake request. + +Only the two reviewed suites have refreshed fingerprints. Global validation has +71 other source-fingerprint errors, including recovery-recipe dependents after +the operator instructions changed. It is not globally green. No unrelated review +fingerprints or visual baselines were accepted. ## Inherited activity, inbox and recovery recipes — 2026-10-12 @@ -435,3 +464,52 @@ and three also reference the Slack-only settings copy. Other source bytes and all classifications, fixture boundaries, story IDs and sidebar titles remain unchanged from that baseline. Catalog checks validate these references and fingerprints, not live Slack behavior. + +### Current-master reconciliation + +Merged master `5e8e992df2`. The newer catalog supplies the baseline associations +and reviews. Compared every declared input against that master: only seven +inputs changed for this feature (recovery recipe, recovery card, task page, quiet +status, two stories and their fixture). Reviewed those changes and the unchanged +subjects/fixtures they affect; refreshed the 26 dependent fingerprints while +retaining their classifications and simulation boundaries. The task-page change +adds the quiet status and suppresses only the owner-unverified duplicate card; +other task variants retain their behavior. The recovery-card change blocks +unsafe generic dismissal. Existing card regressions cover that denial. + +Five remaining errors have unchanged source bytes and metadata inherited from +master: shared setup, decision-card, decisions-desk, search and what-needs-me. +These require a separate review; no hashes or visual baselines were accepted for +them. The earlier 71-error count above describes the pre-merge baseline. + +### Inherited index metadata follow-up + +The current master's personal-toast recipe also left shared setup and four inbox +suites with stale metadata. Reviewed the recipe's added toast capability and +verification limits: decision-card, decisions-desk and search still demonstrate +their existing isolated subjects; what-needs-me's toast fixture does not prove +viewer eligibility. Their subject and fixture bytes are unchanged from master; +retained all unreviewed classifications and fixture boundaries. The generator +adds the missing toast capability and description to the existing inbox feature, +without changing feature order or IDs. Reviewed that generated diff and refreshed +these four entries, the shared setup and Overview fingerprints. No toast behavior, +new browser acceptance or visual baseline is claimed. Global catalog validation +now passes. This supersedes the five inherited-error count above. + +### Confirmed recovery status + +Reviewed the shipped notice, server evidence contract and both recovery fixtures. +Platform identity alone no longer means work is restoring: the server marks the +exact recovery receipt resuming only after verified stop proof and a successful +owner claim. Missing proof remains operator-required. The new unavailable-proof +story covers a platform-bound receipt without progress; confirmed and queued +stories retain the quiet status. Updated only the two affected fingerprints. +This is component/fixture evidence, not deployed harness acceptance. + +### Latest master reconciliation + +Retained master’s task-creation visibility scenarios, source-review notes and +unreviewed classifications. Seven overlapping entries differ from master only +in the already reviewed recovery recipe or task-detail recovery notice wiring. +Refreshed those combined-source fingerprints and regenerated the index. No new +creation-visibility acceptance or visual baseline is claimed. diff --git a/maps/storybook-map/catalog.json b/maps/storybook-map/catalog.json index 1c81c9a3adf..005c7a6f2bf 100644 --- a/maps/storybook-map/catalog.json +++ b/maps/storybook-map/catalog.json @@ -304,7 +304,7 @@ "ui/storybook/prototypes/chat-entry/ChatEntryReview.tsx", "ui/storybook/stories/agent-chat/agent-chat-sidebar.stories.tsx" ], - "fingerprint": "6994574cfda7901b4cec355add8dadb58146bd4cf0d3488796fe27e19e34cdc3", + "fingerprint": "0fc8fe3663215c79493116ec27e13daf92b8893806ee4e19e7e4143e97e4fe5f", "stableId": "design-explorations-agent-chat-sidebar-pages", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat sidebar/Pages" }, @@ -364,7 +364,7 @@ "ui/storybook/prototypes/chat-entry/ChatEntryReview.tsx", "ui/storybook/stories/agent-chat/agent-chat.stories.tsx" ], - "fingerprint": "c3faad0b551a84f6c7ddf5cf9566b96e0d435d28a4ac3035d147b069723b7152", + "fingerprint": "96b384fffda6225b3f00714d1fa33dc208806d9a502e2dcc46fba3c667010d18", "stableId": "design-explorations-agent-chat", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat" }, @@ -513,7 +513,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/agents/agent-persona-pages.stories.tsx" ], - "fingerprint": "35454c02bbae403677d0d79644dc92da6c96172265d0f7581508e7d111463e0f", + "fingerprint": "672f406658a3ade96fed4f5a18d2a413ec9888e348b74244b5d37cb3db760dde", "stableId": "agents-personas-full-pages", "sidebarTitle": "Agents and reusable capabilities/Hiring, configuration, and organization/Agents/Personas/Full pages" }, @@ -1196,7 +1196,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx" ], - "fingerprint": "cdb63642517b31c68df9ca8d62e60039f9f4d69d95d1a96dc96235fb7db1487d", + "fingerprint": "d6e735a2c56092274f139709aba0c1c555f3f3cb0fbf9820d50ee02874907b79", "stableId": "paperclip-assigned-backlog-safeguards", "sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards" }, @@ -1220,7 +1220,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx" ], - "fingerprint": "bcb6c0a3cb34fcd06e7b056521ad84fb968f73ba46a7ba00eda2a86bb060b11b", + "fingerprint": "70b54f5ce7c083b8e64c28501d66247d4753adabd732b377f7ad18309dc90ffe", "stableId": "product-inbox-blocked-tab", "sidebarTitle": "Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab" }, @@ -1456,7 +1456,7 @@ "ui/storybook/prototypes/chat-entry/ChatEntryReview.tsx", "ui/storybook/stories/agent-chat/chat-entry.stories.tsx" ], - "fingerprint": "444465b306686cc36d85d4d2af43f9cd31f3c7c34a5fd05f8413aa639becf535", + "fingerprint": "3e85e934a82dae8089b73a1bddb0538640259ce1c70e7d16da38ee1e91ab0efc", "replacement": "ui/storybook/stories/agents/primary-agent/05-chat.stories.tsx", "stableId": "design-explorations-chat-entry", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Chat entry" @@ -1499,13 +1499,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/CloudAccessGate.tsx", "ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx" ], - "fingerprint": "bbec224b127ec6898cf95dfae922e6747483c0d0d4ec55ff7d1560c0c4a5c041", + "fingerprint": "64b2422bdabe2b07b563354481000171455841e49ccbf6a72270066d4e27e9e0", "stableId": "app-connection-recovery", "sidebarTitle": "Projects and execution/Recovery/App/Connection recovery" }, @@ -1829,14 +1829,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "maps/feature-map/secrets.md", "ui/src/components/task-chat/TaskChatMarker.tsx", "ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx" ], - "fingerprint": "2eba045db9127e5cd8d8bf7c9664e4598a5c84be117ae6159ce7171b24f2005d", + "fingerprint": "14757d026311d143e9301d6294d2f46f40dad25af8ea3f8d5cf1893b260f82db", "stableId": "task-credential-access-01-notice", "sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice" }, @@ -1862,7 +1862,7 @@ "ui/storybook/stories/secrets/credential-access/02-task-page.stories.tsx", "ui/storybook/support/credential-access/CredentialTaskPage.tsx" ], - "fingerprint": "70a07a45f3633984db4bbadd599c3745e098f17100f8421b0c3cd1fec46e2186", + "fingerprint": "1289c5e6659cd318f00b63d4f16790bbac988d8d18f6726e10262223c11e66fe", "stableId": "task-credential-access-02-task-page", "sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page" }, @@ -2210,13 +2210,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/ExecutionBlockerNotice.tsx", "ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx" ], - "fingerprint": "bee366ba1266e6692c8f0ee027c449fdda6b51762369ba431871a5af5ee018c6", + "fingerprint": "018d8d687921ea109cb98f587abd59b94722f1f148940852326417ce8a457b04", "stableId": "task-chat-recovery-notice", "sidebarTitle": "Projects and execution/Recovery/Task chat/Recovery notice" }, @@ -2230,7 +2230,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/ActiveAgentsPanel.tsx", @@ -2240,7 +2240,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/execution-recovery.stories.tsx" ], - "fingerprint": "6f8b245b169d95046bdb52513dfaf4937a96f807b72470c198863471f59c8149", + "fingerprint": "82ddff0cf6d98e558f9038d59960758347ad39f3c7a9aaf18f29214a8ad81756", "stableId": "tasks-execution-recovery", "sidebarTitle": "Projects and execution/Recovery/Tasks/Execution recovery" }, @@ -2861,7 +2861,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "maps/feature-map/task-coordination.md", @@ -2869,7 +2869,7 @@ "ui/src/components/task-chat/TaskChatBlockerLinks.tsx", "ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx" ], - "fingerprint": "6a84d7f60318d33a1835a668873095aaad1c215a83bbbb4735823fdcb3291140", + "fingerprint": "52f861389de245bda4d9c582581b49ce0b9442d60ee7eff950adc6801aaca8a4", "stableId": "product-issue-blocked-notice", "sidebarTitle": "Projects and execution/Recovery/Product/Issue/Blocked notice" }, @@ -3093,13 +3093,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueScheduledRetryCard.tsx", "ui/storybook/stories/recovery/model-capacity-retry.stories.tsx" ], - "fingerprint": "5e08f681f3f96cb7be32e316d3f4066e5239dcf0f4393cb9ec09d599734f201e", + "fingerprint": "1db367a2b77b8c747ab5c81c1fe3317da02bf294308e9e09af827bcd12ee0b9f", "stableId": "tasks-model-capacity-retry", "sidebarTitle": "Projects and execution/Recovery/Tasks/Model capacity retry" }, @@ -3358,7 +3358,7 @@ "ui/storybook/prototypes/originating-tasks/OriginatingTasks.tsx", "ui/storybook/stories/task-coordination/originating-tasks.stories.tsx" ], - "fingerprint": "59123e73a0286fd11a48376ee152a974d730c59bd5421fd273cfca3a185f6e08", + "fingerprint": "32c9981ce024550b26636b97d7337f7b34a1207e8bcdd5759a11d9a7ddb204fe", "stableId": "ux-labs-tasks-created-from-a-task", "sidebarTitle": "Tasks and conversations/Delegation, dependencies, and signoff/UX Labs/Tasks Created From a Task" }, @@ -3370,7 +3370,7 @@ "implementation": "guide", "boundary": "static", "notes": "Matches the upstream feature index and order, including features with no cataloged examples. Related suite links and classifications do not imply execution coverage.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "ui/src/components/ui/badge.tsx", "ui/src/components/ui/input.tsx", @@ -3793,7 +3793,7 @@ "ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "d4b5ba6bce33a9657d859d3278b1f84bbc9b77cffd24e370214a0fce24c0a02b", + "fingerprint": "ebd9132a1f0228f775b1f382fc30dcae68e65284c8118c7b8a2eee3219c741ed", "stableId": "private-tasks-01-creation", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/01 Creation" }, @@ -3939,7 +3939,7 @@ "ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "5fb218d88c3b6815839732e4226a27f4b40a2df46c8cbad6709cb665546b4248", + "fingerprint": "6069116fffc3a661df1fb10245f3f6d8045086efd71c886a8064a27eed09ac42", "stableId": "private-tasks-06-full-product-pages", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/06 Full product pages" }, @@ -3986,7 +3986,7 @@ "ui/storybook/stories/access/private-tasks/07-user-stories.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "5fbe5c61f8754bc0c1db1c6a990b9c44bc5d941b7b61de77fac76300e6c42a22", + "fingerprint": "483a8bfae265844bfa10ebd8529572f43c23c8474d414b45ba0b663efc280dbd", "stableId": "private-tasks-07-user-stories", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/07 User stories" }, @@ -4437,7 +4437,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/DispositionRecoveryNotice.tsx", @@ -4446,7 +4446,7 @@ "ui/storybook/prototypes/recovery-notice/RecoveryNoticePreview.tsx", "ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx" ], - "fingerprint": "cfd7345545112ea9ecc13035ebd187451117de5ed010f439cfb8c6962461fb3f", + "fingerprint": "e83bcaf099227bfcd17d571f680121b5b51d5f605d76479165000a575229a884", "stableId": "design-previews-recovery-notice", "sidebarTitle": "Projects and execution/Recovery/Design previews/Recovery notice" }, @@ -4697,7 +4697,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueProperties.tsx", @@ -4705,7 +4705,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/scheduled-retry.stories.tsx" ], - "fingerprint": "43ef1917079251d558b3f52e5af750dfb69a54d3326d31494dbf037baeefe193", + "fingerprint": "f360f4207ef2b1cef18eee2307c50a7bce3a5f5bbf8966b24cd86b3ec50d21b4", "stableId": "product-issue-scheduled-retry-surfaces", "sidebarTitle": "Projects and execution/Recovery/Product/Issue Scheduled retry surfaces" }, @@ -5009,8 +5009,8 @@ "status": "unreviewed", "implementation": "production", "boundary": "fixtures", - "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite.", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueBlockedNotice.tsx", @@ -5019,7 +5019,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/source-issue-recovery.stories.tsx" ], - "fingerprint": "1b6e44d4fb4ab7e2e63fb8816aa31fbe3267e10647f238ed82e6591a82d2e918", + "fingerprint": "949814263788ae0e3501b67bbd26118f150db219fc04e5cf653f18dd345ec8d5", "stableId": "paperclip-source-issue-recovery", "sidebarTitle": "Projects and execution/Recovery/Paperclip/Source Issue Recovery" }, @@ -5034,14 +5034,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/SourceResolvedFoldBadge.tsx", "ui/src/components/SourceResolvedFoldCallout.tsx", "ui/storybook/stories/recovery/source-resolved-fold.stories.tsx" ], - "fingerprint": "3af3daff618b8c278bcbf10ba3e5d081d87240c680758ccb3078eccdf9283122", + "fingerprint": "70cab38574df19267e35ab13c10c4a3eea444bb3b6e72c81f739f21cfc519df0", "stableId": "paperclip-source-resolved-fold", "sidebarTitle": "Projects and execution/Recovery/Paperclip/Source-resolved Fold" }, @@ -5122,7 +5122,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/tasks/successful-run-handoff.stories.tsx" ], - "fingerprint": "2a7db108d46f35f37dd9346d2fc039e61dff8c688e09d9ed7b0e777e7f79af0d", + "fingerprint": "27ad7c3c2b09acf5ac07163d44026511bd0c2b806387b591fef5fca343b0c63b", "stableId": "paperclip-successful-run-handoff", "sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff" }, @@ -5575,14 +5575,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/WorkspaceBaseRefRecoveryNotice.tsx", "ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx", "ui/storybook/support/workspace-branch-recovery/fixtures.ts" ], - "fingerprint": "77f708cc660f6829cadaffa6139930a102109976c41f7e24e2f662ed687d2412", + "fingerprint": "eeaa7044a1d5a2b5355a8ac2375820e3611c71920e88c04d87e5d7ce64ed929a", "stableId": "workspace-branch-recovery-01-repair-card", "sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/01 Repair card" }, @@ -5607,7 +5607,7 @@ "ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx", "ui/storybook/support/workspace-branch-recovery/fixtures.ts" ], - "fingerprint": "5eedfcb5738c125d3fa8ed7dc4b3d20a300b216de51f1ae5e58ede98d38fd75c", + "fingerprint": "f8a090cfdbadfe77d8ee064b99fc11b8f5e097d4e72173df2496280ef52152d1", "stableId": "workspace-branch-recovery-02-task-journey", "sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/02 Task journey" }, @@ -5671,6 +5671,53 @@ "fingerprint": "b9cd51c58b31ab445e2997b9f2a23c7a7b3f4ecd7b6dd39fc0f35ae0779311df", "stableId": "workspaces-service-control-bar", "sidebarTitle": "Projects and execution/Workspaces, services, and files/Workspaces/Service control bar" + }, + { + "file": "ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx", + "capabilities": [ + "recovery/workspace", + "recovery/source-action" + ], + "purpose": "Workspace finalization recovery/01 Notice", + "status": "current", + "implementation": "production", + "boundary": "fixtures", + "notes": "Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests.", + "reviewedAt": "2026-10-13", + "sources": [ + "maps/feature-map/recovery.md", + "ui/src/components/WorkspaceExportRecovery.tsx", + "ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx", + "ui/storybook/support/workspace-finalization-recovery/fixtures.tsx" + ], + "fingerprint": "45bf8c793c91362634b7ce32ed0ec2007861b7fefbfd0e2da692c3dc804a53c9", + "stableId": "workspace-finalization-recovery-01-notice", + "sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/01 Notice" + }, + { + "file": "ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx", + "capabilities": [ + "recovery/workspace", + "recovery/source-action" + ], + "purpose": "Workspace finalization recovery/02 Task journey", + "status": "unreviewed", + "implementation": "mixed", + "boundary": "fixtures", + "notes": "Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance.", + "reviewedAt": "2026-10-13", + "sources": [ + "maps/feature-map/recovery.md", + "ui/src/components/Layout.tsx", + "ui/src/components/WorkspaceExportRecovery.tsx", + "ui/src/components/task-chat/TaskChatBubble.tsx", + "ui/src/components/task-chat/TaskChatComposer.tsx", + "ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx", + "ui/storybook/support/workspace-finalization-recovery/fixtures.tsx" + ], + "fingerprint": "aa76cd28dde8d50abd208b34283249170f3a85435ba82fff169a1c33433dfca1", + "stableId": "workspace-finalization-recovery-02-task-journey", + "sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/02 Task journey" } ] } diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index b1655cf3169..48c3110dac1 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1999,6 +1999,7 @@ export { resolveIssueRecoveryActionSchema, isValidExistingBranchName, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index e7b3197a550..8e224de0924 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -441,6 +441,7 @@ export { resolveIssueRecoveryActionSchema, isValidExistingBranchName, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, diff --git a/packages/shared/src/validators/issue.ts b/packages/shared/src/validators/issue.ts index 8fddad793ba..0d9ea3bdceb 100644 --- a/packages/shared/src/validators/issue.ts +++ b/packages/shared/src/validators/issue.ts @@ -557,6 +557,14 @@ const RESOLVE_ISSUE_RECOVERY_ACTION_OUTCOMES = [ "cancelled", ] as const; +export const resumeWorkspaceFinalizationSchema = z.object({ + actionId: z.string().guid(), + runId: z.string().guid(), + ownerToken: z.string().guid(), + controllerAndCopybackStopped: z.literal(true), + stopEvidence: z.string().trim().min(20).max(12000), +}).strict(); + export const retryWorkspaceExportSchema = z.object({ actionId: z.string().guid(), runId: z.string().guid(), diff --git a/server/src/__tests__/authorization-service.test.ts b/server/src/__tests__/authorization-service.test.ts index 4e1f3f63c2e..a2b28921c5e 100644 --- a/server/src/__tests__/authorization-service.test.ts +++ b/server/src/__tests__/authorization-service.test.ts @@ -1810,6 +1810,41 @@ describeEmbeddedPostgres("authorization service", () => { expect(await auth.decide({ actor, action: "agent:wake", resource: { ...resource, companyId: otherCompany.id } })).toMatchObject({ allowed: false }); }); + it.each(["session", "cloud_tenant"] as const)("restricts %s workspace-owner stop confirmation to administrators", async source => { + const company = await createCompany(db, "OwnerStop"); + const otherCompany = await createCompany(db, "OtherOwnerStop"); + const userId = await createUser(db); + await db.insert(companyMemberships).values({ companyId: company.id, + principalType: "user", principalId: userId, status: "active", membershipRole: "owner" }); + const actor = { type: "board" as const, source, userId, companyIds: [company.id] }; + const auth = authorizationService(db); + const request = { actor, action: "runtime:confirm_owner_stop" as const, + resource: { type: "company" as const, companyId: company.id } }; + for (const membershipRole of ["owner", "admin", "operator", "member", "viewer", null]) { + await db.update(companyMemberships).set({ membershipRole }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide(request)).allowed, `role ${membershipRole}`).toBe(membershipRole === "owner" || membershipRole === "admin"); + } + await db.update(companyMemberships).set({ membershipRole: "admin", status: "suspended" }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide(request)).allowed).toBe(false); + await db.update(companyMemberships).set({ status: "active" }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide({ ...request, resource: { type: "company", companyId: otherCompany.id } })).allowed).toBe(false); + }); + + it("does not allow agent delegation or stale cloud admin rows to confirm an owner stop", async () => { + const company = await createCompany(db, "OwnerStopDelegation"); + const userId = await createUser(db); + const agent = await createAgent(db, company.id, { role: "ceo" }); + await db.insert(companyMemberships).values({ companyId: company.id, + principalType: "user", principalId: userId, status: "active", membershipRole: "operator" }); + await db.insert(instanceUserRoles).values({ userId, role: "instance_admin" }); + const auth = authorizationService(db); + const request = { action: "runtime:confirm_owner_stop" as const, resource: { type: "company" as const, companyId: company.id } }; + expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: false } })).allowed).toBe(false); + expect((await auth.decide({ ...request, actor: { type: "agent", agentId: agent.id, companyId: company.id, onBehalfOfUserId: userId } })).allowed).toBe(false); + expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: true } })).allowed).toBe(true); + expect((await auth.decide({ ...request, actor: { type: "board", source: "local_implicit" } })).allowed).toBe(true); + }); + it("limits viewer members to read-only visibility actions", async () => { const company = await createCompany(db, "BoardViewerVisibility"); const userId = `user-${randomUUID()}`; diff --git a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts index 6aa18723cba..4afb8bad37c 100644 --- a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts +++ b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts @@ -162,8 +162,10 @@ const mockLogActivity = vi.hoisted(() => vi.fn(async () => undefined)); const mockObserveCrossIssueInfluence = vi.hoisted(() => vi.fn(async () => null)); const mockRetryWorkspaceExport = vi.hoisted(() => vi.fn()); +const mockResumeWorkspaceFinalization = vi.hoisted(() => vi.fn()); function registerRouteMocks() { + vi.doMock("../services/native-runtime/native-workspace-owner-recovery.js", () => ({ resumeNativeWorkspaceAfterOwnerStop: mockResumeWorkspaceFinalization })); vi.doMock("../services/native-runtime/native-workspace-export-retry.js", () => ({ retryNativeWorkspaceExport: mockRetryWorkspaceExport })); vi.doMock("@paperclipai/shared/telemetry", () => ({ trackAgentTaskCompleted: vi.fn(), @@ -2009,6 +2011,52 @@ describe("agent issue mutation checkout ownership", () => { ); }); + it("resumes a verified stopped workspace owner without waking the provider", async () => { + mockAccessService.decide.mockResolvedValue({ allowed: true }); + mockResumeWorkspaceFinalization.mockResolvedValue({ runId: ownerRunId, status: "queued" }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }); + expect(res.status).toBe(202); + expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({ action: "runtime:confirm_owner_stop" })); + expect(mockResumeWorkspaceFinalization).toHaveBeenCalledWith(expect.objectContaining({ companyId, issueId, runId: ownerRunId, ownerToken: ownerRunId, actorId: "board-user" })); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + + it.each([{ controllerAndCopybackStopped: false }, { stopEvidence: "" }])("requires explicit stop evidence: %j", async invalid => { + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited.", ...invalid }); + expect(res.status).toBe(400); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + }); + + it.each(["agent", "viewer"])("rejects %s workspace-owner release before admission", async kind => { + mockAccessService.decide.mockImplementation(async (input: { action: string }) => ({ allowed: input.action === "issue:read", explanation: "No runtime access" })); + const res = await request(await createApp(kind === "agent" ? ownerActor() : boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }); + expect(res.status).toBe(403); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + }); + + it("rejects an ordinary runtime manager's claimed foreign-owner stop", async () => { + mockAccessService.decide.mockImplementation(async ({ action }: { action: string }) => ({ + allowed: action === "issue:read" || action === "runtime:manage", + explanation: "A deployment administrator must verify the controller stopped", + })); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "I claim that the foreign controller and all its processes have stopped." }); + expect(res.status).toBe(403); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + it("queues board export-only recovery without calling provider wake", async () => { mockAccessService.decide.mockResolvedValue({ allowed: true }); mockRetryWorkspaceExport.mockResolvedValue({ runId: ownerRunId, status: "queued" }); @@ -2030,7 +2078,25 @@ describe("agent issue mutation checkout ownership", () => { expect(res.status).toBe(403); expect(mockRetryWorkspaceExport).not.toHaveBeenCalled(); }); - it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => { + it.each(["done", "in_review"])("does not bypass unverified copyback via false_positive to %s", async sourceIssueStatus => { + const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }); + mockIssueService.getById.mockResolvedValue(sourceIssue); + mockIssueService.update.mockImplementation(async (_id: string, patch: Record) => ({ ...sourceIssue, ...patch })); + mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({ + id: recoveryActionId, status: "active", kind: "active_run_watchdog", ownerType: "board", + ownerAgentId: null, returnOwnerAgentId: ownerAgentId, + cause: "native_workspace_finalization_owner_unverified", evidence: { runId: ownerRunId }, + }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resolve`) + .send({ actionId: recoveryActionId, outcome: "false_positive", sourceIssueStatus }); + expect(res.status).toBe(409); + expect(res.body.details?.code).toBe("workspace_owner_stop_required"); + expect(mockIssueService.update).not.toHaveBeenCalled(); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + + it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted", "native_workspace_finalization_owner_unverified"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => { const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }); mockIssueService.getById.mockResolvedValue(sourceIssue); mockIssueService.update.mockImplementation(async (_id: string, patch: Record) => ({ ...sourceIssue, ...patch })); @@ -2043,7 +2109,7 @@ describe("agent issue mutation checkout ownership", () => { .post(`/api/issues/${issueId}/recovery-actions/resolve`) .send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus }); expect.soft(res.status).toBe(409); - expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : "workspace_export_retry_required"); + expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : cause === "native_workspace_finalization_owner_unverified" ? "workspace_owner_stop_required" : "workspace_export_retry_required"); expect.soft(mockIssueService.update).not.toHaveBeenCalled(); expect.soft(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); }); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 7af8d4c1239..fa2abf0c522 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -17,6 +17,7 @@ import { setIssueTitleSchema } from "@paperclipai/shared"; import { resolveConfirmationFromComment } from "../services/confirmation-comment-resolution.js"; import { createIssueReadTiming } from "../services/issue-read-timing.js"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js"; import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js"; import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js"; import { deliverConversationComments, isConversation } from "../services/agent-conversations.js"; @@ -96,6 +97,7 @@ import { resolveCreateIssueStatusDefault, resolveIssueRecoveryActionSchema, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, runnerGoalActionRequestSchema, feedbackTargetTypeSchema, feedbackTraceStatusSchema, @@ -9511,6 +9513,17 @@ export function issueRoutes( }); }); + router.post("/issues/:id/recovery-actions/resume-workspace-finalization", validate(resumeWorkspaceFinalizationSchema), async (req, res) => { + assertBoard(req); + const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found"); + if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return; + const decision = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", resource: { type: "company", companyId: issue.companyId } }); + if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision)); + const receipt = await resumeNativeWorkspaceAfterOwnerStop({ db, companyId: issue.companyId, issueId: issue.id, + ...req.body, actorId: getActorInfo(req).actorId }); + res.status(202).json(receipt); + }); + router.post("/issues/:id/recovery-actions/retry-workspace-export", validate(retryWorkspaceExportSchema), async (req, res) => { assertBoard(req); const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found"); @@ -9757,6 +9770,10 @@ export function issueRoutes( throw conflict("Unsafe workspace export recovers automatically without another provider turn.", { code: "workspace_export_automatic_recovery" }); } + if ((outcome === "restored" || outcome === "false_positive") && activeRecoveryAction.cause === "native_workspace_finalization_owner_unverified") { + throw conflict("Verify the previous controller and its copyback processes stopped, then use Resume saved result.", { code: "workspace_owner_stop_required" }); + } + if (outcome === "restored" && isNativeWorkspaceExportRepairCause(activeRecoveryAction.cause)) { throw conflict("Repair the retained sandbox, then use Retry workspace export to finish the accepted result without another provider turn.", { code: "workspace_export_retry_required" }); } diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 532e3a66a22..e749eaeef3c 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -240,6 +240,7 @@ import { createAcceptedPlanDecompositionSchema, resolveIssueRecoveryActionSchema, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, cancelIssueThreadInteractionSchema, // Secret provider configs and remote import createSecretProviderConfigSchema, @@ -10635,6 +10636,15 @@ registerCurrentRoute({ summary: "List issue recovery actions", }); +registerCurrentRoute({ + method: "post", + path: "/api/issues/{id}/recovery-actions/resume-workspace-finalization", + tags: ["issues"], + summary: "Resume a saved native result after operator verification of the prior workspace owner's stop", + body: resumeWorkspaceFinalizationSchema, + responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict }, +}); + registerCurrentRoute({ method: "post", path: "/api/issues/{id}/recovery-actions/retry-workspace-export", diff --git a/server/src/services/authorization.ts b/server/src/services/authorization.ts index 7bc119b86a4..9b68990080e 100644 --- a/server/src/services/authorization.ts +++ b/server/src/services/authorization.ts @@ -94,6 +94,7 @@ export type AuthorizationAction = | "project:read" | "project:write_workspace" | "runtime:manage" + | "runtime:confirm_owner_stop" | "secrets:read" | "secrets:propose"; @@ -184,6 +185,7 @@ function permissionForAction(action: AuthorizationAction): PermissionKey | null action === "issue:read" || action === "project:read" || action === "runtime:manage" || + action === "runtime:confirm_owner_stop" || action === "secrets:read" || action === "secrets:propose" ) { @@ -1858,6 +1860,11 @@ export function authorizationService(db: Db | DbTransaction) { }); } + if (input.action === "runtime:confirm_owner_stop" && input.actor.type !== "board") { + return deny({ action: input.action, reason: "deny_unsupported_action", + explanation: "Only a deployment administrator can confirm a workspace controller has stopped." }); + } + if (input.actor.type === "board") { let taskAssignmentPolicyEffect: AssignmentPolicyEffect | null = null; if (input.actor.source === "local_implicit") { @@ -1886,6 +1893,15 @@ export function authorizationService(db: Db | DbTransaction) { explanation: "Allowed because the actor is an instance admin.", }); } + if (input.action === "runtime:confirm_owner_stop") { + const membership = await getActiveMembership(companyId, "user", input.actor.userId ?? ""); + if (membership?.membershipRole === "owner" || membership?.membershipRole === "admin") { + return allow({ action: input.action, reason: "allow_role_default", + explanation: "Allowed by the active company owner or administrator role." }); + } + return deny({ action: input.action, reason: "deny_missing_grant", + explanation: "Only a company owner, company administrator, or instance administrator can confirm a workspace controller has stopped." }); + } // What instance-admin elevation used to give cloud tenant users is // replaced by company-scoped visibility: an active membership in the // resource company grants the same read surface a same-company agent diff --git a/server/src/services/cloud-runtime-identity.ts b/server/src/services/cloud-runtime-identity.ts index e6567f2cf9e..435c7f7562b 100644 --- a/server/src/services/cloud-runtime-identity.ts +++ b/server/src/services/cloud-runtime-identity.ts @@ -231,7 +231,7 @@ function decodeJsonPart(part: string, label: string): Record { return parsed as Record; } -function publicKeyForKid(env: NodeJS.ProcessEnv, kid: string) { +export function cloudRuntimePublicKeyForKid(env: NodeJS.ProcessEnv, kid: string) { const raw = nonEmpty(env.PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS); if (!raw) throw new Error("PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS is not configured"); let parsed: unknown; @@ -276,7 +276,7 @@ function verifyClaims(input: { ) { throw new Error("Cloud runtime identity protected header is invalid"); } - const key = publicKeyForKid(input.env, header.kid); + const key = cloudRuntimePublicKeyForKid(input.env, header.kid); const signature = Buffer.from(encodedSignature, "base64url"); const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii"); if (!verify(null, signingInput, key, signature)) { @@ -523,7 +523,7 @@ export function verifyCloudControlAssertion(input: { ) { throw new Error("Cloud control protected header is invalid"); } - const key = publicKeyForKid(env, header.kid); + const key = cloudRuntimePublicKeyForKid(env, header.kid); const signature = Buffer.from(encodedSignature, "base64url"); const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii"); if (!verify(null, signingInput, key, signature)) { diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index aeb7384bee3..6dd31f091cb 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -375,6 +375,18 @@ export function issueRecoveryActionService(db: Db) { // beyond the advertised cap. A distinct identity can still supersede the // exhausted action through the branch above. if (isRecoveryBudgetExhausted(existing.evidence ?? {})) { + // Older sweeps replaced this operator-only hold with instructions to + // start another run. Restore the exact copyback recovery instructions + // without reopening its budget or erasing a recorded stop confirmation. + if (existing.cause === "native_workspace_finalization_owner_unverified" + && input.cause === existing.cause && input.fingerprint === existing.fingerprint + && !existing.evidence?.workspaceOwnerStop && existing.nextAction !== input.nextAction) { + const [updated] = await db.update(issueRecoveryActions).set({ nextAction: input.nextAction, updatedAt: now }) + .where(and(eq(issueRecoveryActions.id, existing.id), + inArray(issueRecoveryActions.status, [...ACTIVE_RECOVERY_ACTION_STATUSES]), + sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).returning(); + if (updated) return toReadModel(updated); + } return existing; } const nextAttemptCount = @@ -417,8 +429,9 @@ export function issueRecoveryActionService(db: Db) { fingerprint: existing.fingerprint, }, }, - nextAction: - `Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` + + nextAction: existing.cause === "native_workspace_finalization_owner_unverified" + ? input.nextAction + : `Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` + "Review the infrastructure failure and explicitly choose a replacement run or provider configuration.", wakePolicy: null, monitorPolicy: null, diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts index fcbff05fdee..5168b9bf856 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -1,15 +1,19 @@ +import { logActivity } from "../activity-log.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { nativeSha256 } from "./canonical.js"; import { NATIVE_WORKSPACE_OWNER_UNVERIFIED } from "./native-workspace-finalization-recovery-state.js"; import { randomUUID } from "node:crypto"; import os from "node:os"; -import { and, eq, sql } from "drizzle-orm"; -import { heartbeatRuns, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js"; import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, } from "./native-restart-recovery.js"; +import { currentWorkspaceWorker, requestWorkspaceWorkerStopProof, type WorkspaceWorker, type WorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js"; + const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; const activeOwnerTokens = new Set(); // Positive same-boot evidence, retained only when durable cleanup could not be @@ -21,6 +25,7 @@ type Owner = { hostname: string; pid: number; processStartedAt: string; + worker?: WorkspaceWorker; }; export type NativeWorkspaceFinalizationOwnership = { token: string; @@ -58,7 +63,7 @@ export async function withNativeWorkspaceFinalizationOwnership( const identity = await currentNativeControllerIdentity(); const owner: Owner = { token: randomUUID(), controllerBootId: identity.bootId, hostname: os.hostname(), pid: identity.pid, - processStartedAt: identity.processStartedAt.toISOString(), + processStartedAt: identity.processStartedAt.toISOString(), worker: currentWorkspaceWorker(), }; const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.runtimeMode, "native")); const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`); @@ -73,13 +78,42 @@ export async function withNativeWorkspaceFinalizationOwnership( if (!run) throw new Error("native_workspace_finalization_binding_missing"); const rawPrior = run.profile?.[OWNER_KEY]; const prior = readOwner(rawPrior); + let automaticResultId: string | undefined; + const currentRecoveryResult = async (db: Db) => { + if (!run.issueId) return null; + // Use the arbiter's lock order and release row locks before copyback I/O. + const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId), + eq(nativeRunFinalizations.issueId, run.issueId), + )).for("update"); + const [issue] = await db.select().from(issues).where(and( + eq(issues.companyId, input.companyId), eq(issues.id, run.issueId), + )).for("update"); + const [currentRun] = await db.select().from(heartbeatRuns).where(scope).for("update"); + return coordinator && issue && currentRun ? currentNativeWorkspaceRecoveryResult(db, { + companyId: input.companyId, coordinator, issue, run: currentRun, + }) : null; + }; + let platformProof: WorkspaceWorkerStopProof | null = null; if (rawPrior != null) { const requireStopVerification = async () => { const fingerprint = nativeSha256({ runId: input.runId, owner: rawPrior }); + const nextAction = "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn."; const recovery = issueRecoveryActionService(input.db); const existing = run.issueId ? await recovery.getActiveForIssue(input.companyId, run.issueId) : null; // Rechecking the same physical owner is not a new recovery attempt. if (existing?.cause === NATIVE_WORKSPACE_OWNER_UNVERIFIED && existing.fingerprint === fingerprint) { + if (!existing.evidence?.workspaceOwnerStop) { + await input.db.update(issueRecoveryActions).set({ nextAction, updatedAt: new Date(), + evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"operator_required"}'::jsonb)`, + }).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, existing.id), + eq(issueRecoveryActions.fingerprint, fingerprint), eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`, + sql`(${issueRecoveryActions.nextAction} is distinct from ${nextAction} or ${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' is distinct from 'operator_required')`, + )); + } return { acquired: false } as const; } if (run.issueId) await recovery.upsertSourceScoped({ @@ -87,18 +121,21 @@ export async function withNativeWorkspaceFinalizationOwnership( kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: run.agentId, cause: NATIVE_WORKSPACE_OWNER_UNVERIFIED, fingerprint, - evidence: { runId: input.runId, owner: rawPrior }, - nextAction: "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn.", + evidence: { runId: input.runId, owner: rawPrior, workspaceOwnerRecovery: { state: "operator_required" } }, + nextAction, wakePolicy: null, maxAttempts: 1, supersedeOnIdentityChange: true, }); return { acquired: false } as const; }; - // A foreign host or malformed receipt cannot prove the old physical owner stopped. - if (!prior || prior.hostname !== owner.hostname) return requireStopVerification(); + // The harness checks the exact owned deployment, including orphaned children. + // A live deployment, unsigned answer or older receipt stays fenced. + if (prior?.worker) platformProof = await requestWorkspaceWorkerStopProof({ runId: input.runId, ownerToken: prior.token, worker: prior.worker }); + if (!prior) return requireStopVerification(); + if (!platformProof && prior.hostname !== owner.hostname) return requireStopVerification(); const knownJoinedHere = prior.controllerBootId === identity.bootId && prior.pid === identity.pid && joinedOwnerTokens.has(prior.token) && !activeOwnerTokens.has(prior.token); - if (!knownJoinedHere) { + if (!platformProof && !knownJoinedHere) { const takeover = await evaluateNativeControllerTakeover({ owner: { leaseOwner: prior.token, leaseExpiresAt: new Date(0), controllerPid: prior.pid, controllerProcessStartedAt: new Date(prior.processStartedAt) }, now: new Date(), @@ -119,15 +156,32 @@ export async function withNativeWorkspaceFinalizationOwnership( } // Claim before the potentially ambiguous write so finally can remove // only our token even if the server committed but its response was lost. - activeOwnerTokens.add(owner.token); - claimed = true; - const [claimedRun] = await input.db.update(heartbeatRuns).set({ - runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb)`, - }).where(and(scope, prior - ? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${prior.token}` - : sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null` - )).returning({ id: heartbeatRuns.id }); + const claimOwner = async (db: Db) => { + activeOwnerTokens.add(owner.token); + claimed = true; + const [claimedRun] = await db.update(heartbeatRuns).set({ + runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb) || ${JSON.stringify(platformProof ? { nativeWorkspaceOwnerStopProof: platformProof } : {})}::jsonb`, + }).where(and(scope, prior + ? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY} = ${JSON.stringify(rawPrior)}::jsonb` + : sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null` + )).returning({ id: heartbeatRuns.id }); + return claimedRun; + }; + // A stop receipt proves the old worker exited, not that its task is still + // current. Revalidate after the provider read and atomically with the CAS. + const claimedRun = platformProof ? await input.db.transaction(async tx => { + const result = await currentRecoveryResult(tx as unknown as Db); + if (!result) return null; + const claimedRun = await claimOwner(tx as unknown as Db); + if (claimedRun) automaticResultId = result.id; + return claimedRun; + }) : await claimOwner(input.db); if (!claimedRun) return { acquired: false } as const; + if (platformProof && run.issueId) await logActivity(input.db, { + companyId: input.companyId, actorType: "system", actorId: "workspace-recovery", + action: "issue.workspace_owner_stop_verified", entityType: "issue", entityId: run.issueId, + runId: input.runId, details: { proof: platformProof }, + }); if (prior) joinedOwnerTokens.delete(prior.token); const ownership: NativeWorkspaceFinalizationOwnership = { token: owner.token, @@ -137,12 +191,25 @@ export async function withNativeWorkspaceFinalizationOwnership( await lock.execute(sql`select 1`); const [current] = await input.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(owned).limit(1); if (!current) throw new Error("owner_replaced"); + if (automaticResultId) { + const result = await input.db.transaction(tx => currentRecoveryResult(tx as unknown as Db)); + if (result?.id !== automaticResultId) throw new Error("task_replaced"); + } } catch (cause) { throw new NativeWorkspaceFinalizationOwnershipLostError({ cause }); } }, }; await ownership.assertHeld(); + if (platformProof && run.issueId) await input.db.update(issueRecoveryActions).set({ + evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"resuming"}'::jsonb)`, + updatedAt: new Date(), + }).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId), + eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED), + eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + )); actionPromise = action(ownership); const value = await actionPromise; await ownership.assertHeld(); diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index 766dd1cf539..30737efb30b 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -1,3 +1,8 @@ +import * as platformProof from "./native-workspace-platform-proof.js"; +import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { nativeSha256 } from "./canonical.js"; +import { resumeNativeWorkspaceAfterOwnerStop } from "./native-workspace-owner-recovery.js"; +import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { spawn } from "node:child_process"; import { once } from "node:events"; import { readProcessStartedAt } from "../hot-restart.js"; @@ -9,6 +14,8 @@ import path from "node:path"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { and, desc, eq, sql } from "drizzle-orm"; import { + activityLog, + agentWakeupRequests, agents, companies, completionContracts, @@ -19,11 +26,15 @@ import { nativeRunFinalizations, nativeRunResults, issueRecoveryActions, + issueThreadInteractions, projects, workspaceOperations, + workAssessments, + statusDecisions, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js"; +import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js"; describe("native workspace finalization recovery", () => { let temporary: Awaited>; @@ -159,6 +170,225 @@ describe("native workspace finalization recovery", () => { } }); + async function seedAbandonedCopyback() { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Accepted yield, abandoned copyback" }); + const owner = { token: randomUUID(), hostname: "replaced-controller", pid: 2, + processStartedAt: "2026-10-10T00:02:55.560Z", controllerBootId: randomUUID() }; + await db.update(heartbeatRuns).set({ status: "running", nativePhase: "workspace_finalizing", + runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" }, + }).where(eq(heartbeatRuns.id, seeded.runId)); + await db.update(issues).set({ executionRunId: seeded.runId }).where(eq(issues.id, seeded.issueId)); + await db.update(nativeRunFinalizations).set({ phase: "workspace_finalizing" }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const recovery = issueRecoveryActionService(db); + const hold = { companyId, sourceIssueId: seeded.issueId, kind: "active_run_watchdog" as const, + ownerType: "board" as const, cause: "native_workspace_finalization_owner_unverified", + fingerprint: nativeSha256({ runId: seeded.runId, owner }), evidence: { runId: seeded.runId, owner }, maxAttempts: 1, + nextAction: "Verify the previous controller and its workspace-copyback processes have stopped. Resume workspace finalization without another provider turn." }; + const action = await recovery.upsertSourceScoped(hold); + await recovery.upsertSourceScoped(hold); + return { ...seeded, owner, actionId: action.id, companyId, db, ownerToken: owner.token, + actorId: "operator", controllerAndCopybackStopped: true as const, + stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }; + } + + it("retains copyback-specific instructions after repeated recovery sweeps", async () => { + const seed = await seedAbandonedCopyback(); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(action.status).toBe("escalated"); + expect(action.nextAction).toContain("workspace-copyback processes have stopped"); + expect(action.nextAction).not.toContain("replacement run"); + await db.update(issueRecoveryActions).set({ nextAction: "Start a replacement run" }).where(eq(issueRecoveryActions.id, seed.actionId)); + await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected"); + const [repaired] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(repaired.nextAction).toContain("workspace-copyback processes have stopped"); + expect(repaired.attemptCount).toBe(action.attemptCount); + expect(repaired.evidence?.recoveryBudget).toEqual(action.evidence?.recoveryBudget); + }); + + it("releases only the verified owner and resumes the saved result without changing its disposition", async () => { + const seed = await seedAbandonedCopyback(); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false }); + const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" }); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(after).toEqual(before); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson).toEqual({ preserved: "workspace descriptor" }); + expect(run.status).toBe("running"); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(action.evidence?.workspaceOwnerStop).toMatchObject({ owner: seed.owner, actorId: "operator", resultId: before.resultId, stopEvidence: seed.stopEvidence }); + expect(action.status).toBe("escalated"); // Only commitment resolves the hold. + expect(action.wakePolicy).toBeNull(); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1); + const file = path.join(workspaceRoot, seed.runId); + expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => { + await fs.writeFile(file, "accepted work"); + return "resumed"; + })).toEqual({ acquired: true, value: "resumed" }); + expect(await fs.readFile(file, "utf8")).toBe("accepted work"); + // A duplicate confirmation neither clears a newer owner nor writes another audit. + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toEqual(receipt); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1); + }); + + it.each(["expired", "live", "unknown"] as const)("checks the coordinator lease before verified recovery: %s", async state => { + const seed = await seedAbandonedCopyback(); + const leaseExpiresAt = state === "unknown" ? null : new Date(Date.now() + (state === "expired" ? -60_000 : 60_000)); + await db.update(nativeRunFinalizations).set({ leaseOwner: "previous-reconciler", leaseExpiresAt }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (state === "expired") { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].leaseOwner).toBe("previous-reconciler"); + } else { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0].runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(seed.owner); + } + }); + + it.each(["pending", "resolved", "superseded", "unassigned"] as const)("requires authoritative reviewer recovery: %s", async state => { + const seed = await seedAbandonedCopyback(); + const reviewerId = randomUUID(), sourceRunId = randomUUID(), sourceResultId = randomUUID(), assessmentId = randomUUID(); + const decisionId = randomUUID(), interactionId = randomUUID(); + await db.insert(agents).values({ id: reviewerId, companyId, name: "Review agent", adapterType: "paperclip_runner", status: "running" }); + const [actingRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, nativeIssueId: seed.issueId, runtimeMode: "native", status: "succeeded", completionContractId: actingRun.completionContractId, createdAt: new Date(actingRun.createdAt.getTime() - 60_000) }); + await db.insert(nativeRunResults).values({ id: sourceResultId, companyId, issueId: seed.issueId, runId: sourceRunId, completionContractId: actingRun.completionContractId!, serverFingerprint: sourceResultId, schemaStatus: "accepted", resultJson: {}, canonicalSha256: sourceResultId }); + await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId: seed.issueId, runId: sourceRunId, contractId: actingRun.completionContractId!, resultId: sourceResultId, triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, policyVersion: "fixture", assessmentJson: {}, inputDigest: assessmentId }); + const [issue] = await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, seed.issueId)).returning(); + await db.insert(statusDecisions).values({ id: decisionId, companyId, issueId: seed.issueId, runId: sourceRunId, assessmentId, decisionVersion: 1, policyVersion: "fixture", fromStatus: "in_progress", toStatus: "in_review", reasonCode: "explicit_review", decisionJson: { projectedStatusVersion: issue.statusVersion }, decisionDigest: decisionId, applicationState: "applied" }); + await db.update(issues).set({ lastStatusDecisionId: state === "superseded" ? null : decisionId }).where(eq(issues.id, seed.issueId)); + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seed.issueId, kind: "request_confirmation", sourceRunId, addresseeAgentId: state === "unassigned" ? agentId : reviewerId, createdByAgentId: agentId, continuationPolicy: "wake_assignee", status: state === "resolved" ? "accepted" : "pending", ...(state === "resolved" ? { resolvedByRunId: seed.runId, resolvedByAgentId: reviewerId } : {}), payload: { version: 1, prompt: "Review the completed work.", target: { type: "custom", key: "native_completion_review", revisionId: decisionId } } }); + await db.update(heartbeatRuns).set({ agentId: reviewerId, contextSnapshot: { issueId: seed.issueId, nativeReviewInteractionId: interactionId, nativeReviewDecisionId: decisionId } }).where(eq(heartbeatRuns.id, seed.runId)); + if (state === "pending" || state === "resolved") { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" }); + expect((await db.select().from(issues).where(eq(issues.id, seed.issueId)))[0].assigneeAgentId).toBe(agentId); + } else { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 }); + } + }); + + it.each(([ + ["workspace_finalizing", "monitor"], ["workspace_finalizing", "approval"], + ["retryable_failure", "monitor"], ["retryable_failure", "approval"], + ["result_accepted", "monitor"], ["result_accepted", "approval"], + ] as const).flatMap(([phase, wait]) => (["operator", "automatic"] as const).map(mode => [phase, wait, mode] as const)))("commits the saved yield from %s preserving its %s wait via %s recovery", async (phase, wait, mode) => { + const seed = await seedAbandonedCopyback(); + await db.update(nativeRunFinalizations).set({ phase, + ...(phase === "retryable_failure" ? { attempt: 1, nextAttemptAt: new Date(Date.now() - 60_000), + failureCode: "native_workspace_sync_out_failed" } : {}), + }).where(eq(nativeRunFinalizations.runId, seed.runId)); + await db.update(heartbeatRuns).set({ nativePhase: phase }).where(eq(heartbeatRuns.id, seed.runId)); + const workspaceId = randomUUID(); + const cwd = path.join(workspaceRoot, seed.runId); + await fs.mkdir(cwd); + await fs.writeFile(path.join(cwd, "saved.txt"), "accepted work"); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, + mode: "local", strategyType: "local_directory", name: "Retained workspace", cwd }); + const nextCheckAt = new Date(Date.now() - 60_000); // Already due, like the staging runs. + const policy = { monitor: { nextCheckAt: nextCheckAt.toISOString(), notes: "Check the existing PR." } }; + await db.update(issues).set({ executionWorkspaceId: workspaceId, + ...(wait === "monitor" ? { executionPolicy: policy, monitorNextCheckAt: nextCheckAt } : {}), + }).where(eq(issues.id, seed.issueId)); + const interactionId = randomUUID(); + if (wait === "approval") await db.insert(issueThreadInteractions).values({ + id: interactionId, companyId, issueId: seed.issueId, sourceRunId: seed.runId, + createdByAgentId: agentId, kind: "request_confirmation", status: "pending", + payload: { version: 1, prompt: "Approve the plan?", target: { type: "issue_document", + issueId: seed.issueId, key: "plan", revisionId: randomUUID() } }, + }); + const resultJson = { + result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the existing continuation.", + completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [], + remainingWork: [{ description: "Wait for the check or plan confirmation.", blocksCompletion: true }] }, + continuation: { kind: wait === "monitor" ? "monitor" : "response_wake", + summary: "Wait for the existing continuation.", idempotencyKey: "saved-wait" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", + runTerminalState: "succeeded", reportedWorkDisposition: "yielded" }, + }; + await db.update(nativeRunResults).set({ resultJson }).where(eq(nativeRunResults.runId, seed.runId)); + const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId)); + + // The real reconciler must remain fenced before operator confirmation. + await reconcileNativeFinalizations(db, [seed.runId]); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe(phase); + const [beforeConfirmation] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (mode === "operator") { + await resumeNativeWorkspaceAfterOwnerStop(seed); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toEqual(beforeConfirmation); + await reconcileNativeFinalizations(db, [seed.runId]); + } else { + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceFinalizationOwner: { ...seed.owner, worker } } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: seed.owner.token, worker, stopped: true, iat: 1, exp: 61 }); + try { await reconcileNativeFinalizations(db, [seed.runId]); } finally { probe.mockRestore(); } + } + + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toMatchObject({ phase: "committed", resultId: accepted.id }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0]).toMatchObject({ status: "succeeded", nativePhase: "committed" }); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId))).toEqual([accepted]); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)))[0].status).toBe("resolved"); + const [issue] = await db.select().from(issues).where(eq(issues.id, seed.issueId)); + expect(issue.status).toBe(wait === "monitor" ? "in_progress" : "in_review"); + if (wait === "monitor") { + expect(issue.monitorNextCheckAt).toEqual(nextCheckAt); + expect(issue.executionPolicy).toEqual(policy); + } else { + expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].status).toBe("pending"); + } + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, seed.issueId))).toHaveLength(1); + expect(await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.payload}->>'issueId' = ${seed.issueId}`)).toHaveLength(0); + expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, seed.runId), + eq(workspaceOperations.phase, "workspace_finalize"), eq(workspaceOperations.status, "succeeded")))).toHaveLength(1); + expect(await fs.readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("accepted work"); + }); + + it.each(["token", "company", "result", "newer owner", "missing confirmation", "malformed owner"])("rejects stale or unverified copyback recovery: %s", async changed => { + const seed = await seedAbandonedCopyback(); + if (changed === "malformed owner") { + const owner = { token: seed.ownerToken }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId)); + } + if (changed === "token") seed.ownerToken = randomUUID(); + if (changed === "company") seed.companyId = foreignCompanyId; + if (changed === "result") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (changed === "newer owner") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { ...seed.owner, token: randomUUID() } } }).where(eq(heartbeatRuns.id, seed.runId)); + if (changed === "missing confirmation") Object.assign(seed, { controllerAndCopybackStopped: false }); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeDefined(); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(0); + }); + + it("rejects a locally live owner even after its advisory connection is gone", async () => { + const seed = await seedAbandonedCopyback(); + const identity = await currentNativeControllerIdentity(); + const owner = { ...seed.owner, hostname: os.hostname(), pid: identity.pid, + processStartedAt: identity.processStartedAt.toISOString(), controllerBootId: identity.bootId }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId)); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + }); + + it("does not reopen an old result after a newer task execution exists", async () => { + const seed = await seedAbandonedCopyback(); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", nativeIssueId: seed.issueId }); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("no longer current"); + }); + + it("rejects stop confirmation while another controller holds the physical lock", async () => { + const seed = await seedAbandonedCopyback(); + await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${companyId}:${seed.runId}`}, 0))`); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active"); + }); + }); + it("reuses a successful export even if an old controller wrote a later failed barrier", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Late failed barrier" }); const [success] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, @@ -311,6 +541,137 @@ describe("native workspace finalization recovery", () => { .toEqual({ acquired: true, value: "recovered" }); }); + it("automatically adopts a proven stopped deployment without another agent turn", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "fixture-project", environmentId: "fixture-env", serviceId: "fixture-service", deploymentId: "fixture-old" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ fingerprint: nativeSha256({ runId: seed.runId, owner }), evidence: { runId: seed.runId, owner, preserved: "recovery evidence" } }).where(eq(issueRecoveryActions.id, seed.actionId)); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + const proof = { v: 1 as const, iss: "paperclip-cloud" as const, aud: "paperclip-workspace-worker-stop/v1" as const, sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true as const, iat: 1, exp: 61 }; + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(proof); + try { + const work = vi.fn(async () => { + const [recovery] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "resuming" }); + expect(recovery.evidence?.preserved).toBe("recovery evidence"); + return "saved workspace copied"; + }); + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: true, value: "saved workspace copied" }); + expect(probe).toHaveBeenCalledWith({ runId: seed.runId, ownerToken: owner.token, worker }); + expect(work).toHaveBeenCalledOnce(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined(); + expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toEqual(proof); + expect(run.runnerProfileJson?.preserved).toBe("workspace descriptor"); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(after).toEqual(before); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0); + } finally { probe.mockRestore(); } + }); + + it("retains the fence for unknown platform evidence and owner changes during verification", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(null); + const work = vi.fn(); + try { + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false }); + const [recovery] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, seed.issueId), eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: seed.runId, owner })))); + expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "operator_required" }); + const replacement = { ...owner, token: randomUUID() }; + probe.mockImplementation(async () => { + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: replacement } }).where(eq(heartbeatRuns.id, seed.runId)); + return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }; + }); + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false }); + expect(work).not.toHaveBeenCalled(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(replacement); + } finally { probe.mockRestore(); } + }); + + it.each(["done", "cancelled", "execution replaced", "checkout replaced", "newer run", "queued legacy run", "reassigned", "result unbound", "result rejected", "run stopped"])("does not copy saved files after the task moves on during platform verification: %s", async changed => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const file = path.join(workspaceRoot, `${seed.runId}.txt`); + await fs.writeFile(file, "new task work"); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockImplementation(async () => { + if (changed === "done" || changed === "cancelled") await db.update(issues).set({ status: changed }).where(eq(issues.id, seed.issueId)); + if (changed === "execution replaced") await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, seed.issueId)); + if (changed === "checkout replaced") { + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded" }).returning(); + await db.update(issues).set({ checkoutRunId: other.id }).where(eq(issues.id, seed.issueId)); + } + if (changed === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, seed.issueId)); + if (changed === "result unbound") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (changed === "result rejected") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.runId, seed.runId)); + if (changed === "run stopped") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seed.runId)); + if (changed === "newer run" || changed === "queued legacy run") { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.insert(heartbeatRuns).values({ companyId, agentId, + runtimeMode: changed === "newer run" ? "native" : "legacy", + nativeIssueId: changed === "newer run" ? seed.issueId : null, + contextSnapshot: { issueId: seed.issueId }, status: changed === "newer run" ? "succeeded" : "queued", + createdAt: new Date(run.createdAt.getTime() + (changed === "newer run" ? 60_000 : -60_000)), + }); + } + return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }; + }); + const copyback = vi.fn(async () => fs.writeFile(file, "old sandbox work")); + try { + expect(await withNativeWorkspaceFinalizationOwnership(seed, copyback)).toEqual({ acquired: false }); + expect(copyback).not.toHaveBeenCalled(); + expect(await fs.readFile(file, "utf8")).toBe("new task work"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toBeUndefined(); + } finally { probe.mockRestore(); } + }); + + it("rechecks automatic task ownership at the copyback publication boundary", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }); + const write = vi.fn(); + try { + await expect(withNativeWorkspaceFinalizationOwnership(seed, async ownership => { + await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, seed.issueId)); + await ownership.assertHeld(); + write(); + })).rejects.toThrow("native_workspace_finalization_ownership_lost"); + expect(write).not.toHaveBeenCalled(); + } finally { probe.mockRestore(); } + }); + + it.each(["closed", "superseded"])("does not start a workspace operation when the real reconciler selects a %s task", async changed => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + const workspaceId = randomUUID(), cwd = path.join(workspaceRoot, seed.runId); + await fs.mkdir(cwd); + await fs.writeFile(path.join(cwd, "current.txt"), "new task work"); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, mode: "local", strategyType: "local_directory", name: "Current workspace", cwd }); + await db.update(issues).set({ executionWorkspaceId: workspaceId, + ...(changed === "closed" ? { status: "done" } : { executionRunId: null }), + }).where(eq(issues.id, seed.issueId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }); + try { + await reconcileNativeFinalizations(db, [seed.runId]); + expect(probe).toHaveBeenCalledOnce(); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seed.runId))).toHaveLength(0); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe("workspace_finalizing"); + expect(await fs.readFile(path.join(cwd, "current.txt"), "utf8")).toBe("new task work"); + } finally { probe.mockRestore(); } + }); + it("surfaces an unverifiable foreign controller as board recovery without physical writes", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Foreign copyback controller" }); const owner = { token: randomUUID(), hostname: "other-controller.invalid", pid: 42, processStartedAt: new Date().toISOString() }; diff --git a/server/src/services/native-runtime/native-workspace-owner-recovery.ts b/server/src/services/native-runtime/native-workspace-owner-recovery.ts new file mode 100644 index 00000000000..4b3e19e3311 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-owner-recovery.ts @@ -0,0 +1,87 @@ +import os from "node:os"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { resumeWorkspaceFinalizationSchema } from "@paperclipai/shared"; +import { conflict } from "../../errors.js"; +import { persistActivity, publishActivity } from "../activity-log.js"; +import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js"; +import { evaluateNativeControllerTakeover } from "./native-restart-recovery.js"; + +const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; +const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +const changed = () => conflict("The recorded workspace owner is no longer current. Refresh the task and inspect its run.", { code: "workspace_owner_recovery_stale" }); + +/** Board-only admission after deployment-platform verification. A missing DB + * lock or an expired lease alone never proves physical copyback has stopped. */ +export async function resumeNativeWorkspaceAfterOwnerStop(input: { + db: Db; companyId: string; issueId: string; actionId: string; runId: string; + ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string; actorId: string; +}) { + const confirmation = resumeWorkspaceFinalizationSchema.parse({ + actionId: input.actionId, runId: input.runId, ownerToken: input.ownerToken, + controllerAndCopybackStopped: input.controllerAndCopybackStopped, stopEvidence: input.stopEvidence, + }); + const admitted = await input.db.transaction(async tx => { + // Share the physical copyback lock. Do not wait behind an active exporter + // and then apply a confirmation that described an earlier owner. + const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`); + if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); + // Match the status arbiter's coordinator -> issue -> run lock order. + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); + const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.id, input.runId))).for("update"); + const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update"); + if (!issue || !run || !coordinator || !action || action.evidence?.runId !== run.id + || action.cause !== "native_workspace_finalization_owner_unverified" || action.ownerType !== "board" + || action.kind !== "active_run_watchdog" || !["active", "escalated"].includes(action.status) + || (coordinator.leaseOwner && (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt.getTime() > Date.now()))) throw changed(); + const result = await currentNativeWorkspaceRecoveryResult(tx as unknown as Db, { + companyId: input.companyId, issue, run, coordinator, + }); + if (!result) throw changed(); + const prior = record(run.runnerProfileJson?.[OWNER_KEY]); + const recordedOwner = record(action.evidence?.owner); + const stop = record(action.evidence?.workspaceOwnerStop); + const receipt = { runId: run.id, resultId: result.id, status: "queued" as const }; + if (run.runnerProfileJson?.[OWNER_KEY] == null && record(stop.owner).token === input.ownerToken + && stop.resultId === result.id) return { receipt, publication: null }; + if (typeof prior.hostname !== "string" || !prior.hostname + || !Number.isInteger(prior.pid) || Number(prior.pid) <= 0 + || typeof prior.processStartedAt !== "string" || !Number.isFinite(Date.parse(prior.processStartedAt)) + || prior.token !== input.ownerToken || recordedOwner.token !== input.ownerToken + || prior.hostname !== recordedOwner.hostname || prior.pid !== recordedOwner.pid + || prior.processStartedAt !== recordedOwner.processStartedAt + || prior.controllerBootId !== recordedOwner.controllerBootId) throw changed(); + // Reject a confirmation contradicted by a locally observable live owner, + // even if that owner's advisory connection has already disconnected. + if (prior.hostname === os.hostname()) { + const takeover = await evaluateNativeControllerTakeover({ owner: { + leaseOwner: input.ownerToken, leaseExpiresAt: new Date(0), + controllerPid: typeof prior.pid === "number" ? prior.pid : null, + controllerProcessStartedAt: typeof prior.processStartedAt === "string" ? new Date(prior.processStartedAt) : null, + }, now: new Date() }); + if (!takeover.allowed) throw conflict("The previous workspace controller is still active or its stop cannot be verified."); + } + const now = new Date(); + const proof = { owner: prior, resultId: result.id, actorId: input.actorId, + stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() }; + await tx.update(heartbeatRuns).set({ + runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now, + }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id), + sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${input.ownerToken}`)); + await tx.update(issueRecoveryActions).set({ evidence: { ...action.evidence, workspaceOwnerStop: proof }, + nextAction: "Workspace finalization is queued for the saved result. The agent will not repeat its work.", + wakePolicy: null, monitorPolicy: null, updatedAt: now, + }).where(eq(issueRecoveryActions.id, action.id)); + const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId, + actorType: "user", actorId: input.actorId, action: "issue.workspace_owner_stop_confirmed", + entityType: "issue", entityId: input.issueId, runId: input.runId, + details: { recoveryActionId: input.actionId, ...proof }, + }); + // Leave the result, coordinator, source lease, task status and wait paths + // untouched. The normal reconciler resumes copyback and arbitrates it. + return { receipt, publication }; + }); + if (admitted.publication) publishActivity(admitted.publication.publication); + return admitted.receipt; +} diff --git a/server/src/services/native-runtime/native-workspace-platform-proof.test.ts b/server/src/services/native-runtime/native-workspace-platform-proof.test.ts new file mode 100644 index 00000000000..fb3297ca80b --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-platform-proof.test.ts @@ -0,0 +1,34 @@ +import { generateKeyPairSync, sign } from "node:crypto"; +import { describe, expect, it, vi } from "vitest"; +import { verifyWorkspaceWorkerStopReceipt, requestWorkspaceWorkerStopProof, currentWorkspaceWorker } from "./native-workspace-platform-proof.js"; +const { privateKey, publicKey } = generateKeyPairSync("ed25519"); +const env = { PAPERCLIP_CLOUD_STACK_ID: "stack", PAPERCLIP_CLOUD_API_ORIGIN: "https://cloud.example", PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "fixture-token", PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [{ ...publicKey.export({ format: "jwk" }), kid: "test", alg: "EdDSA", use: "sig" }] }) }; +const worker = { provider: "railway" as const, projectId: "project", environmentId: "environment", serviceId: "service", deploymentId: "old-deployment" }; +const binding = { runId: "run", ownerToken: "owner", worker }; +const now = new Date("2099-01-01T00:00:00Z"); +function receipt(overrides = {}) { + const header = Buffer.from(JSON.stringify({ alg: "EdDSA", kid: "test", typ: "paperclip-workspace-worker-stop+jwt" })).toString("base64url"); + const payload = Buffer.from(JSON.stringify({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", ...binding, stopped: true, iat: now.getTime()/1000, exp: now.getTime()/1000+60, ...overrides })).toString("base64url"); + return `${header}.${payload}.${sign(null, Buffer.from(`${header}.${payload}`), privateKey).toString("base64url")}`; +} +describe("workspace platform stop proof", () => { + it("accepts only signed proof for the exact stack, owner, run and full stopped deployment", () => { + expect(verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, now)).toMatchObject({ stopped: true, worker }); + for (const invalid of [{ sub: "other-stack" }, { runId: "other-run" }, { ownerToken: "other-owner" }, { stopped: false }, { exp: now.getTime()/1000-1 }, { worker: { ...worker, deploymentId: "other" } }, { aud: "paperclip-cloud-control/v1" }]) { + expect(verifyWorkspaceWorkerStopReceipt(receipt(invalid), binding, env, now)).toBeNull(); + } + expect(verifyWorkspaceWorkerStopReceipt(receipt().slice(0,-5)+"wrong", binding, env, now)).toBeNull(); + }); + it("fails closed on live, unknown, unreachable and unsigned platform responses", async () => { + for (const response of [{ stopped: false }, { stopped: true }, { receipt: "forged" }]) { + const fetchImpl = vi.fn().mockResolvedValue(Response.json(response)); + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl })).toBeNull(); + } + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockRejectedValue(new Error("offline")) })).toBeNull(); + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockResolvedValue(Response.json({ receipt: receipt() })) })).toMatchObject({ stopped: true }); + }); + it("records a complete platform identity, and does not guess legacy ownership", () => { + expect(currentWorkspaceWorker({ RAILWAY_PROJECT_ID: "project", RAILWAY_ENVIRONMENT_ID: "environment", RAILWAY_SERVICE_ID: "service", RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toEqual(worker); + expect(currentWorkspaceWorker({ RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toBeUndefined(); + }); +}); diff --git a/server/src/services/native-runtime/native-workspace-platform-proof.ts b/server/src/services/native-runtime/native-workspace-platform-proof.ts new file mode 100644 index 00000000000..04b452d1d76 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-platform-proof.ts @@ -0,0 +1,65 @@ +import { verify } from "node:crypto"; +import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js"; +import { getCloudStackContext } from "../cloud-instance.js"; + +/** Deployment identity covers the controller and every orphaned tar/Git child. */ +export type WorkspaceWorker = { + provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string; +}; +export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker }; +export type WorkspaceWorkerStopProof = WorkspaceStopBinding & { + v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string; + stopped: true; iat: number; exp: number; +}; +function validWorker(worker: unknown): worker is WorkspaceWorker { + if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false; + const value = worker as Record; + return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key => + typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200); +} +export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined { + const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID]; + if (ids.some(id => !id?.trim())) return undefined; + return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! }; +} +export function verifyWorkspaceWorkerStopReceipt( + receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(), +): WorkspaceWorkerStopProof | null { + try { + if (receipt.length > 16_000 || !validWorker(binding.worker)) return null; + const parts = receipt.split("."); + if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null; + const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8")); + if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null; + if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null; + const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof; + const seconds = Math.floor(now.getTime()/1000); + if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1" + || !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true + || proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken + || !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30 + || proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60 + || !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null; + return proof; + } catch { return null; } +} +/** Unsupported platforms, legacy receipts and unavailable evidence never authorize takeover. */ +export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: { + env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date; +} = {}): Promise { + const env = options.env ?? process.env; + const context = getCloudStackContext(env); + const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim(); + if (!context?.stackId || !context.cloudOrigin || !token || !validWorker(binding.worker)) return null; + try { + const origin = new URL(context.cloudOrigin); + if (origin.protocol !== "https:" || origin.username || origin.password) return null; + const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, { + method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" }, + body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) return null; + const payload = await response.json() as { receipt?: unknown }; + return typeof payload.receipt === "string" ? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null; + } catch { return null; } +} diff --git a/server/src/services/native-runtime/native-workspace-recovery-binding.ts b/server/src/services/native-runtime/native-workspace-recovery-binding.ts new file mode 100644 index 00000000000..20c4ee6861f --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-recovery-binding.ts @@ -0,0 +1,36 @@ +import { and, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { getNativeReviewAssignment } from "./native-review-participant.js"; + +/** Both operator and platform recovery must still own the task's accepted work. + * Call with coordinator -> issue -> run rows locked in a short transaction. */ +export async function currentNativeWorkspaceRecoveryResult(db: Db, input: { + companyId: string; + issue: typeof issues.$inferSelect; + run: typeof heartbeatRuns.$inferSelect; + coordinator: typeof nativeRunFinalizations.$inferSelect; +}) { + const { companyId, issue, run, coordinator } = input; + if (issue.companyId !== companyId || run.companyId !== companyId || coordinator.companyId !== companyId + || run.nativeIssueId !== issue.id || coordinator.issueId !== issue.id || coordinator.runId !== run.id + || run.runtimeMode !== "native" || run.status !== "running" + || !["result_accepted", "workspace_finalizing", "retryable_failure"].includes(coordinator.phase) + || !coordinator.resultId || ["done", "cancelled"].includes(issue.status) + || issue.executionRunId !== run.id || (issue.checkoutRunId && issue.checkoutRunId !== run.id)) return null; + if (issue.assigneeAgentId !== run.agentId && !await getNativeReviewAssignment(db, { + companyId, issueId: issue.id, agentId: run.agentId, + contextSnapshot: run.contextSnapshot, actingRunId: run.id, issueExecutionRunId: issue.executionRunId, + allowResolvedByRunId: run.id, + })) return null; + const [result] = await db.select().from(nativeRunResults).where(and( + eq(nativeRunResults.id, coordinator.resultId), eq(nativeRunResults.companyId, companyId), + eq(nativeRunResults.issueId, issue.id), eq(nativeRunResults.runId, run.id), eq(nativeRunResults.schemaStatus, "accepted"), + )); + if (!result || result.completionContractId !== run.completionContractId) return null; + const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, run.id), + or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), + or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])), + )).limit(1); + return newer ? null : result; +} diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts index 314064ea27a..f3a56c5ef63 100644 --- a/ui/src/api/issues.ts +++ b/ui/src/api/issues.ts @@ -283,6 +283,8 @@ export const issuesApi = { `/issues/${id}/stalled-review-decision`, data, ), + resumeWorkspaceFinalization: (id: string, data: { actionId: string; runId: string; ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string }) => + api.post<{ runId: string; resultId: string; status: "queued" }>(`/issues/${id}/recovery-actions/resume-workspace-finalization`, data), retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) => api.post<{ runId: string; resultId: string; leaseId: string; status: "queued" }>(`/issues/${id}/recovery-actions/retry-workspace-export`, data), resolveRecoveryAction: ( diff --git a/ui/src/components/IssueRecoveryActionCard.test.tsx b/ui/src/components/IssueRecoveryActionCard.test.tsx index 88e6cc7b133..cd513226984 100644 --- a/ui/src/components/IssueRecoveryActionCard.test.tsx +++ b/ui/src/components/IssueRecoveryActionCard.test.tsx @@ -1161,3 +1161,10 @@ it("requires export retry for ordinary restoration while keeping explicit board expect(document.body.textContent).not.toContain("Send for review"); expect(document.body.textContent).toContain("False positive, done"); }); + +it("requires owner-stop recovery even for board false-positive dispositions", () => { + const node = render( {}} canFalsePositive />); + expect(node.querySelector("[data-testid='recovery-action-resolve-trigger']")).toBeNull(); + expect(node.textContent).not.toContain("False positive"); +}); diff --git a/ui/src/components/IssueRecoveryActionCard.tsx b/ui/src/components/IssueRecoveryActionCard.tsx index beb84076e1b..e61adbbb966 100644 --- a/ui/src/components/IssueRecoveryActionCard.tsx +++ b/ui/src/components/IssueRecoveryActionCard.tsx @@ -1067,6 +1067,7 @@ export function IssueRecoveryActionCard({ } satisfies Record)[cardState]; const visibleResolveOptions = RESOLVE_OPTIONS.filter((option) => { + if (action.cause === "native_workspace_finalization_owner_unverified") return false; if (isNativeWorkspaceExportRepairCause(action.cause) && ["todo", "done", "in_review"].includes(option.outcome)) return false; if (option.outcome === "todo" && requiresExecutionReconciliation(action.cause)) return false; if (option.boardOnly && !canFalsePositive) return false; diff --git a/ui/src/components/WorkspaceExportRecovery.test.tsx b/ui/src/components/WorkspaceExportRecovery.test.tsx index 10f4c243eac..28a2409d2f1 100644 --- a/ui/src/components/WorkspaceExportRecovery.test.tsx +++ b/ui/src/components/WorkspaceExportRecovery.test.tsx @@ -1,72 +1,44 @@ // @vitest-environment jsdom import { act } from "react"; import { createRoot, type Root } from "react-dom/client"; -import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { IssueRecoveryAction } from "@paperclipai/shared"; import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery"; -const retry = vi.hoisted(() => vi.fn()); -vi.mock("../api/issues", () => ({ issuesApi: { retryWorkspaceExport: retry } })); -const action = { id: "action", cause: "native_workspace_sync_out_retry_exhausted", status: "active", updatedAt: "2026-01-01T00:00:00Z", ownerType: "board", evidence: { runId: "run" }, wakePolicy: null } as unknown as IssueRecoveryAction; +const action = { cause: "native_workspace_finalization_owner_unverified", status: "escalated", ownerType: "board", evidence: { runId: "run", owner: { token: "owner", hostname: "secret-host", pid: 2 } }, wakePolicy: null } as unknown as IssueRecoveryAction; describe("WorkspaceExportRecovery", () => { - let root: Root, container: HTMLDivElement, client: QueryClient; - const onQueued = vi.fn(); - beforeEach(() => { - retry.mockReset().mockResolvedValue({ status: "queued" }); onQueued.mockReset(); - container = document.createElement("div"); document.body.append(container); root = createRoot(container); - client = new QueryClient({ defaultOptions: { mutations: { retry: false } } }); - }); - afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); }); - async function mount(overrides: Partial[0]> = {}) { - await act(async () => root.render()); + let root: Root, container: HTMLDivElement; + beforeEach(() => { container = document.createElement("div"); document.body.append(container); root = createRoot(container); }); + afterEach(async () => { await act(async () => root.unmount()); container.remove(); }); + async function mount(value: IssueRecoveryAction | null) { + await act(async () => root.render()); + expect(container.querySelector("textarea, button, details, input")).toBeNull(); + expect(container.textContent).not.toContain("secret-host"); } - async function enterNote() { - const textarea = container.querySelector("textarea")!; - await act(async () => { - Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(textarea, "Restored provider connectivity while preserving all saved files."); - textarea.dispatchEvent(new Event("input", { bubbles: true })); - }); - } - async function submit() { - await act(async () => container.querySelector("button")!.click()); - await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); }); - } - it("requires a repair note and submits only the exact recorded run", async () => { - await mount(); expect(container.querySelector("button")!.disabled).toBe(true); - await enterNote(); await submit(); - expect(retry).toHaveBeenCalledWith("issue", { actionId: "action", runId: "run", repairNote: "Restored provider connectivity while preserving all saved files." }); - expect(onQueued).toHaveBeenCalledOnce(); - expect(container.querySelector('[role="status"]')?.textContent).toContain("agent will not repeat its work"); + it("shows a quiet status while the harness restores work", async () => { + await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerRecovery: { state: "resuming" } } }); + expect(container.querySelector('[role="status"]')?.textContent).toBe("Restoring saved work…"); }); - it("offers repair again when a queued export publishes a new permanent failure", async () => { - await mount(); await enterNote(); await submit(); - expect(container.querySelector("button")).toBeNull(); - await mount({ action: { ...action, updatedAt: "2026-01-01T00:01:00Z", wakePolicy: null } }); - expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); + it("does not mistake platform identity for confirmed recovery", async () => { + await mount({ ...action, evidence: { ...action.evidence, owner: { ...(action.evidence.owner as object), worker: { provider: "railway", deploymentId: "old" } } } }); + expect(container.textContent).toContain("operator assistance"); }); - it("keeps unavailable-sandbox errors actionable inline", async () => { - retry.mockRejectedValueOnce(new Error("Resume the retained sandbox before retrying.")); await mount(); - await enterNote(); await submit(); - expect(container.querySelector('[role="alert"]')?.textContent).toContain("Resume the retained sandbox"); + it("states that an operator is needed for legacy receipts", async () => { + await mount(action); + expect(container.textContent).toContain("Workspace recovery needs operator assistance"); }); - it("does not offer the action without runtime access", async () => { - await mount({ canManage: false }); expect(container.querySelector("button")).toBeNull(); + it("keeps export repair out of the task", async () => { + const cause = "native_workspace_sync_out_retry_exhausted"; + await mount({ ...action, cause }); + expect(container.textContent).toContain("operator assistance"); }); - it("offers saved-result export after transient retries are exhausted without claiming an unsafe link", async () => { - await mount({ action: { ...action, cause: "native_workspace_sync_out_retry_exhausted" } }); - expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); - expect(container.textContent).toContain("export failure"); - expect(container.textContent).not.toContain("unsafe link"); - await enterNote(); await submit(); - expect(retry).toHaveBeenCalledWith("issue", expect.objectContaining({ runId: "run" })); + it("removes the status when recovery resolves, including updates in place", async () => { + await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerStop: {} } }); + expect(container.textContent).toBe("Restoring saved work…"); + await mount({ ...action, status: "resolved" }); expect(container.textContent).toBe(""); + await mount(null); expect(container.textContent).toBe(""); }); - it("never asks users to repair historical unsafe exports", async () => { - await mount({ action: { ...action, cause: "native_workspace_sync_out_unsafe_archive" } }); - expect(container.textContent).toBe(""); - expect(retry).not.toHaveBeenCalled(); - }); - it("does not offer a second retry while the same export is queued", async () => { - await mount({ action: { ...action, wakePolicy: { kind: "resume_native_run" } } }); - expect(container.querySelector('[role="status"]')?.textContent).toContain("Export is queued"); expect(container.querySelector("button")).toBeNull(); + it("does not show unrelated or historical unsafe-export notices", async () => { + await mount({ ...action, cause: "no_live_path", evidence: {} }); expect(container.textContent).toBe(""); + await mount({ ...action, cause: "native_workspace_sync_out_unsafe_archive" }); expect(container.textContent).toBe(""); }); }); diff --git a/ui/src/components/WorkspaceExportRecovery.tsx b/ui/src/components/WorkspaceExportRecovery.tsx index 98376816817..a5b3dcb45e8 100644 --- a/ui/src/components/WorkspaceExportRecovery.tsx +++ b/ui/src/components/WorkspaceExportRecovery.tsx @@ -1,45 +1,17 @@ -import { useId, useState } from "react"; -import { useMutation } from "@tanstack/react-query"; import type { IssueRecoveryAction } from "@paperclipai/shared"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; -import { issuesApi } from "../api/issues"; -import { cn } from "../lib/utils"; -import { Button } from "./ui/button"; -import { Label } from "./ui/label"; -import { Textarea } from "./ui/textarea"; +import { asObject, cn } from "../lib/utils"; -/** Copyback repair preserves the accepted result and never wakes the provider. */ -export function WorkspaceExportRecovery({ issueId, action, canManage, onQueued, className }: { - className?: string; - issueId: string; action: IssueRecoveryAction | null; canManage: boolean; onQueued: () => void; +/** Infrastructure repair belongs to the harness/operator, never a task form. */ +export function WorkspaceExportRecovery({ action, className }: { + className?: string; action: IssueRecoveryAction | null; }) { - const noteId = useId(); - const [repairNote, setRepairNote] = useState(""); - const [queuedActionVersion, setQueuedActionVersion] = useState(null); - const retry = useMutation({ - mutationFn: () => issuesApi.retryWorkspaceExport(issueId, { - actionId: action!.id, runId: action!.evidence.runId as string, repairNote: repairNote.trim(), - }), - onSuccess: () => { setQueuedActionVersion(String(action!.updatedAt)); onQueued(); }, - }); - if (!action || !isNativeWorkspaceExportRepairCause(action.cause) || action.ownerType !== "board" + const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified"; + if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board" || !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null; - const queued = queuedActionVersion === String(action.updatedAt) || action.wakePolicy?.kind === "resume_native_run"; - return
-

Workspace export needs repair

- {queued ?

Export is queued for the saved result. The agent will not repeat its work.

: <> -

{"Automatic workspace export retries stopped. Inspect the export failure, restore provider or destination availability, and preserve the saved files in the retained sandbox. Retry export here when the cause is resolved."}

- {canManage ? <> - -