diff --git a/doc/native-workspace-finalization-recovery.md b/doc/native-workspace-finalization-recovery.md index 01201315417..a9a6e6365f1 100644 --- a/doc/native-workspace-finalization-recovery.md +++ b/doc/native-workspace-finalization-recovery.md @@ -35,6 +35,46 @@ leave tar/Git children alive, so incomplete copyback requires operator stop verification even on the same host. PID reuse is checked against its recorded start time rather than trusted by PID alone. +## Automatic cloud recovery + +New Railway copyback receipts record the exact project, environment, service and +deployment ID. On takeover, the existing finalization reconciler asks the cloud +harness to inspect that deployment. The harness resolves the stack's owned +resources independently and issues a short-lived Ed25519 receipt only when the +exact deployment is `REMOVED` with `deploymentStopped: true`. This covers all +processes in the deployment, including orphaned copyback children. Missing, +truncated, live or unavailable provider evidence leaves the physical fence intact. + +The tenant verifies the configured harness key, disjoint receipt audience, stack, +run, owner token, deployment identity and expiry. It records proof and an activity +entry when replacing only the exact owner under the existing advisory lock. The +normal reconciler resumes the accepted result; it does not create a provider turn +or change approval/monitor waits. A changed owner during verification fails closed. + +Stop proof alone does not authorize old work to overwrite a current task. After +the provider read, automatic recovery uses the operator path's shared binding +checks under short coordinator, issue and run row locks, atomically with the +owner claim. The task must still be open, assigned to the run or its authoritative +review participant, and bound to that running native execution and accepted +result. A different checkout, newer execution or queued/running legacy execution +prevents takeover. Publication checks repeat this validation; row locks are never +held across physical copyback I/O. + +The harness endpoint is `POST /v1/tenant/workspace-worker-stop`. The fleet-wide +server token authorizes a read request, never stop evidence: responses must be +signed by the harness. This endpoint requires the companion cloud harness change. +Deploying only the application keeps unknown owners fenced. No cloud provider +credentials are delivered to tenant applications. + +The task view shows only “Restoring saved work…” after the server confirms recovery +or an operator queues it, +then removes the notice when commitment resolves recovery. It exposes no process +identifiers, shutdown form or manual resume control. Legacy receipts without +platform identity and unsupported platforms say operator assistance is needed; +operators use the documented exact-owner API below after independent stop proof. +The three diagnosed historical receipts do not gain a deployment identity merely +by upgrading. Never infer one from elapsed time or a hostname. + ## Unverified copyback after controller replacement Generic orphan recovery does not turn a missing provider PID into `process_lost` @@ -66,6 +106,48 @@ exact prior controller **and its copyback subprocesses** have stopped. Retain th sandbox, accepted native result, and workspace descriptor. Do not run a new provider turn, delete workspace contents, or relax archive confinement. +For legacy receipts, an authorized company owner, company administrator or +instance administrator records independently verified platform evidence through +the operator API. The task has no shutdown form or resume control. The API is +`POST /api/issues/:id/recovery-actions/resume-workspace-finalization` with: + +```json +{ + "actionId": "", + "runId": "", + "ownerToken": "", + "controllerAndCopybackStopped": true, + "stopEvidence": "" +} +``` + +This is an operator attestation, not an automatic deployment-platform probe. +The separate `runtime:confirm_owner_stop` authorization action limits this +attestation to active company owners or administrators, the trusted local board, +or an instance administrator. Ordinary runtime managers and agents cannot release +the physical owner receipt. Missing role data keeps the client notice read-only. +A +missing advisory connection alone is insufficient. Admission rejects an active +copyback lock, an observably live local controller, a changed owner, a newer run, +or an unbound result. It records the exact owner and operator evidence atomically +with releasing the receipt. The accepted result, workspace descriptor, source +lease, task status, monitors, and approval waits are unchanged. The ordinary +reconciler performs copyback and applies the saved result through the status +arbiter. No provider wake is enqueued. The recovery action remains visible until +finalization commits. Admission supports saved-result copyback from `result_accepted`, +`workspace_finalizing`, and `retryable_failure`; an abandoned retry does not need +a replacement run. The existing retry schedule remains unchanged. +Repeating the same confirmation before reconciliation is +idempotent; it cannot release a newer owner. Generic recovery resolution, including +false-positive dispositions, cannot dismiss this hold or bypass saved-result +finalization. + +Repeated sweeps preserve these copyback-specific instructions, including for +older actions whose recovery budget was already exhausted. They must never +recommend starting a replacement provider run for this hold. + +For older servers without this action, the maintenance fallback follows. + After that platform verification, use a database maintenance transaction to release only the exact receipt shown by the recovery action. Replace the four placeholders with the action's company, run, token, and source issue. The advisory @@ -138,8 +220,8 @@ A board member with runtime management access can complete the saved result: the export failure while preserving all user files. Restore transport access or other provider prerequisites before retrying. Extending the sandbox retention window can be necessary during this repair. -3. In the task's **Workspace export needs repair** notice, describe the repair and - choose **Retry workspace export**. The equivalent board API is +3. Use the operator API to record the repair and request export-only retry. The + task shows operator assistance is needed, without an infrastructure form. The board API is `POST /api/issues/:id/recovery-actions/retry-workspace-export` with `{ "actionId": "", "runId": "", "repairNote": "" }`. 4. Confirm the same run commits, the saved result determines the task outcome, diff --git a/doc/observability.md b/doc/observability.md index 4a331900469..626e84dced1 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -161,6 +161,24 @@ This document also holds three local instrumentation contracts: native runner traces, sandbox startup traces, and sandbox duplex transport instrumentation. Those sections follow below. +## Controller preparation phases + +With an operator OTLP endpoint, `paperclip.heartbeat-preparation` emits active +`heartbeat.prepare.` spans around environment selection, execution +configuration, session preparation, workspace preparation, GitHub environment +and launcher preparation, run output initialization, workspace validation, +runtime services, instruction files, and runtime selection. Database and HTTP +instrumentation can attach child spans during each awaited operation. + +These spans exist before provider dispatch, including setup failures. Each has +only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal +characters of the run UUID's SHA-256 hash. Failed operations set error status +without recording exception messages, paths, inputs, or returned content. +Search by that hash without a root-name filter to include these preparation +traces; the provider lifecycle trace is created later and can have a different +trace ID. Without an endpoint, tracing remains a no-op. These spans add no +first-party Telemetry events or database run-log writes. + ## Native Runner Trace Spans Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is diff --git a/doc/plans/2026-10-06-default-agent-runner.md b/doc/plans/2026-10-06-default-agent-runner.md index dfdfffa5ea7..5356284579c 100644 --- a/doc/plans/2026-10-06-default-agent-runner.md +++ b/doc/plans/2026-10-06-default-agent-runner.md @@ -1660,6 +1660,12 @@ The sanitized report is uploaded and independently download-verified as attachme The human requested that #15683 be finished and merged normally. A separate incremental $100 ceiling and as many bounded messages as needed are explicitly approved. The previous four-message/$25 campaign and original$250 ledger retain all unknown-charge and cleanup holds; they are not refunded or reassigned. Reuse local3136 and owned staging stack7998; preserve their data and receipts. Root owns actual-master reconciliation, exact-head checks/review, ordinary Mac/managed onboarding, checked task/follow-up and normal protected merge. Claude owner retains dependent branches and separate allowance. No production deployment or provisioning-default promotion is authorized. No timer is recreated. +Current-master integration includes `833769ab557255b11dd8f650f43ed7333653f5e4`, then `e3029c632b26b68595adc1a704a387833b3cefcb`. The ordinary merge retains the published Codex history and prerequisite squash history. Its audited logical source delta preserves master fixes, the prepared managed Connect qualification, fresh scoped discovery, and existing first-task eval assertions. The remaining merge conflicts concern Storybook indexes and dated reviews, not execution code. Independent review of both source compositions found no concrete P1/P2 regression; source review does not establish live acceptance. + +On the integrated checkout, two focused Codex runs pass 261 tests each across onboarding, runner selection, setup, editing, imports and readiness. Server and UI direct TypeScript checks pass after JavaScript-only shared/DB/adapter-utils/SDK preparation. All token gates pass. New master recovery controls initially report a missing declared trace test dependency and a sandbox-denied PostgreSQL listener; the four unaffected suites pass 74 cases. Isolated installation of the exact declared trace dependencies and an authorized temporary listener allow the two affected suites to pass all 60 cases. No tests are disabled, no lockfile or production policy is changed, and no local native/Docker build occurs. Full canonical workspace checks remain assigned to fresh hosted CI. + +Fresh-source remote package qualification, ordinary Apple Silicon onboarding and immutable managed staging onboarding remain pending. The previous canary60bc and stagingd67 receipts stay attributed to those sources. The new budget record reserves resources60/provider30/cleanup10 dollars and retains all prior unknown holds. The supported private Cloud skill and CLI are read from exact private master `4f3a63ad6f3f84c87e0e129da35909d8ccc7ffb0`; a deliberate pre-merge Core preview is separate from actual private Cloud distribution. Freeze the final reviewed product source before building; deploy only the owned pinned QA stack, with `set_default=false`. + Current published Codex head1a0ca662 is draft with green historical checks. Actual master833769ab is newer than merged prerequisite d67 and release repair60bc. Prepared private onboarding/Storybook work will be reconciled by source, without claiming its private/historical checks as final qualification. Native Connect correction is already in the Codex slice. Final managed defaults require the saved immutable OCI environment probe owned by ordinary Connect. Existing suites/fixtures/assertions/reporting are reused; no duplicate grader or silent Legacy fallback is added. All heavy builds stay remote, Codex payloads remain external official npm dependencies, and saved agents retain runners. Next action: preserve/commit current evidence, integrate actual master plus reviewed prepared corrections, run targeted checks, freeze/push candidate, require fresh fullCI/review, then qualify ordinary creation/onboarding on Mac and the one staging target before protected squash merge. diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index 1b90fa083d4..c7cc042f969 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -129,23 +129,40 @@ Review: `baseline`. API: `present` — representative operation IDs: - `POST /api/issues/{id}/recovery-actions/retry-workspace-export`. +- `POST /api/issues/{id}/recovery-actions/resume-workspace-finalization`. - `POST /api/execution-workspaces/{id}/reconcile-branch`. Agent access: `restricted` — Board/runtime management APIs; agent reads diagnosis and performs permitted workspace work. -Policy: Task/run visibility, run ownership and source/action policy; workspace export -retry and some operator controls require board. +Policy: Export retry and branch reconciliation require board runtime management. +Physical-owner stop confirmation uses the separate `runtime:confirm_owner_stop` +action. It requires active company owner/admin membership, trusted local board +access, or instance administration. Agents and ordinary runtime managers are denied. +The exact source task, running native run, accepted result and owner receipt must +match; active copyback locks, live local writers, and newer runs prevent admission. +An authorized administrator must first verify through the deployment platform that +the exact controller and all copyback children stopped, then attest that evidence. +Elapsed time, an expired coordinator lease or a lost advisory connection is not proof. +Only the exact owner receipt is released; normal reconciliation performs export and +commitment without another provider turn or bypassing existing task waits. -Evidence: `source-only` — [implementation](../../server/src/routes/issues.ts#L9478) at -the reviewed commit above. +Evidence: `source-only` — [route](../../server/src/routes/issues.ts), +[administrator authorization](../../server/src/services/authorization.ts), and +[owner admission](../../server/src/services/native-runtime/native-workspace-owner-recovery.ts) +reviewed for this change. Local authorization, route, UI and finalization regressions +cover member/agent denial, exact-owner admission and the real reconciler preserving +monitor and approval waits. This is not live provider-stop proof. -Verification: `not-run` — assert the documented agent denial with an owned test fixture; +Verification: `not-run` for live agent/deployment execution — assert the documented agent denial with an owned test fixture; verify any listed agent request/result path separately. A delegated operation needs its own policy and access review. Gaps: Export retry and branch reconciliation require board runtime:manage at -their HTTP routes. Native `call_api` preserves these denials. +their HTTP routes. The exact-owner operator fallback additionally requires +runtime:confirm_owner_stop. Native call_api preserves these agent denials. +Automatic managed-cloud recovery uses independently verified, signed platform +proof through the finalization reconciler; agents cannot attest owner shutdown. ## `unknown-outcome` diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 3cd71abaffc..7d5a5945d6d 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -119,7 +119,22 @@ Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNoti tests pending/error/gated states and waiting for the real retry acknowledgement. The recovery-card suite covers workspace divergence, reissue, reconcile, and confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx) -covers its separate repair surface. +covers quiet restoration, legacy/operator assistance and resolved states. + +For an abandoned native copyback owner, the harness verifies the exact stopped +deployment and the reconciler restores saved work automatically. Task users see +no process details or shutdown form. The documented operator API handles older +receipts after independent platform verification, through a separate administrator +gate. The [authorization regressions](../../server/src/__tests__/authorization-service.test.ts) +and [route regressions](../../server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts) +cover administrator admission, ordinary-member and agent denial, suspended and +foreign membership, missing role data, and stale cloud admin rows. +The [native workspace finalizer suite](../../server/src/services/native-runtime/native-workspace-finalizer.test.ts) +runs the real reconciler with a valid saved yield from `result_accepted`, +`workspace_finalizing`, and `retryable_failure`. It verifies physical fencing +before verified stop, automatic and operator recovery, successful export and commitment, hold resolution, +unchanged saved results, retained approval/monitor waits, and no new provider wake. +These local regressions do not prove a deployment platform stopped a foreign writer. Manual: reproduce an exhausted disposable action. Click once, verify pending state prevents repeated clicks, and inspect a successful continuation or an @@ -128,6 +143,17 @@ repair, record the compared branches/revisions and preserve uncommitted work; confirm the repair's actual result before resuming. Check the denied and stale action paths. A changed card label is insufficient proof of workspace integrity. +For an abandoned native controller, follow the [stop-verification procedure](../../doc/native-workspace-finalization-recovery.md) +with a disposable deployment. A new platform-bound owner recovers automatically +only after the harness signs proof that the exact deployment fully stopped. +Legacy owners require independently verified stop evidence through the operator +API; the task has no shutdown form. Confirm +the original accepted result commits, the recovery hold resolves, saved files +remain intact, and existing approval waits or monitor checks remain. No new agent +turn should start. Repeat with a retry-phase owner and an ordinary runtime manager; +the first can resume through automatic proof or authorized operator confirmation, +while the second cannot use the operator API. + ### `post-stop-message` Automated: [legacy continuation](../../server/src/services/recovery/legacy-continuation.test.ts), diff --git a/maps/storybook-map/INDEX.md b/maps/storybook-map/INDEX.md index 644f1692416..d9ec42885ff 100644 --- a/maps/storybook-map/INDEX.md +++ b/maps/storybook-map/INDEX.md @@ -4,7 +4,7 @@ [Contract and maintenance](README.md) · [Review findings](REVIEW.md) -224 story files; 2233 indexed stories. Counts describe inventory, not passed tests or complete feature coverage. +226 story files; 2242 indexed stories. Counts describe inventory, not passed tests or complete feature coverage. Status describes product alignment; implementation describes the rendered subject; boundary describes simulated data. Sources and limitations are in catalog.json. Related capabilities cross-list one canonical story file. @@ -336,10 +336,12 @@ Stopped work, workspace repair, bounded continuation, and reconnect. | [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | current / production / fixtures | 9 / 3 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | -| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | +| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite. | | [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 8 / 4 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. | | [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 2 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Master reconciliation inspected assignment-history recipient suggestions and manual-selection retention; existing stories do not establish the new assignment-history behavior. Expanded upstream verification recipes were inspected; the existing fixture scenarios do not prove the added backend handoff, workspace, recovery or authorization contracts. | +| [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) | `recovery/workspace`, `recovery/source-action` | current / production / fixtures | 6 / 2 | Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests. | +| [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) | `recovery/workspace`, `recovery/source-action` | unreviewed / mixed / fixtures | 3 / 0 | Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance. | ## Apps, channels, and extensions @@ -668,9 +670,9 @@ References are navigation links, not proof of parity. An absent link means no ca - `execution-environments/images`: No cataloged story. - `recovery/diagnosis`: [Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards](../../ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx) (unreviewed), [Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab](../../ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice](../../ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page](../../ui/storybook/stories/secrets/credential-access/02-task-page.stories.tsx) (unreviewed), [Projects and execution/Recovery/Task chat/Recovery notice](../../ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Tasks/Execution recovery](../../ui/storybook/stories/recovery/execution-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue/Blocked notice](../../ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff](../../ui/storybook/stories/tasks/successful-run-handoff.stories.tsx) (unreviewed) - `recovery/bounded-continuation`: [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) (unreviewed) -- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed) +- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed) - `recovery/exhaustion`: No cataloged story. -- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed) +- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed) - `recovery/unknown-outcome`: No cataloged story. - `recovery/user-continuation`: No cataloged story. - `recovery/reconnect`: [Projects and execution/Recovery/App/Connection recovery](../../ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx) (unreviewed) diff --git a/maps/storybook-map/REVIEW.md b/maps/storybook-map/REVIEW.md index ba5dee90411..954448ee6d8 100644 --- a/maps/storybook-map/REVIEW.md +++ b/maps/storybook-map/REVIEW.md @@ -274,6 +274,35 @@ Reviewed the same twelve workspace-recipe-dependent entries after extending the Reviewed the twelve unchanged workspace recipe subjects and fixtures after correcting implicit default selection. Task creation and controller classification now share configured-policy, primary, creation-time and ID precedence. Explicit choices and retained denial markers still suppress controller hooks. Project strategy fixtures show configuration, while runtime status and file/browser/artifact/service fixtures do not execute source scripts or prove this backend ordering. Refreshed only their reviewed recipe fingerprints and retained all classifications and stable IDs. No browser render, play, screenshot, Storybook build or live runtime acceptance was performed. +## Workspace finalization UI refinement — 2026-10-13 + +Source base: `f07efa2802`, followed by scoped edits to the production recovery notice and its two story suites. Reviewed `recovery/workspace` and `recovery/source-action`; stable story IDs and administrator admission remain unchanged. The notice leads with saved work, discloses controller metadata, explains evidence/confirmation requirements, and distinguishes queue acknowledgement from completion. A stale controller error preserves the note and clears confirmation so the operator must verify again. The task composition uses viewer identity for human bubbles and omits a routine status badge. + +MacOS in-app browser inspection passed for the task journey at desktop and 390px mobile widths: controller disclosure with Enter, evidence without confirmation remaining disabled, explicit confirmation enabling submission, and a simulated queued result in place. The `owner-changed` story's play assertions passed; the friendly error was visible, the note remained, confirmation cleared and the action was disabled. The `long-controller-name` story rendered at 390px without horizontal overflow. The mobile task shell showed its bottom navigation; evidence text computed to 16px and Tab reached the enabled action. No physical mobile keyboard, Linux screenshot baseline, live deployment stop, real API admission or provider copyback acceptance is claimed. Other story states received source review, not a full browser execution pass. + +Refreshed the two reviewed catalog entries only. Global catalog validation already reported unrelated source-fingerprint drift before this work; those entries require their own review. + +## Automatic workspace recovery revision — 2026-10-13 + +Supersedes the task shutdown form reviewed above. The production task now renders +only a quiet restoration status or an honest operator-assistance fallback. It +contains no process metadata, evidence field, confirmation checkbox or resume +button. The task thread omits the duplicate technical recovery card. Obsolete +form stories and their simulated API fixture were removed; suite IDs remain +stable. The composed task journey includes a saved state without the notice. + +Observed in the macOS in-app browser: automatic status in the production shell, +completed example without the status, retained approval-wait message, and mobile +bottom navigation with equal content and viewport widths (no horizontal overflow). +These are fixture UI observations, not live provider shutdown or task commitment. +The production component test verifies removal on a resolved action update. The +real reconciler's database tests separately preserve accepted-result identity, +monitor and approval waits, and create no new run or wake request. + +Only the two reviewed suites have refreshed fingerprints. Global validation has +71 other source-fingerprint errors, including recovery-recipe dependents after +the operator instructions changed. It is not globally green. No unrelated review +fingerprints or visual baselines were accepted. ## Inherited activity, inbox and recovery recipes — 2026-10-12 @@ -534,3 +563,77 @@ capability IDs. These establish inventory, associations, references and review fingerprints. No new Storybook build, rendered `play`, built-index comparison, visual baseline, live authorization, cleanup, persistence or provider execution is claimed by this reconciliation. Those checks remain with final qualification. + +### Current-master reconciliation + +Merged master `5e8e992df2`. The newer catalog supplies the baseline associations +and reviews. Compared every declared input against that master: only seven +inputs changed for this feature (recovery recipe, recovery card, task page, quiet +status, two stories and their fixture). Reviewed those changes and the unchanged +subjects/fixtures they affect; refreshed the 26 dependent fingerprints while +retaining their classifications and simulation boundaries. The task-page change +adds the quiet status and suppresses only the owner-unverified duplicate card; +other task variants retain their behavior. The recovery-card change blocks +unsafe generic dismissal. Existing card regressions cover that denial. + +Five remaining errors have unchanged source bytes and metadata inherited from +master: shared setup, decision-card, decisions-desk, search and what-needs-me. +These require a separate review; no hashes or visual baselines were accepted for +them. The earlier 71-error count above describes the pre-merge baseline. + +### Inherited index metadata follow-up + +The current master's personal-toast recipe also left shared setup and four inbox +suites with stale metadata. Reviewed the recipe's added toast capability and +verification limits: decision-card, decisions-desk and search still demonstrate +their existing isolated subjects; what-needs-me's toast fixture does not prove +viewer eligibility. Their subject and fixture bytes are unchanged from master; +retained all unreviewed classifications and fixture boundaries. The generator +adds the missing toast capability and description to the existing inbox feature, +without changing feature order or IDs. Reviewed that generated diff and refreshed +these four entries, the shared setup and Overview fingerprints. No toast behavior, +new browser acceptance or visual baseline is claimed. Global catalog validation +now passes. This supersedes the five inherited-error count above. + +### Confirmed recovery status + +Reviewed the shipped notice, server evidence contract and both recovery fixtures. +Platform identity alone no longer means work is restoring: the server marks the +exact recovery receipt resuming only after verified stop proof and a successful +owner claim. Missing proof remains operator-required. The new unavailable-proof +story covers a platform-bound receipt without progress; confirmed and queued +stories retain the quiet status. Updated only the two affected fingerprints. +This is component/fixture evidence, not deployed harness acceptance. + +### Latest master reconciliation + +Retained master’s task-creation visibility scenarios, source-review notes and +unreviewed classifications. Seven overlapping entries differ from master only +in the already reviewed recovery recipe or task-detail recovery notice wiring. +Refreshed those combined-source fingerprints and regenerated the index. No new +creation-visibility acceptance or visual baseline is claimed. + +## Codex and workspace-recovery catalog merge — 2026-10-13 + +Reconciled Codex head `7956056b2f793aa1043f4a4454630cdedd90185c` with current +master `e3029c632b26b68595adc1a704a387833b3cefcb`. Both dated review histories above +retain their original evidence scope. The current master catalog supplies the +new workspace-finalization recovery suites and their source classifications; +Codex creation, onboarding and Legacy/native editing states remain additive. +Existing association, sidebar, home and simulation-boundary metadata is preserved. + +The authored source ID union is intact: 2,224 IDs from current master plus 2,233 +from the Codex head produce 2,242 stories across 226 suites, with no missing or +unexpected IDs. All nine newly merged workspace-recovery states, the five focused +Codex preparation states, twenty existing settings states and 35 generated feature +Overview IDs remain. The recovery notice is a production component; the task +journey remains an explicitly mixed local composition, not a live task-page test. + +Canonical regeneration and validation pass. The existing catalog/generator/map +tests pass 27/27; map validation passes for 35 areas and 175 capability IDs. +Inspected the combined Codex source dependencies and generated harness-description +change before refreshing 79 affected entry fingerprints and the shared setup. +The incoming recovery entry metadata is retained. No classifications or visual +baselines were promoted. These results prove source inventory, references and +structural contracts; no new rendering, `play`, provider, permission, cleanup or +production-continuity acceptance is claimed for the merged source. diff --git a/maps/storybook-map/catalog.json b/maps/storybook-map/catalog.json index a9bfd520e33..2c692e1be1c 100644 --- a/maps/storybook-map/catalog.json +++ b/maps/storybook-map/catalog.json @@ -305,7 +305,7 @@ "ui/storybook/stories/agent-chat/agent-chat-sidebar.stories.tsx", "ui/src/adapters/adapter-display-registry.ts" ], - "fingerprint": "d95a1e9a7a4cf997b6d11c72cd26dce0ffd1c5d976b189e572fb4d716d943a51", + "fingerprint": "2973cbd7b859db2b2535a0e3f80402c3755801627d4efa3ff44de380050a2a55", "stableId": "design-explorations-agent-chat-sidebar-pages", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat sidebar/Pages" }, @@ -366,7 +366,7 @@ "ui/storybook/stories/agent-chat/agent-chat.stories.tsx", "ui/src/adapters/adapter-display-registry.ts" ], - "fingerprint": "991e34e4a986a8d85f59ba28a71b875236c2bdd24a15872448963801d1aa674a", + "fingerprint": "052bff02117a14b972e48ee4b773f8258f05bbc1dd481d36b535d3240b8b992f", "stableId": "design-explorations-agent-chat", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat" }, @@ -521,7 +521,7 @@ "ui/storybook/stories/agents/agent-persona-pages.stories.tsx", "ui/src/adapters/adapter-display-registry.ts" ], - "fingerprint": "b8756ae1b4b6c713793d979e540167a641451261de455cceb4395ca4a06b4fc9", + "fingerprint": "11e416bdddb44ece283d934eaf87f30eb85b04e5270c06d7bbba815c89484a76", "stableId": "agents-personas-full-pages", "sidebarTitle": "Agents and reusable capabilities/Hiring, configuration, and organization/Agents/Personas/Full pages" }, @@ -1207,7 +1207,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx" ], - "fingerprint": "cdb63642517b31c68df9ca8d62e60039f9f4d69d95d1a96dc96235fb7db1487d", + "fingerprint": "d6e735a2c56092274f139709aba0c1c555f3f3cb0fbf9820d50ee02874907b79", "stableId": "paperclip-assigned-backlog-safeguards", "sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards" }, @@ -1231,7 +1231,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx" ], - "fingerprint": "bcb6c0a3cb34fcd06e7b056521ad84fb968f73ba46a7ba00eda2a86bb060b11b", + "fingerprint": "70b54f5ce7c083b8e64c28501d66247d4753adabd732b377f7ad18309dc90ffe", "stableId": "product-inbox-blocked-tab", "sidebarTitle": "Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab" }, @@ -1470,7 +1470,7 @@ "ui/storybook/stories/agent-chat/chat-entry.stories.tsx", "ui/src/adapters/adapter-display-registry.ts" ], - "fingerprint": "4919a8431260abef52e018a03d607655484d728377bad92ae658aa4c0eff063e", + "fingerprint": "1063df44a6ff6e8972e15c4fa4718ea12d1edbe1ae5816055ab12a9964f007a8", "replacement": "ui/storybook/stories/agents/primary-agent/05-chat.stories.tsx", "stableId": "design-explorations-chat-entry", "sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Chat entry" @@ -1513,13 +1513,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/CloudAccessGate.tsx", "ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx" ], - "fingerprint": "bbec224b127ec6898cf95dfae922e6747483c0d0d4ec55ff7d1560c0c4a5c041", + "fingerprint": "64b2422bdabe2b07b563354481000171455841e49ccbf6a72270066d4e27e9e0", "stableId": "app-connection-recovery", "sidebarTitle": "Projects and execution/Recovery/App/Connection recovery" }, @@ -1843,14 +1843,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "maps/feature-map/secrets.md", "ui/src/components/task-chat/TaskChatMarker.tsx", "ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx" ], - "fingerprint": "2eba045db9127e5cd8d8bf7c9664e4598a5c84be117ae6159ce7171b24f2005d", + "fingerprint": "14757d026311d143e9301d6294d2f46f40dad25af8ea3f8d5cf1893b260f82db", "stableId": "task-credential-access-01-notice", "sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice" }, @@ -1877,7 +1877,7 @@ "ui/storybook/support/credential-access/CredentialTaskPage.tsx", "ui/src/adapters/adapter-display-registry.ts" ], - "fingerprint": "2fe1a1618237eb46648432c017fd19a5798a68b1ba29b293fa025b3ed293bdc0", + "fingerprint": "0c7ad2d8456115e0907edeab6925234516918387b9a6a7cb809464612a6cb01c", "stableId": "task-credential-access-02-task-page", "sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page" }, @@ -2227,13 +2227,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/ExecutionBlockerNotice.tsx", "ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx" ], - "fingerprint": "bee366ba1266e6692c8f0ee027c449fdda6b51762369ba431871a5af5ee018c6", + "fingerprint": "018d8d687921ea109cb98f587abd59b94722f1f148940852326417ce8a457b04", "stableId": "task-chat-recovery-notice", "sidebarTitle": "Projects and execution/Recovery/Task chat/Recovery notice" }, @@ -2247,7 +2247,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/ActiveAgentsPanel.tsx", @@ -2257,7 +2257,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/execution-recovery.stories.tsx" ], - "fingerprint": "6f8b245b169d95046bdb52513dfaf4937a96f807b72470c198863471f59c8149", + "fingerprint": "82ddff0cf6d98e558f9038d59960758347ad39f3c7a9aaf18f29214a8ad81756", "stableId": "tasks-execution-recovery", "sidebarTitle": "Projects and execution/Recovery/Tasks/Execution recovery" }, @@ -2881,7 +2881,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "maps/feature-map/task-coordination.md", @@ -2889,7 +2889,7 @@ "ui/src/components/task-chat/TaskChatBlockerLinks.tsx", "ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx" ], - "fingerprint": "6a84d7f60318d33a1835a668873095aaad1c215a83bbbb4735823fdcb3291140", + "fingerprint": "52f861389de245bda4d9c582581b49ce0b9442d60ee7eff950adc6801aaca8a4", "stableId": "product-issue-blocked-notice", "sidebarTitle": "Projects and execution/Recovery/Product/Issue/Blocked notice" }, @@ -3113,13 +3113,13 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueScheduledRetryCard.tsx", "ui/storybook/stories/recovery/model-capacity-retry.stories.tsx" ], - "fingerprint": "5e08f681f3f96cb7be32e316d3f4066e5239dcf0f4393cb9ec09d599734f201e", + "fingerprint": "1db367a2b77b8c747ab5c81c1fe3317da02bf294308e9e09af827bcd12ee0b9f", "stableId": "tasks-model-capacity-retry", "sidebarTitle": "Projects and execution/Recovery/Tasks/Model capacity retry" }, @@ -3383,7 +3383,7 @@ "ui/storybook/prototypes/originating-tasks/OriginatingTasks.tsx", "ui/storybook/stories/task-coordination/originating-tasks.stories.tsx" ], - "fingerprint": "59123e73a0286fd11a48376ee152a974d730c59bd5421fd273cfca3a185f6e08", + "fingerprint": "32c9981ce024550b26636b97d7337f7b34a1207e8bcdd5759a11d9a7ddb204fe", "stableId": "ux-labs-tasks-created-from-a-task", "sidebarTitle": "Tasks and conversations/Delegation, dependencies, and signoff/UX Labs/Tasks Created From a Task" }, @@ -3825,7 +3825,7 @@ "ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "d4b5ba6bce33a9657d859d3278b1f84bbc9b77cffd24e370214a0fce24c0a02b", + "fingerprint": "ebd9132a1f0228f775b1f382fc30dcae68e65284c8118c7b8a2eee3219c741ed", "stableId": "private-tasks-01-creation", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/01 Creation" }, @@ -3971,7 +3971,7 @@ "ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "5fb218d88c3b6815839732e4226a27f4b40a2df46c8cbad6709cb665546b4248", + "fingerprint": "6069116fffc3a661df1fb10245f3f6d8045086efd71c886a8064a27eed09ac42", "stableId": "private-tasks-06-full-product-pages", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/06 Full product pages" }, @@ -4018,7 +4018,7 @@ "ui/storybook/stories/access/private-tasks/07-user-stories.stories.tsx", "ui/storybook/support/private-tasks/PrivacyStory.tsx" ], - "fingerprint": "5fbe5c61f8754bc0c1db1c6a990b9c44bc5d941b7b61de77fac76300e6c42a22", + "fingerprint": "483a8bfae265844bfa10ebd8529572f43c23c8474d414b45ba0b663efc280dbd", "stableId": "private-tasks-07-user-stories", "sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/07 User stories" }, @@ -4477,7 +4477,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/DispositionRecoveryNotice.tsx", @@ -4486,7 +4486,7 @@ "ui/storybook/prototypes/recovery-notice/RecoveryNoticePreview.tsx", "ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx" ], - "fingerprint": "cfd7345545112ea9ecc13035ebd187451117de5ed010f439cfb8c6962461fb3f", + "fingerprint": "e83bcaf099227bfcd17d571f680121b5b51d5f605d76479165000a575229a884", "stableId": "design-previews-recovery-notice", "sidebarTitle": "Projects and execution/Recovery/Design previews/Recovery notice" }, @@ -4737,7 +4737,7 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueProperties.tsx", @@ -4745,7 +4745,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/scheduled-retry.stories.tsx" ], - "fingerprint": "43ef1917079251d558b3f52e5af750dfb69a54d3326d31494dbf037baeefe193", + "fingerprint": "f360f4207ef2b1cef18eee2307c50a7bce3a5f5bbf8966b24cd86b3ec50d21b4", "stableId": "product-issue-scheduled-retry-surfaces", "sidebarTitle": "Projects and execution/Recovery/Product/Issue Scheduled retry surfaces" }, @@ -5049,8 +5049,8 @@ "status": "unreviewed", "implementation": "production", "boundary": "fixtures", - "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite.", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/IssueBlockedNotice.tsx", @@ -5059,7 +5059,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/recovery/source-issue-recovery.stories.tsx" ], - "fingerprint": "1b6e44d4fb4ab7e2e63fb8816aa31fbe3267e10647f238ed82e6591a82d2e918", + "fingerprint": "949814263788ae0e3501b67bbd26118f150db219fc04e5cf653f18dd345ec8d5", "stableId": "paperclip-source-issue-recovery", "sidebarTitle": "Projects and execution/Recovery/Paperclip/Source Issue Recovery" }, @@ -5074,14 +5074,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/SourceResolvedFoldBadge.tsx", "ui/src/components/SourceResolvedFoldCallout.tsx", "ui/storybook/stories/recovery/source-resolved-fold.stories.tsx" ], - "fingerprint": "3af3daff618b8c278bcbf10ba3e5d081d87240c680758ccb3078eccdf9283122", + "fingerprint": "70cab38574df19267e35ab13c10c4a3eea444bb3b6e72c81f739f21cfc519df0", "stableId": "paperclip-source-resolved-fold", "sidebarTitle": "Projects and execution/Recovery/Paperclip/Source-resolved Fold" }, @@ -5162,7 +5162,7 @@ "ui/storybook/fixtures/paperclipData.ts", "ui/storybook/stories/tasks/successful-run-handoff.stories.tsx" ], - "fingerprint": "2a7db108d46f35f37dd9346d2fc039e61dff8c688e09d9ed7b0e777e7f79af0d", + "fingerprint": "27ad7c3c2b09acf5ac07163d44026511bd0c2b806387b591fef5fca343b0c63b", "stableId": "paperclip-successful-run-handoff", "sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff" }, @@ -5615,14 +5615,14 @@ "implementation": "production", "boundary": "fixtures", "notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.", - "reviewedAt": "2026-10-12", + "reviewedAt": "2026-10-13", "sources": [ "maps/feature-map/recovery.md", "ui/src/components/WorkspaceBaseRefRecoveryNotice.tsx", "ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx", "ui/storybook/support/workspace-branch-recovery/fixtures.ts" ], - "fingerprint": "77f708cc660f6829cadaffa6139930a102109976c41f7e24e2f662ed687d2412", + "fingerprint": "eeaa7044a1d5a2b5355a8ac2375820e3611c71920e88c04d87e5d7ce64ed929a", "stableId": "workspace-branch-recovery-01-repair-card", "sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/01 Repair card" }, @@ -5647,7 +5647,7 @@ "ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx", "ui/storybook/support/workspace-branch-recovery/fixtures.ts" ], - "fingerprint": "5eedfcb5738c125d3fa8ed7dc4b3d20a300b216de51f1ae5e58ede98d38fd75c", + "fingerprint": "f8a090cfdbadfe77d8ee064b99fc11b8f5e097d4e72173df2496280ef52152d1", "stableId": "workspace-branch-recovery-02-task-journey", "sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/02 Task journey" }, @@ -5712,6 +5712,53 @@ "stableId": "workspaces-service-control-bar", "sidebarTitle": "Projects and execution/Workspaces, services, and files/Workspaces/Service control bar" }, + { + "file": "ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx", + "capabilities": [ + "recovery/workspace", + "recovery/source-action" + ], + "purpose": "Workspace finalization recovery/01 Notice", + "status": "current", + "implementation": "production", + "boundary": "fixtures", + "notes": "Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests.", + "reviewedAt": "2026-10-13", + "sources": [ + "maps/feature-map/recovery.md", + "ui/src/components/WorkspaceExportRecovery.tsx", + "ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx", + "ui/storybook/support/workspace-finalization-recovery/fixtures.tsx" + ], + "fingerprint": "45bf8c793c91362634b7ce32ed0ec2007861b7fefbfd0e2da692c3dc804a53c9", + "stableId": "workspace-finalization-recovery-01-notice", + "sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/01 Notice" + }, + { + "file": "ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx", + "capabilities": [ + "recovery/workspace", + "recovery/source-action" + ], + "purpose": "Workspace finalization recovery/02 Task journey", + "status": "unreviewed", + "implementation": "mixed", + "boundary": "fixtures", + "notes": "Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance.", + "reviewedAt": "2026-10-13", + "sources": [ + "maps/feature-map/recovery.md", + "ui/src/components/Layout.tsx", + "ui/src/components/WorkspaceExportRecovery.tsx", + "ui/src/components/task-chat/TaskChatBubble.tsx", + "ui/src/components/task-chat/TaskChatComposer.tsx", + "ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx", + "ui/storybook/support/workspace-finalization-recovery/fixtures.tsx" + ], + "fingerprint": "aa76cd28dde8d50abd208b34283249170f3a85435ba82fff169a1c33433dfca1", + "stableId": "workspace-finalization-recovery-02-task-journey", + "sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/02 Task journey" + }, { "file": "ui/storybook/stories/agents/codex-runner/creation.stories.tsx", "capabilities": [ diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index e09c671f2d5..80e95fce011 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1999,6 +1999,7 @@ export { resolveIssueRecoveryActionSchema, isValidExistingBranchName, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index e7b3197a550..8e224de0924 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -441,6 +441,7 @@ export { resolveIssueRecoveryActionSchema, isValidExistingBranchName, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, issueReviewRequestSchema, issueExecutionWorkspaceSettingsSchema, checkoutIssueSchema, diff --git a/packages/shared/src/validators/issue.ts b/packages/shared/src/validators/issue.ts index 8fddad793ba..0d9ea3bdceb 100644 --- a/packages/shared/src/validators/issue.ts +++ b/packages/shared/src/validators/issue.ts @@ -557,6 +557,14 @@ const RESOLVE_ISSUE_RECOVERY_ACTION_OUTCOMES = [ "cancelled", ] as const; +export const resumeWorkspaceFinalizationSchema = z.object({ + actionId: z.string().guid(), + runId: z.string().guid(), + ownerToken: z.string().guid(), + controllerAndCopybackStopped: z.literal(true), + stopEvidence: z.string().trim().min(20).max(12000), +}).strict(); + export const retryWorkspaceExportSchema = z.object({ actionId: z.string().guid(), runId: z.string().guid(), diff --git a/server/package.json b/server/package.json index 82088f5548b..0ce4ac72f0f 100644 --- a/server/package.json +++ b/server/package.json @@ -125,6 +125,8 @@ "acpx" ], "devDependencies": { + "@opentelemetry/context-async-hooks": "2.10.0", + "@opentelemetry/sdk-trace": "2.10.0", "@paperclipai/paperclip-runner": "workspace:*", "@types/compression": "^1.8.1", "@types/express": "^5.0.0", diff --git a/server/src/__tests__/authorization-service.test.ts b/server/src/__tests__/authorization-service.test.ts index 4e1f3f63c2e..a2b28921c5e 100644 --- a/server/src/__tests__/authorization-service.test.ts +++ b/server/src/__tests__/authorization-service.test.ts @@ -1810,6 +1810,41 @@ describeEmbeddedPostgres("authorization service", () => { expect(await auth.decide({ actor, action: "agent:wake", resource: { ...resource, companyId: otherCompany.id } })).toMatchObject({ allowed: false }); }); + it.each(["session", "cloud_tenant"] as const)("restricts %s workspace-owner stop confirmation to administrators", async source => { + const company = await createCompany(db, "OwnerStop"); + const otherCompany = await createCompany(db, "OtherOwnerStop"); + const userId = await createUser(db); + await db.insert(companyMemberships).values({ companyId: company.id, + principalType: "user", principalId: userId, status: "active", membershipRole: "owner" }); + const actor = { type: "board" as const, source, userId, companyIds: [company.id] }; + const auth = authorizationService(db); + const request = { actor, action: "runtime:confirm_owner_stop" as const, + resource: { type: "company" as const, companyId: company.id } }; + for (const membershipRole of ["owner", "admin", "operator", "member", "viewer", null]) { + await db.update(companyMemberships).set({ membershipRole }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide(request)).allowed, `role ${membershipRole}`).toBe(membershipRole === "owner" || membershipRole === "admin"); + } + await db.update(companyMemberships).set({ membershipRole: "admin", status: "suspended" }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide(request)).allowed).toBe(false); + await db.update(companyMemberships).set({ status: "active" }).where(eq(companyMemberships.principalId, userId)); + expect((await auth.decide({ ...request, resource: { type: "company", companyId: otherCompany.id } })).allowed).toBe(false); + }); + + it("does not allow agent delegation or stale cloud admin rows to confirm an owner stop", async () => { + const company = await createCompany(db, "OwnerStopDelegation"); + const userId = await createUser(db); + const agent = await createAgent(db, company.id, { role: "ceo" }); + await db.insert(companyMemberships).values({ companyId: company.id, + principalType: "user", principalId: userId, status: "active", membershipRole: "operator" }); + await db.insert(instanceUserRoles).values({ userId, role: "instance_admin" }); + const auth = authorizationService(db); + const request = { action: "runtime:confirm_owner_stop" as const, resource: { type: "company" as const, companyId: company.id } }; + expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: false } })).allowed).toBe(false); + expect((await auth.decide({ ...request, actor: { type: "agent", agentId: agent.id, companyId: company.id, onBehalfOfUserId: userId } })).allowed).toBe(false); + expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: true } })).allowed).toBe(true); + expect((await auth.decide({ ...request, actor: { type: "board", source: "local_implicit" } })).allowed).toBe(true); + }); + it("limits viewer members to read-only visibility actions", async () => { const company = await createCompany(db, "BoardViewerVisibility"); const userId = `user-${randomUUID()}`; diff --git a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts index 6aa18723cba..4afb8bad37c 100644 --- a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts +++ b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts @@ -162,8 +162,10 @@ const mockLogActivity = vi.hoisted(() => vi.fn(async () => undefined)); const mockObserveCrossIssueInfluence = vi.hoisted(() => vi.fn(async () => null)); const mockRetryWorkspaceExport = vi.hoisted(() => vi.fn()); +const mockResumeWorkspaceFinalization = vi.hoisted(() => vi.fn()); function registerRouteMocks() { + vi.doMock("../services/native-runtime/native-workspace-owner-recovery.js", () => ({ resumeNativeWorkspaceAfterOwnerStop: mockResumeWorkspaceFinalization })); vi.doMock("../services/native-runtime/native-workspace-export-retry.js", () => ({ retryNativeWorkspaceExport: mockRetryWorkspaceExport })); vi.doMock("@paperclipai/shared/telemetry", () => ({ trackAgentTaskCompleted: vi.fn(), @@ -2009,6 +2011,52 @@ describe("agent issue mutation checkout ownership", () => { ); }); + it("resumes a verified stopped workspace owner without waking the provider", async () => { + mockAccessService.decide.mockResolvedValue({ allowed: true }); + mockResumeWorkspaceFinalization.mockResolvedValue({ runId: ownerRunId, status: "queued" }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }); + expect(res.status).toBe(202); + expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({ action: "runtime:confirm_owner_stop" })); + expect(mockResumeWorkspaceFinalization).toHaveBeenCalledWith(expect.objectContaining({ companyId, issueId, runId: ownerRunId, ownerToken: ownerRunId, actorId: "board-user" })); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + + it.each([{ controllerAndCopybackStopped: false }, { stopEvidence: "" }])("requires explicit stop evidence: %j", async invalid => { + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited.", ...invalid }); + expect(res.status).toBe(400); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + }); + + it.each(["agent", "viewer"])("rejects %s workspace-owner release before admission", async kind => { + mockAccessService.decide.mockImplementation(async (input: { action: string }) => ({ allowed: input.action === "issue:read", explanation: "No runtime access" })); + const res = await request(await createApp(kind === "agent" ? ownerActor() : boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }); + expect(res.status).toBe(403); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + }); + + it("rejects an ordinary runtime manager's claimed foreign-owner stop", async () => { + mockAccessService.decide.mockImplementation(async ({ action }: { action: string }) => ({ + allowed: action === "issue:read" || action === "runtime:manage", + explanation: "A deployment administrator must verify the controller stopped", + })); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`) + .send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId, + controllerAndCopybackStopped: true, stopEvidence: "I claim that the foreign controller and all its processes have stopped." }); + expect(res.status).toBe(403); + expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled(); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + it("queues board export-only recovery without calling provider wake", async () => { mockAccessService.decide.mockResolvedValue({ allowed: true }); mockRetryWorkspaceExport.mockResolvedValue({ runId: ownerRunId, status: "queued" }); @@ -2030,7 +2078,25 @@ describe("agent issue mutation checkout ownership", () => { expect(res.status).toBe(403); expect(mockRetryWorkspaceExport).not.toHaveBeenCalled(); }); - it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => { + it.each(["done", "in_review"])("does not bypass unverified copyback via false_positive to %s", async sourceIssueStatus => { + const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }); + mockIssueService.getById.mockResolvedValue(sourceIssue); + mockIssueService.update.mockImplementation(async (_id: string, patch: Record) => ({ ...sourceIssue, ...patch })); + mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({ + id: recoveryActionId, status: "active", kind: "active_run_watchdog", ownerType: "board", + ownerAgentId: null, returnOwnerAgentId: ownerAgentId, + cause: "native_workspace_finalization_owner_unverified", evidence: { runId: ownerRunId }, + }); + const res = await request(await createApp(boardActor())) + .post(`/api/issues/${issueId}/recovery-actions/resolve`) + .send({ actionId: recoveryActionId, outcome: "false_positive", sourceIssueStatus }); + expect(res.status).toBe(409); + expect(res.body.details?.code).toBe("workspace_owner_stop_required"); + expect(mockIssueService.update).not.toHaveBeenCalled(); + expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); + }); + + it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted", "native_workspace_finalization_owner_unverified"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => { const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId }); mockIssueService.getById.mockResolvedValue(sourceIssue); mockIssueService.update.mockImplementation(async (_id: string, patch: Record) => ({ ...sourceIssue, ...patch })); @@ -2043,7 +2109,7 @@ describe("agent issue mutation checkout ownership", () => { .post(`/api/issues/${issueId}/recovery-actions/resolve`) .send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus }); expect.soft(res.status).toBe(409); - expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : "workspace_export_retry_required"); + expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : cause === "native_workspace_finalization_owner_unverified" ? "workspace_owner_stop_required" : "workspace_export_retry_required"); expect.soft(mockIssueService.update).not.toHaveBeenCalled(); expect.soft(mockHeartbeatService.wakeup).not.toHaveBeenCalled(); }); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 7af8d4c1239..fa2abf0c522 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -17,6 +17,7 @@ import { setIssueTitleSchema } from "@paperclipai/shared"; import { resolveConfirmationFromComment } from "../services/confirmation-comment-resolution.js"; import { createIssueReadTiming } from "../services/issue-read-timing.js"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; +import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js"; import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js"; import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js"; import { deliverConversationComments, isConversation } from "../services/agent-conversations.js"; @@ -96,6 +97,7 @@ import { resolveCreateIssueStatusDefault, resolveIssueRecoveryActionSchema, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, runnerGoalActionRequestSchema, feedbackTargetTypeSchema, feedbackTraceStatusSchema, @@ -9511,6 +9513,17 @@ export function issueRoutes( }); }); + router.post("/issues/:id/recovery-actions/resume-workspace-finalization", validate(resumeWorkspaceFinalizationSchema), async (req, res) => { + assertBoard(req); + const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found"); + if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return; + const decision = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", resource: { type: "company", companyId: issue.companyId } }); + if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision)); + const receipt = await resumeNativeWorkspaceAfterOwnerStop({ db, companyId: issue.companyId, issueId: issue.id, + ...req.body, actorId: getActorInfo(req).actorId }); + res.status(202).json(receipt); + }); + router.post("/issues/:id/recovery-actions/retry-workspace-export", validate(retryWorkspaceExportSchema), async (req, res) => { assertBoard(req); const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found"); @@ -9757,6 +9770,10 @@ export function issueRoutes( throw conflict("Unsafe workspace export recovers automatically without another provider turn.", { code: "workspace_export_automatic_recovery" }); } + if ((outcome === "restored" || outcome === "false_positive") && activeRecoveryAction.cause === "native_workspace_finalization_owner_unverified") { + throw conflict("Verify the previous controller and its copyback processes stopped, then use Resume saved result.", { code: "workspace_owner_stop_required" }); + } + if (outcome === "restored" && isNativeWorkspaceExportRepairCause(activeRecoveryAction.cause)) { throw conflict("Repair the retained sandbox, then use Retry workspace export to finish the accepted result without another provider turn.", { code: "workspace_export_retry_required" }); } diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 532e3a66a22..e749eaeef3c 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -240,6 +240,7 @@ import { createAcceptedPlanDecompositionSchema, resolveIssueRecoveryActionSchema, retryWorkspaceExportSchema, + resumeWorkspaceFinalizationSchema, cancelIssueThreadInteractionSchema, // Secret provider configs and remote import createSecretProviderConfigSchema, @@ -10635,6 +10636,15 @@ registerCurrentRoute({ summary: "List issue recovery actions", }); +registerCurrentRoute({ + method: "post", + path: "/api/issues/{id}/recovery-actions/resume-workspace-finalization", + tags: ["issues"], + summary: "Resume a saved native result after operator verification of the prior workspace owner's stop", + body: resumeWorkspaceFinalizationSchema, + responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict }, +}); + registerCurrentRoute({ method: "post", path: "/api/issues/{id}/recovery-actions/retry-workspace-export", diff --git a/server/src/services/authorization.ts b/server/src/services/authorization.ts index 7bc119b86a4..9b68990080e 100644 --- a/server/src/services/authorization.ts +++ b/server/src/services/authorization.ts @@ -94,6 +94,7 @@ export type AuthorizationAction = | "project:read" | "project:write_workspace" | "runtime:manage" + | "runtime:confirm_owner_stop" | "secrets:read" | "secrets:propose"; @@ -184,6 +185,7 @@ function permissionForAction(action: AuthorizationAction): PermissionKey | null action === "issue:read" || action === "project:read" || action === "runtime:manage" || + action === "runtime:confirm_owner_stop" || action === "secrets:read" || action === "secrets:propose" ) { @@ -1858,6 +1860,11 @@ export function authorizationService(db: Db | DbTransaction) { }); } + if (input.action === "runtime:confirm_owner_stop" && input.actor.type !== "board") { + return deny({ action: input.action, reason: "deny_unsupported_action", + explanation: "Only a deployment administrator can confirm a workspace controller has stopped." }); + } + if (input.actor.type === "board") { let taskAssignmentPolicyEffect: AssignmentPolicyEffect | null = null; if (input.actor.source === "local_implicit") { @@ -1886,6 +1893,15 @@ export function authorizationService(db: Db | DbTransaction) { explanation: "Allowed because the actor is an instance admin.", }); } + if (input.action === "runtime:confirm_owner_stop") { + const membership = await getActiveMembership(companyId, "user", input.actor.userId ?? ""); + if (membership?.membershipRole === "owner" || membership?.membershipRole === "admin") { + return allow({ action: input.action, reason: "allow_role_default", + explanation: "Allowed by the active company owner or administrator role." }); + } + return deny({ action: input.action, reason: "deny_missing_grant", + explanation: "Only a company owner, company administrator, or instance administrator can confirm a workspace controller has stopped." }); + } // What instance-admin elevation used to give cloud tenant users is // replaced by company-scoped visibility: an active membership in the // resource company grants the same read surface a same-company agent diff --git a/server/src/services/cloud-runtime-identity.ts b/server/src/services/cloud-runtime-identity.ts index e6567f2cf9e..435c7f7562b 100644 --- a/server/src/services/cloud-runtime-identity.ts +++ b/server/src/services/cloud-runtime-identity.ts @@ -231,7 +231,7 @@ function decodeJsonPart(part: string, label: string): Record { return parsed as Record; } -function publicKeyForKid(env: NodeJS.ProcessEnv, kid: string) { +export function cloudRuntimePublicKeyForKid(env: NodeJS.ProcessEnv, kid: string) { const raw = nonEmpty(env.PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS); if (!raw) throw new Error("PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS is not configured"); let parsed: unknown; @@ -276,7 +276,7 @@ function verifyClaims(input: { ) { throw new Error("Cloud runtime identity protected header is invalid"); } - const key = publicKeyForKid(input.env, header.kid); + const key = cloudRuntimePublicKeyForKid(input.env, header.kid); const signature = Buffer.from(encodedSignature, "base64url"); const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii"); if (!verify(null, signingInput, key, signature)) { @@ -523,7 +523,7 @@ export function verifyCloudControlAssertion(input: { ) { throw new Error("Cloud control protected header is invalid"); } - const key = publicKeyForKid(env, header.kid); + const key = cloudRuntimePublicKeyForKid(env, header.kid); const signature = Buffer.from(encodedSignature, "base64url"); const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii"); if (!verify(null, signingInput, key, signature)) { diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 4a6d6a32611..fa3b28245a8 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js"; import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js"; import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js"; import { prepareHeartbeatExecutionConfiguration } from "./heartbeat/execution-configuration.js"; @@ -1918,6 +1919,7 @@ export function heartbeatService( if (run.runtimeMode === "legacy" && run.controllerBootId && run.controllerBootId !== legacyControllerBootId) return; activeRunExecutions.add(run.id); + const timePreparation = createHeartbeatPreparationTiming(run.id); const executionControl = createAdapterExecutionControl(); // This coarse scope also covers host finalization after the adapter returns. // Nested scopes refine the label without making any termination claim. @@ -2613,7 +2615,7 @@ export function heartbeatService( materializeNativeChatWorktree, explicitTaskDirectory, useIsolatedTaskDirectory, - } = await selectHeartbeatEnvironment(db, { + } = await timePreparation("environment_selection", () => selectHeartbeatEnvironment(db, { run, agent, task: { issueRef, issueContext, issueId, context, executionProjectId, workspaceProjectContext, boundSourceWorkspace, hasSelectedWorkspaceSource: Boolean(selectedWorkspaceSource), taskWorkspaceIntent: taskWorkspaceIntentRow?.intent ?? null, responsibleUserId }, @@ -2623,9 +2625,9 @@ export function heartbeatService( }, config: { config, sourceWorkspaceConfig, issueAssigneeOverrides }, services: { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder }, - }); + })); let nativeChatExpectedCwd = selectedNativeChatExpectedCwd; - const executionConfiguration = await prepareHeartbeatExecutionConfiguration(db, { + const executionConfiguration = await timePreparation("execution_configuration", () => prepareHeartbeatExecutionConfiguration(db, { run, agent, task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext }, config: { sourceWorkspaceConfig, aiBinding }, @@ -2635,7 +2637,7 @@ export function heartbeatService( onFailureSecretsResolved: (readSecrets) => { readFailureReportSecrets = readSecrets; }, onManagedAiRuntimeAcquired: (runtime) => { managedAiRuntime = runtime; }, onIdentityRedactorPrepared: (redactor) => { identityRedactor = redactor; }, - }); + })); if (!executionConfiguration.prepared) return; const { mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub, @@ -2649,7 +2651,7 @@ export function heartbeatService( ...identityRedactor.values, ...collectRunFailureSecretValues(runtimeConfig.env, secretKeys), ]; - const preparedSession = await prepareHeartbeatSession(db, { + const preparedSession = await timePreparation("session", () => prepareHeartbeatSession(db, { run, agent, task: { context, issueRef, issueContext, taskProjectContext, workspaceProjectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision }, session: { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec }, @@ -2663,7 +2665,7 @@ export function heartbeatService( }, environment: { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig }, getLatestAgentConfigRevision, - }); + })); const { sessionConfigMetadata, configuredModel, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams } = preparedSession; const { resolvedWorkspace, @@ -2685,7 +2687,7 @@ export function heartbeatService( workspaceRealization, executionTarget, workspaceRestoreSource, - } = await prepareHeartbeatWorkspace(db, { + } = await timePreparation("workspace", () => prepareHeartbeatWorkspace(db, { run, agent, controllerLease, nativeRunnerPreparationSpans, task: { issueRef, @@ -2737,7 +2739,7 @@ export function heartbeatService( resolveReusedGitWorkspaceAnchor, appendRunEvent, }, - }); + })); const preparedInstructions = instructionWorkingCopy = createHeartbeatInstructionWorkingCopy({ db, instructionCopies, readRun: () => run!, agent, context, session: { taskSession, taskSessionForRun, sessionConfigMetadata }, @@ -2901,7 +2903,7 @@ export function heartbeatService( } else { delete context.paperclipScratch; } - const gitExecutionEnv = await prepareGitHubExecutionEnvironment({ + const gitExecutionEnv = await timePreparation("github_environment", () => prepareGitHubExecutionEnvironment({ target: executionTarget, cwd: executionWorkspace.cwd, env: Object.fromEntries( @@ -2915,12 +2917,12 @@ export function heartbeatService( networkAccess: trustPreset.kind === "standard" && process.env.PAPERCLIP_RUNNER_NETWORK_ACCESS !== "disabled", - }); + })); runtimeConfig = { ...runtimeConfig, env: gitExecutionEnv }; for (const key of MANAGED_GITHUB_TOKEN_KEYS) secretKeys.add(key); context.githubAuthenticationMode = useHostGitHub ? "host" : "managed"; if (!useHostGitHub) { - const githubLaunchers = await prepareHeartbeatGitHubLaunchers({ + const githubLaunchers = await timePreparation("github_launchers", () => prepareHeartbeatGitHubLaunchers({ native: agent.adapterType === "paperclip_runner", githubConfigured: githubSelection.configured, agentId: agent.id, @@ -2936,7 +2938,7 @@ export function heartbeatService( responsibleUserId: responsibleUserId ?? "", scope: "github_credentials", })?.token ?? "", - }); + })); githubLauncherLocation = githubLaunchers.cleanupLocation; runtimeConfig = { ...runtimeConfig, env: githubLaunchers.env }; secretKeys.add("PAPERCLIP_GITHUB_BROKER_TOKEN"); @@ -3167,7 +3169,7 @@ export function heartbeatService( }); const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent } = - await runOutput.begin(currentRun); + await timePreparation("run_output", () => runOutput.begin(currentRun)); if (runScopedMentionedSkillKeys.length > 0) { await onLog( "stdout", @@ -3178,7 +3180,7 @@ export function heartbeatService( const logEntry = formatRuntimeWorkspaceWarningLog(warning); await onLog(logEntry.stream, logEntry.chunk); } - await assertGitSensitiveAdapterWorkspaceValid({ + await timePreparation("workspace_validation", () => assertGitSensitiveAdapterWorkspaceValid({ adapterType: agent.adapterType, agentId: agent.id, issue: issueRef @@ -3195,17 +3197,17 @@ export function heartbeatService( executionTarget, environmentDriver: selectedEnvironment.driver, leaseMetadata: activeEnvironmentLease.lease.metadata, - }); + })); const adapterEnv = Object.fromEntries( Object.entries(parseObject(runtimeConfig.env)).filter( (entry): entry is [string, string] => typeof entry[0] === "string" && typeof entry[1] === "string", ), ); - const runtimeServices = await startHeartbeatRuntimeServices(db, { + const runtimeServices = await timePreparation("runtime_services", () => startHeartbeatRuntimeServices(db, { run, agent, issueId, issueRef, executionWorkspace, context, issuesSvc, onLog, persistedExecutionWorkspace, hostExecutionWorkspaceConfig, skipHostProvision, adapterEnv, workspaceOperationRecorder, - }); + })); const adapter = getServerAdapter(agent.adapterType); const durableGoalControlRun = readNonEmptyString(context.goalControlRequestId) !== null || @@ -3251,8 +3253,8 @@ export function heartbeatService( const hasInstructionFilesystem = nativeRuntimeResolution.kind !== "native" ? adapter.supportsInstructionsBundle === true : !["claude_managed_agents_api", "aws_agentcore_harness_api", "openai_dot_mcp"].includes(nativeRuntimeResolution.profile.backend); - runtimeConfig = await preparedInstructions.prepare({ runtimeConfig, nativeRuntimeResolution, hasInstructionFilesystem }); - const runtimeSelection = await selectHeartbeatRuntime(db, { + runtimeConfig = await timePreparation("instructions", () => preparedInstructions.prepare({ runtimeConfig, nativeRuntimeResolution, hasInstructionFilesystem })); + const runtimeSelection = await timePreparation("runtime_selection", () => selectHeartbeatRuntime(db, { run, agent, nativeRuntimeResolution, @@ -3319,7 +3321,7 @@ export function heartbeatService( nativeOwnershipHeld = true; }, stopControllerLease: () => controllerLease.stop(), - }); + })); if (!runtimeSelection.selected) return; const { nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff } = runtimeSelection; nativeWorkspaceSync = runtimeSelection.nativeWorkspaceSync; diff --git a/server/src/services/heartbeat/preparation-timing-context.test.ts b/server/src/services/heartbeat/preparation-timing-context.test.ts new file mode 100644 index 00000000000..8ae9daf918c --- /dev/null +++ b/server/src/services/heartbeat/preparation-timing-context.test.ts @@ -0,0 +1,53 @@ +import { createHash } from "node:crypto"; +import { setImmediate } from "node:timers/promises"; +import { context, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { TracerProvider, InMemorySpanExporter, SimpleSpanProcessor } from "@opentelemetry/sdk-trace"; +import { expect, it } from "vitest"; +import { createHeartbeatPreparationTiming } from "./preparation-timing.js"; + +it("parents child spans to their own preparation phase across overlapping awaited runs", async () => { + const manager = new AsyncLocalStorageContextManager().enable(); + const exporter = new InMemorySpanExporter(); + const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] }); + expect(context.setGlobalContextManager(manager)).toBe(true); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); + try { + const gates = [0, 1].map(() => { + let resolve!: () => void; + const promise = new Promise(release => { resolve = release; }); + return { promise, resolve }; + }); + const runs = ["first-run", "second-run"]; + const pending = runs.map((run, index) => createHeartbeatPreparationTiming(run)("workspace", async () => { + await gates[index].promise; + await setImmediate(); + const child = trace.getTracer("preparation-context-test").startSpan(`child.${run}`); + await setImmediate(); + child.end(); + })); + expect(trace.getActiveSpan()).toBeUndefined(); + gates[1].resolve(); + gates[0].resolve(); + await Promise.all(pending); + await provider.forceFlush(); + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(4); + const phases = runs.map(run => spans.find(span => span.name === "heartbeat.prepare.workspace" + && span.attributes["paperclip.task.run.run_id"] === createHash("sha256").update(run).digest("hex").slice(0, 12))!); + expect(phases.every(Boolean)).toBe(true); + expect(phases[0].spanContext().traceId).not.toBe(phases[1].spanContext().traceId); + for (const [index, run] of runs.entries()) { + const child = spans.find(span => span.name === `child.${run}`)!; + expect(child.parentSpanContext?.spanId).toBe(phases[index].spanContext().spanId); + expect(child.spanContext().traceId).toBe(phases[index].spanContext().traceId); + expect(phases[index].parentSpanContext).toBeUndefined(); + } + expect(trace.getActiveSpan()).toBeUndefined(); + } finally { + trace.disable(); + context.disable(); + manager.disable(); + await provider.shutdown(); + } +}); diff --git a/server/src/services/heartbeat/preparation-timing.test.ts b/server/src/services/heartbeat/preparation-timing.test.ts new file mode 100644 index 00000000000..d6b1005f527 --- /dev/null +++ b/server/src/services/heartbeat/preparation-timing.test.ts @@ -0,0 +1,49 @@ +import { createHash } from "node:crypto"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { createHeartbeatPreparationTiming } from "./preparation-timing.js"; + +const handles = vi.hoisted(() => ({ spans: [] as Array<{ setStatus: ReturnType; end: ReturnType }> })); +const startActiveSpan = vi.hoisted(() => vi.fn((_name, _options, callback) => { + const span = { setStatus: vi.fn(), end: vi.fn() }; + handles.spans.push(span); + return callback(span); +})); +vi.mock("@opentelemetry/api", () => ({ + SpanStatusCode: { ERROR: 2 }, + trace: { getTracer: () => ({ startActiveSpan }) }, +})); + +describe("heartbeat preparation timing", () => { + beforeEach(() => { vi.clearAllMocks(); handles.spans.length = 0; }); + + it("correlates preparation with the existing run hash without recording content", async () => { + const runId = "private-run-id"; + const time = createHeartbeatPreparationTiming(runId); + const result = { secret: "private result" }; + await expect(time("instructions", async () => result)).resolves.toBe(result); + expect(startActiveSpan).toHaveBeenCalledExactlyOnceWith("heartbeat.prepare.instructions", { + attributes: { "paperclip.task.run.run_id": createHash("sha256").update(runId).digest("hex").slice(0, 12) }, + }, expect.any(Function)); + expect(handles.spans[0].end).toHaveBeenCalledOnce(); + expect(handles.spans[0].setStatus).not.toHaveBeenCalled(); + }); + + it("ends a failed setup phase and preserves its exact error without recording its text", async () => { + const error = new Error("private filesystem or credential detail"); + await expect(createHeartbeatPreparationTiming("run")("workspace", async () => { throw error; })).rejects.toBe(error); + expect(handles.spans[0].setStatus).toHaveBeenCalledExactlyOnceWith({ code: 2 }); + expect(handles.spans[0].end).toHaveBeenCalledOnce(); + }); + + it("keeps overlapping operations open until their own work settles", async () => { + const time = createHeartbeatPreparationTiming("run"); + let release!: () => void; + const waiting = time("workspace", () => new Promise(resolve => { release = resolve; })); + await time("runtime_services", async () => undefined); + expect(handles.spans[0].end).not.toHaveBeenCalled(); + expect(handles.spans[1].end).toHaveBeenCalledOnce(); + release(); + await waiting; + expect(handles.spans[0].end).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/src/services/heartbeat/preparation-timing.ts b/server/src/services/heartbeat/preparation-timing.ts new file mode 100644 index 00000000000..2d089fe4375 --- /dev/null +++ b/server/src/services/heartbeat/preparation-timing.ts @@ -0,0 +1,25 @@ +import { createHash } from "node:crypto"; +import { SpanStatusCode, trace } from "@opentelemetry/api"; + +type PreparationPhase = "environment_selection" | "execution_configuration" | "session" + | "workspace" | "github_environment" | "github_launchers" | "run_output" + | "workspace_validation" | "runtime_services" | "instructions" | "runtime_selection"; + +export function createHeartbeatPreparationTiming(runId: string) { + const runIdHash = createHash("sha256").update(runId).digest("hex").slice(0, 12); + return (phase: PreparationPhase, work: () => Promise): Promise => + trace.getTracer("paperclip.heartbeat-preparation").startActiveSpan( + `heartbeat.prepare.${phase}`, + { attributes: { "paperclip.task.run.run_id": runIdHash } }, + async (span) => { + try { + return await work(); + } catch (error) { + span.setStatus({ code: SpanStatusCode.ERROR }); + throw error; + } finally { + span.end(); + } + }, + ); +} diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index aeb7384bee3..6dd31f091cb 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -375,6 +375,18 @@ export function issueRecoveryActionService(db: Db) { // beyond the advertised cap. A distinct identity can still supersede the // exhausted action through the branch above. if (isRecoveryBudgetExhausted(existing.evidence ?? {})) { + // Older sweeps replaced this operator-only hold with instructions to + // start another run. Restore the exact copyback recovery instructions + // without reopening its budget or erasing a recorded stop confirmation. + if (existing.cause === "native_workspace_finalization_owner_unverified" + && input.cause === existing.cause && input.fingerprint === existing.fingerprint + && !existing.evidence?.workspaceOwnerStop && existing.nextAction !== input.nextAction) { + const [updated] = await db.update(issueRecoveryActions).set({ nextAction: input.nextAction, updatedAt: now }) + .where(and(eq(issueRecoveryActions.id, existing.id), + inArray(issueRecoveryActions.status, [...ACTIVE_RECOVERY_ACTION_STATUSES]), + sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).returning(); + if (updated) return toReadModel(updated); + } return existing; } const nextAttemptCount = @@ -417,8 +429,9 @@ export function issueRecoveryActionService(db: Db) { fingerprint: existing.fingerprint, }, }, - nextAction: - `Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` + + nextAction: existing.cause === "native_workspace_finalization_owner_unverified" + ? input.nextAction + : `Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` + "Review the infrastructure failure and explicitly choose a replacement run or provider configuration.", wakePolicy: null, monitorPolicy: null, diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts index fcbff05fdee..5168b9bf856 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -1,15 +1,19 @@ +import { logActivity } from "../activity-log.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { nativeSha256 } from "./canonical.js"; import { NATIVE_WORKSPACE_OWNER_UNVERIFIED } from "./native-workspace-finalization-recovery-state.js"; import { randomUUID } from "node:crypto"; import os from "node:os"; -import { and, eq, sql } from "drizzle-orm"; -import { heartbeatRuns, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db"; +import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js"; import { currentNativeControllerIdentity, evaluateNativeControllerTakeover, } from "./native-restart-recovery.js"; +import { currentWorkspaceWorker, requestWorkspaceWorkerStopProof, type WorkspaceWorker, type WorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js"; + const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; const activeOwnerTokens = new Set(); // Positive same-boot evidence, retained only when durable cleanup could not be @@ -21,6 +25,7 @@ type Owner = { hostname: string; pid: number; processStartedAt: string; + worker?: WorkspaceWorker; }; export type NativeWorkspaceFinalizationOwnership = { token: string; @@ -58,7 +63,7 @@ export async function withNativeWorkspaceFinalizationOwnership( const identity = await currentNativeControllerIdentity(); const owner: Owner = { token: randomUUID(), controllerBootId: identity.bootId, hostname: os.hostname(), pid: identity.pid, - processStartedAt: identity.processStartedAt.toISOString(), + processStartedAt: identity.processStartedAt.toISOString(), worker: currentWorkspaceWorker(), }; const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.runtimeMode, "native")); const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`); @@ -73,13 +78,42 @@ export async function withNativeWorkspaceFinalizationOwnership( if (!run) throw new Error("native_workspace_finalization_binding_missing"); const rawPrior = run.profile?.[OWNER_KEY]; const prior = readOwner(rawPrior); + let automaticResultId: string | undefined; + const currentRecoveryResult = async (db: Db) => { + if (!run.issueId) return null; + // Use the arbiter's lock order and release row locks before copyback I/O. + const [coordinator] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId), + eq(nativeRunFinalizations.issueId, run.issueId), + )).for("update"); + const [issue] = await db.select().from(issues).where(and( + eq(issues.companyId, input.companyId), eq(issues.id, run.issueId), + )).for("update"); + const [currentRun] = await db.select().from(heartbeatRuns).where(scope).for("update"); + return coordinator && issue && currentRun ? currentNativeWorkspaceRecoveryResult(db, { + companyId: input.companyId, coordinator, issue, run: currentRun, + }) : null; + }; + let platformProof: WorkspaceWorkerStopProof | null = null; if (rawPrior != null) { const requireStopVerification = async () => { const fingerprint = nativeSha256({ runId: input.runId, owner: rawPrior }); + const nextAction = "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn."; const recovery = issueRecoveryActionService(input.db); const existing = run.issueId ? await recovery.getActiveForIssue(input.companyId, run.issueId) : null; // Rechecking the same physical owner is not a new recovery attempt. if (existing?.cause === NATIVE_WORKSPACE_OWNER_UNVERIFIED && existing.fingerprint === fingerprint) { + if (!existing.evidence?.workspaceOwnerStop) { + await input.db.update(issueRecoveryActions).set({ nextAction, updatedAt: new Date(), + evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"operator_required"}'::jsonb)`, + }).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, existing.id), + eq(issueRecoveryActions.fingerprint, fingerprint), eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`, + sql`(${issueRecoveryActions.nextAction} is distinct from ${nextAction} or ${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' is distinct from 'operator_required')`, + )); + } return { acquired: false } as const; } if (run.issueId) await recovery.upsertSourceScoped({ @@ -87,18 +121,21 @@ export async function withNativeWorkspaceFinalizationOwnership( kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: run.agentId, cause: NATIVE_WORKSPACE_OWNER_UNVERIFIED, fingerprint, - evidence: { runId: input.runId, owner: rawPrior }, - nextAction: "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn.", + evidence: { runId: input.runId, owner: rawPrior, workspaceOwnerRecovery: { state: "operator_required" } }, + nextAction, wakePolicy: null, maxAttempts: 1, supersedeOnIdentityChange: true, }); return { acquired: false } as const; }; - // A foreign host or malformed receipt cannot prove the old physical owner stopped. - if (!prior || prior.hostname !== owner.hostname) return requireStopVerification(); + // The harness checks the exact owned deployment, including orphaned children. + // A live deployment, unsigned answer or older receipt stays fenced. + if (prior?.worker) platformProof = await requestWorkspaceWorkerStopProof({ runId: input.runId, ownerToken: prior.token, worker: prior.worker }); + if (!prior) return requireStopVerification(); + if (!platformProof && prior.hostname !== owner.hostname) return requireStopVerification(); const knownJoinedHere = prior.controllerBootId === identity.bootId && prior.pid === identity.pid && joinedOwnerTokens.has(prior.token) && !activeOwnerTokens.has(prior.token); - if (!knownJoinedHere) { + if (!platformProof && !knownJoinedHere) { const takeover = await evaluateNativeControllerTakeover({ owner: { leaseOwner: prior.token, leaseExpiresAt: new Date(0), controllerPid: prior.pid, controllerProcessStartedAt: new Date(prior.processStartedAt) }, now: new Date(), @@ -119,15 +156,32 @@ export async function withNativeWorkspaceFinalizationOwnership( } // Claim before the potentially ambiguous write so finally can remove // only our token even if the server committed but its response was lost. - activeOwnerTokens.add(owner.token); - claimed = true; - const [claimedRun] = await input.db.update(heartbeatRuns).set({ - runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb)`, - }).where(and(scope, prior - ? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${prior.token}` - : sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null` - )).returning({ id: heartbeatRuns.id }); + const claimOwner = async (db: Db) => { + activeOwnerTokens.add(owner.token); + claimed = true; + const [claimedRun] = await db.update(heartbeatRuns).set({ + runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb) || ${JSON.stringify(platformProof ? { nativeWorkspaceOwnerStopProof: platformProof } : {})}::jsonb`, + }).where(and(scope, prior + ? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY} = ${JSON.stringify(rawPrior)}::jsonb` + : sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null` + )).returning({ id: heartbeatRuns.id }); + return claimedRun; + }; + // A stop receipt proves the old worker exited, not that its task is still + // current. Revalidate after the provider read and atomically with the CAS. + const claimedRun = platformProof ? await input.db.transaction(async tx => { + const result = await currentRecoveryResult(tx as unknown as Db); + if (!result) return null; + const claimedRun = await claimOwner(tx as unknown as Db); + if (claimedRun) automaticResultId = result.id; + return claimedRun; + }) : await claimOwner(input.db); if (!claimedRun) return { acquired: false } as const; + if (platformProof && run.issueId) await logActivity(input.db, { + companyId: input.companyId, actorType: "system", actorId: "workspace-recovery", + action: "issue.workspace_owner_stop_verified", entityType: "issue", entityId: run.issueId, + runId: input.runId, details: { proof: platformProof }, + }); if (prior) joinedOwnerTokens.delete(prior.token); const ownership: NativeWorkspaceFinalizationOwnership = { token: owner.token, @@ -137,12 +191,25 @@ export async function withNativeWorkspaceFinalizationOwnership( await lock.execute(sql`select 1`); const [current] = await input.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(owned).limit(1); if (!current) throw new Error("owner_replaced"); + if (automaticResultId) { + const result = await input.db.transaction(tx => currentRecoveryResult(tx as unknown as Db)); + if (result?.id !== automaticResultId) throw new Error("task_replaced"); + } } catch (cause) { throw new NativeWorkspaceFinalizationOwnershipLostError({ cause }); } }, }; await ownership.assertHeld(); + if (platformProof && run.issueId) await input.db.update(issueRecoveryActions).set({ + evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"resuming"}'::jsonb)`, + updatedAt: new Date(), + }).where(and( + eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId), + eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED), + eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + )); actionPromise = action(ownership); const value = await actionPromise; await ownership.assertHeld(); diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index 766dd1cf539..30737efb30b 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -1,3 +1,8 @@ +import * as platformProof from "./native-workspace-platform-proof.js"; +import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { nativeSha256 } from "./canonical.js"; +import { resumeNativeWorkspaceAfterOwnerStop } from "./native-workspace-owner-recovery.js"; +import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { spawn } from "node:child_process"; import { once } from "node:events"; import { readProcessStartedAt } from "../hot-restart.js"; @@ -9,6 +14,8 @@ import path from "node:path"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { and, desc, eq, sql } from "drizzle-orm"; import { + activityLog, + agentWakeupRequests, agents, companies, completionContracts, @@ -19,11 +26,15 @@ import { nativeRunFinalizations, nativeRunResults, issueRecoveryActions, + issueThreadInteractions, projects, workspaceOperations, + workAssessments, + statusDecisions, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js"; +import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js"; describe("native workspace finalization recovery", () => { let temporary: Awaited>; @@ -159,6 +170,225 @@ describe("native workspace finalization recovery", () => { } }); + async function seedAbandonedCopyback() { + const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Accepted yield, abandoned copyback" }); + const owner = { token: randomUUID(), hostname: "replaced-controller", pid: 2, + processStartedAt: "2026-10-10T00:02:55.560Z", controllerBootId: randomUUID() }; + await db.update(heartbeatRuns).set({ status: "running", nativePhase: "workspace_finalizing", + runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" }, + }).where(eq(heartbeatRuns.id, seeded.runId)); + await db.update(issues).set({ executionRunId: seeded.runId }).where(eq(issues.id, seeded.issueId)); + await db.update(nativeRunFinalizations).set({ phase: "workspace_finalizing" }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const recovery = issueRecoveryActionService(db); + const hold = { companyId, sourceIssueId: seeded.issueId, kind: "active_run_watchdog" as const, + ownerType: "board" as const, cause: "native_workspace_finalization_owner_unverified", + fingerprint: nativeSha256({ runId: seeded.runId, owner }), evidence: { runId: seeded.runId, owner }, maxAttempts: 1, + nextAction: "Verify the previous controller and its workspace-copyback processes have stopped. Resume workspace finalization without another provider turn." }; + const action = await recovery.upsertSourceScoped(hold); + await recovery.upsertSourceScoped(hold); + return { ...seeded, owner, actionId: action.id, companyId, db, ownerToken: owner.token, + actorId: "operator", controllerAndCopybackStopped: true as const, + stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." }; + } + + it("retains copyback-specific instructions after repeated recovery sweeps", async () => { + const seed = await seedAbandonedCopyback(); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(action.status).toBe("escalated"); + expect(action.nextAction).toContain("workspace-copyback processes have stopped"); + expect(action.nextAction).not.toContain("replacement run"); + await db.update(issueRecoveryActions).set({ nextAction: "Start a replacement run" }).where(eq(issueRecoveryActions.id, seed.actionId)); + await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected"); + const [repaired] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(repaired.nextAction).toContain("workspace-copyback processes have stopped"); + expect(repaired.attemptCount).toBe(action.attemptCount); + expect(repaired.evidence?.recoveryBudget).toEqual(action.evidence?.recoveryBudget); + }); + + it("releases only the verified owner and resumes the saved result without changing its disposition", async () => { + const seed = await seedAbandonedCopyback(); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false }); + const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" }); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(after).toEqual(before); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson).toEqual({ preserved: "workspace descriptor" }); + expect(run.status).toBe("running"); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(action.evidence?.workspaceOwnerStop).toMatchObject({ owner: seed.owner, actorId: "operator", resultId: before.resultId, stopEvidence: seed.stopEvidence }); + expect(action.status).toBe("escalated"); // Only commitment resolves the hold. + expect(action.wakePolicy).toBeNull(); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1); + const file = path.join(workspaceRoot, seed.runId); + expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => { + await fs.writeFile(file, "accepted work"); + return "resumed"; + })).toEqual({ acquired: true, value: "resumed" }); + expect(await fs.readFile(file, "utf8")).toBe("accepted work"); + // A duplicate confirmation neither clears a newer owner nor writes another audit. + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toEqual(receipt); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1); + }); + + it.each(["expired", "live", "unknown"] as const)("checks the coordinator lease before verified recovery: %s", async state => { + const seed = await seedAbandonedCopyback(); + const leaseExpiresAt = state === "unknown" ? null : new Date(Date.now() + (state === "expired" ? -60_000 : 60_000)); + await db.update(nativeRunFinalizations).set({ leaseOwner: "previous-reconciler", leaseExpiresAt }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (state === "expired") { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].leaseOwner).toBe("previous-reconciler"); + } else { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0].runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(seed.owner); + } + }); + + it.each(["pending", "resolved", "superseded", "unassigned"] as const)("requires authoritative reviewer recovery: %s", async state => { + const seed = await seedAbandonedCopyback(); + const reviewerId = randomUUID(), sourceRunId = randomUUID(), sourceResultId = randomUUID(), assessmentId = randomUUID(); + const decisionId = randomUUID(), interactionId = randomUUID(); + await db.insert(agents).values({ id: reviewerId, companyId, name: "Review agent", adapterType: "paperclip_runner", status: "running" }); + const [actingRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, nativeIssueId: seed.issueId, runtimeMode: "native", status: "succeeded", completionContractId: actingRun.completionContractId, createdAt: new Date(actingRun.createdAt.getTime() - 60_000) }); + await db.insert(nativeRunResults).values({ id: sourceResultId, companyId, issueId: seed.issueId, runId: sourceRunId, completionContractId: actingRun.completionContractId!, serverFingerprint: sourceResultId, schemaStatus: "accepted", resultJson: {}, canonicalSha256: sourceResultId }); + await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId: seed.issueId, runId: sourceRunId, contractId: actingRun.completionContractId!, resultId: sourceResultId, triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, policyVersion: "fixture", assessmentJson: {}, inputDigest: assessmentId }); + const [issue] = await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, seed.issueId)).returning(); + await db.insert(statusDecisions).values({ id: decisionId, companyId, issueId: seed.issueId, runId: sourceRunId, assessmentId, decisionVersion: 1, policyVersion: "fixture", fromStatus: "in_progress", toStatus: "in_review", reasonCode: "explicit_review", decisionJson: { projectedStatusVersion: issue.statusVersion }, decisionDigest: decisionId, applicationState: "applied" }); + await db.update(issues).set({ lastStatusDecisionId: state === "superseded" ? null : decisionId }).where(eq(issues.id, seed.issueId)); + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seed.issueId, kind: "request_confirmation", sourceRunId, addresseeAgentId: state === "unassigned" ? agentId : reviewerId, createdByAgentId: agentId, continuationPolicy: "wake_assignee", status: state === "resolved" ? "accepted" : "pending", ...(state === "resolved" ? { resolvedByRunId: seed.runId, resolvedByAgentId: reviewerId } : {}), payload: { version: 1, prompt: "Review the completed work.", target: { type: "custom", key: "native_completion_review", revisionId: decisionId } } }); + await db.update(heartbeatRuns).set({ agentId: reviewerId, contextSnapshot: { issueId: seed.issueId, nativeReviewInteractionId: interactionId, nativeReviewDecisionId: decisionId } }).where(eq(heartbeatRuns.id, seed.runId)); + if (state === "pending" || state === "resolved") { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" }); + expect((await db.select().from(issues).where(eq(issues.id, seed.issueId)))[0].assigneeAgentId).toBe(agentId); + } else { + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 }); + } + }); + + it.each(([ + ["workspace_finalizing", "monitor"], ["workspace_finalizing", "approval"], + ["retryable_failure", "monitor"], ["retryable_failure", "approval"], + ["result_accepted", "monitor"], ["result_accepted", "approval"], + ] as const).flatMap(([phase, wait]) => (["operator", "automatic"] as const).map(mode => [phase, wait, mode] as const)))("commits the saved yield from %s preserving its %s wait via %s recovery", async (phase, wait, mode) => { + const seed = await seedAbandonedCopyback(); + await db.update(nativeRunFinalizations).set({ phase, + ...(phase === "retryable_failure" ? { attempt: 1, nextAttemptAt: new Date(Date.now() - 60_000), + failureCode: "native_workspace_sync_out_failed" } : {}), + }).where(eq(nativeRunFinalizations.runId, seed.runId)); + await db.update(heartbeatRuns).set({ nativePhase: phase }).where(eq(heartbeatRuns.id, seed.runId)); + const workspaceId = randomUUID(); + const cwd = path.join(workspaceRoot, seed.runId); + await fs.mkdir(cwd); + await fs.writeFile(path.join(cwd, "saved.txt"), "accepted work"); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, + mode: "local", strategyType: "local_directory", name: "Retained workspace", cwd }); + const nextCheckAt = new Date(Date.now() - 60_000); // Already due, like the staging runs. + const policy = { monitor: { nextCheckAt: nextCheckAt.toISOString(), notes: "Check the existing PR." } }; + await db.update(issues).set({ executionWorkspaceId: workspaceId, + ...(wait === "monitor" ? { executionPolicy: policy, monitorNextCheckAt: nextCheckAt } : {}), + }).where(eq(issues.id, seed.issueId)); + const interactionId = randomUUID(); + if (wait === "approval") await db.insert(issueThreadInteractions).values({ + id: interactionId, companyId, issueId: seed.issueId, sourceRunId: seed.runId, + createdByAgentId: agentId, kind: "request_confirmation", status: "pending", + payload: { version: 1, prompt: "Approve the plan?", target: { type: "issue_document", + issueId: seed.issueId, key: "plan", revisionId: randomUUID() } }, + }); + const resultJson = { + result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", + summary: "Wait for the existing continuation.", + completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [], + remainingWork: [{ description: "Wait for the check or plan confirmation.", blocksCompletion: true }] }, + continuation: { kind: wait === "monitor" ? "monitor" : "response_wake", + summary: "Wait for the existing continuation.", idempotencyKey: "saved-wait" }, + evidence: [], verification: [], attentionRequests: [], artifacts: [] }, + terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", + runTerminalState: "succeeded", reportedWorkDisposition: "yielded" }, + }; + await db.update(nativeRunResults).set({ resultJson }).where(eq(nativeRunResults.runId, seed.runId)); + const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId)); + + // The real reconciler must remain fenced before operator confirmation. + await reconcileNativeFinalizations(db, [seed.runId]); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe(phase); + const [beforeConfirmation] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (mode === "operator") { + await resumeNativeWorkspaceAfterOwnerStop(seed); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toEqual(beforeConfirmation); + await reconcileNativeFinalizations(db, [seed.runId]); + } else { + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceFinalizationOwner: { ...seed.owner, worker } } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: seed.owner.token, worker, stopped: true, iat: 1, exp: 61 }); + try { await reconcileNativeFinalizations(db, [seed.runId]); } finally { probe.mockRestore(); } + } + + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toMatchObject({ phase: "committed", resultId: accepted.id }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0]).toMatchObject({ status: "succeeded", nativePhase: "committed" }); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId))).toEqual([accepted]); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)))[0].status).toBe("resolved"); + const [issue] = await db.select().from(issues).where(eq(issues.id, seed.issueId)); + expect(issue.status).toBe(wait === "monitor" ? "in_progress" : "in_review"); + if (wait === "monitor") { + expect(issue.monitorNextCheckAt).toEqual(nextCheckAt); + expect(issue.executionPolicy).toEqual(policy); + } else { + expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].status).toBe("pending"); + } + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, seed.issueId))).toHaveLength(1); + expect(await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.payload}->>'issueId' = ${seed.issueId}`)).toHaveLength(0); + expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, seed.runId), + eq(workspaceOperations.phase, "workspace_finalize"), eq(workspaceOperations.status, "succeeded")))).toHaveLength(1); + expect(await fs.readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("accepted work"); + }); + + it.each(["token", "company", "result", "newer owner", "missing confirmation", "malformed owner"])("rejects stale or unverified copyback recovery: %s", async changed => { + const seed = await seedAbandonedCopyback(); + if (changed === "malformed owner") { + const owner = { token: seed.ownerToken }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId)); + } + if (changed === "token") seed.ownerToken = randomUUID(); + if (changed === "company") seed.companyId = foreignCompanyId; + if (changed === "result") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (changed === "newer owner") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { ...seed.owner, token: randomUUID() } } }).where(eq(heartbeatRuns.id, seed.runId)); + if (changed === "missing confirmation") Object.assign(seed, { controllerAndCopybackStopped: false }); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeDefined(); + expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(0); + }); + + it("rejects a locally live owner even after its advisory connection is gone", async () => { + const seed = await seedAbandonedCopyback(); + const identity = await currentNativeControllerIdentity(); + const owner = { ...seed.owner, hostname: os.hostname(), pid: identity.pid, + processStartedAt: identity.processStartedAt.toISOString(), controllerBootId: identity.bootId }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId)); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + }); + + it("does not reopen an old result after a newer task execution exists", async () => { + const seed = await seedAbandonedCopyback(); + await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", nativeIssueId: seed.issueId }); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("no longer current"); + }); + + it("rejects stop confirmation while another controller holds the physical lock", async () => { + const seed = await seedAbandonedCopyback(); + await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${companyId}:${seed.runId}`}, 0))`); + await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active"); + }); + }); + it("reuses a successful export even if an old controller wrote a later failed barrier", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Late failed barrier" }); const [success] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId, @@ -311,6 +541,137 @@ describe("native workspace finalization recovery", () => { .toEqual({ acquired: true, value: "recovered" }); }); + it("automatically adopts a proven stopped deployment without another agent turn", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "fixture-project", environmentId: "fixture-env", serviceId: "fixture-service", deploymentId: "fixture-old" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" } }).where(eq(heartbeatRuns.id, seed.runId)); + await db.update(issueRecoveryActions).set({ fingerprint: nativeSha256({ runId: seed.runId, owner }), evidence: { runId: seed.runId, owner, preserved: "recovery evidence" } }).where(eq(issueRecoveryActions.id, seed.actionId)); + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + const proof = { v: 1 as const, iss: "paperclip-cloud" as const, aud: "paperclip-workspace-worker-stop/v1" as const, sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true as const, iat: 1, exp: 61 }; + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(proof); + try { + const work = vi.fn(async () => { + const [recovery] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)); + expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "resuming" }); + expect(recovery.evidence?.preserved).toBe("recovery evidence"); + return "saved workspace copied"; + }); + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: true, value: "saved workspace copied" }); + expect(probe).toHaveBeenCalledWith({ runId: seed.runId, ownerToken: owner.token, worker }); + expect(work).toHaveBeenCalledOnce(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined(); + expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toEqual(proof); + expect(run.runnerProfileJson?.preserved).toBe("workspace descriptor"); + const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); + expect(after).toEqual(before); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0); + } finally { probe.mockRestore(); } + }); + + it("retains the fence for unknown platform evidence and owner changes during verification", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(null); + const work = vi.fn(); + try { + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false }); + const [recovery] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, seed.issueId), eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: seed.runId, owner })))); + expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "operator_required" }); + const replacement = { ...owner, token: randomUUID() }; + probe.mockImplementation(async () => { + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: replacement } }).where(eq(heartbeatRuns.id, seed.runId)); + return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }; + }); + expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false }); + expect(work).not.toHaveBeenCalled(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(replacement); + } finally { probe.mockRestore(); } + }); + + it.each(["done", "cancelled", "execution replaced", "checkout replaced", "newer run", "queued legacy run", "reassigned", "result unbound", "result rejected", "run stopped"])("does not copy saved files after the task moves on during platform verification: %s", async changed => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const file = path.join(workspaceRoot, `${seed.runId}.txt`); + await fs.writeFile(file, "new task work"); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockImplementation(async () => { + if (changed === "done" || changed === "cancelled") await db.update(issues).set({ status: changed }).where(eq(issues.id, seed.issueId)); + if (changed === "execution replaced") await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, seed.issueId)); + if (changed === "checkout replaced") { + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded" }).returning(); + await db.update(issues).set({ checkoutRunId: other.id }).where(eq(issues.id, seed.issueId)); + } + if (changed === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, seed.issueId)); + if (changed === "result unbound") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId)); + if (changed === "result rejected") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.runId, seed.runId)); + if (changed === "run stopped") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seed.runId)); + if (changed === "newer run" || changed === "queued legacy run") { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + await db.insert(heartbeatRuns).values({ companyId, agentId, + runtimeMode: changed === "newer run" ? "native" : "legacy", + nativeIssueId: changed === "newer run" ? seed.issueId : null, + contextSnapshot: { issueId: seed.issueId }, status: changed === "newer run" ? "succeeded" : "queued", + createdAt: new Date(run.createdAt.getTime() + (changed === "newer run" ? 60_000 : -60_000)), + }); + } + return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }; + }); + const copyback = vi.fn(async () => fs.writeFile(file, "old sandbox work")); + try { + expect(await withNativeWorkspaceFinalizationOwnership(seed, copyback)).toEqual({ acquired: false }); + expect(copyback).not.toHaveBeenCalled(); + expect(await fs.readFile(file, "utf8")).toBe("new task work"); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)); + expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toBeUndefined(); + } finally { probe.mockRestore(); } + }); + + it("rechecks automatic task ownership at the copyback publication boundary", async () => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }); + const write = vi.fn(); + try { + await expect(withNativeWorkspaceFinalizationOwnership(seed, async ownership => { + await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, seed.issueId)); + await ownership.assertHeld(); + write(); + })).rejects.toThrow("native_workspace_finalization_ownership_lost"); + expect(write).not.toHaveBeenCalled(); + } finally { probe.mockRestore(); } + }); + + it.each(["closed", "superseded"])("does not start a workspace operation when the real reconciler selects a %s task", async changed => { + const seed = await seedAbandonedCopyback(); + const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" }; + const owner = { ...seed.owner, worker }; + const workspaceId = randomUUID(), cwd = path.join(workspaceRoot, seed.runId); + await fs.mkdir(cwd); + await fs.writeFile(path.join(cwd, "current.txt"), "new task work"); + await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, mode: "local", strategyType: "local_directory", name: "Current workspace", cwd }); + await db.update(issues).set({ executionWorkspaceId: workspaceId, + ...(changed === "closed" ? { status: "done" } : { executionRunId: null }), + }).where(eq(issues.id, seed.issueId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId)); + const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 }); + try { + await reconcileNativeFinalizations(db, [seed.runId]); + expect(probe).toHaveBeenCalledOnce(); + expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seed.runId))).toHaveLength(0); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe("workspace_finalizing"); + expect(await fs.readFile(path.join(cwd, "current.txt"), "utf8")).toBe("new task work"); + } finally { probe.mockRestore(); } + }); + it("surfaces an unverifiable foreign controller as board recovery without physical writes", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Foreign copyback controller" }); const owner = { token: randomUUID(), hostname: "other-controller.invalid", pid: 42, processStartedAt: new Date().toISOString() }; diff --git a/server/src/services/native-runtime/native-workspace-owner-recovery.ts b/server/src/services/native-runtime/native-workspace-owner-recovery.ts new file mode 100644 index 00000000000..4b3e19e3311 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-owner-recovery.ts @@ -0,0 +1,87 @@ +import os from "node:os"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { resumeWorkspaceFinalizationSchema } from "@paperclipai/shared"; +import { conflict } from "../../errors.js"; +import { persistActivity, publishActivity } from "../activity-log.js"; +import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js"; +import { evaluateNativeControllerTakeover } from "./native-restart-recovery.js"; + +const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; +const record = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +const changed = () => conflict("The recorded workspace owner is no longer current. Refresh the task and inspect its run.", { code: "workspace_owner_recovery_stale" }); + +/** Board-only admission after deployment-platform verification. A missing DB + * lock or an expired lease alone never proves physical copyback has stopped. */ +export async function resumeNativeWorkspaceAfterOwnerStop(input: { + db: Db; companyId: string; issueId: string; actionId: string; runId: string; + ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string; actorId: string; +}) { + const confirmation = resumeWorkspaceFinalizationSchema.parse({ + actionId: input.actionId, runId: input.runId, ownerToken: input.ownerToken, + controllerAndCopybackStopped: input.controllerAndCopybackStopped, stopEvidence: input.stopEvidence, + }); + const admitted = await input.db.transaction(async tx => { + // Share the physical copyback lock. Do not wait behind an active exporter + // and then apply a confirmation that described an earlier owner. + const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`); + if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); + // Match the status arbiter's coordinator -> issue -> run lock order. + const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); + const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.id, input.runId))).for("update"); + const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update"); + if (!issue || !run || !coordinator || !action || action.evidence?.runId !== run.id + || action.cause !== "native_workspace_finalization_owner_unverified" || action.ownerType !== "board" + || action.kind !== "active_run_watchdog" || !["active", "escalated"].includes(action.status) + || (coordinator.leaseOwner && (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt.getTime() > Date.now()))) throw changed(); + const result = await currentNativeWorkspaceRecoveryResult(tx as unknown as Db, { + companyId: input.companyId, issue, run, coordinator, + }); + if (!result) throw changed(); + const prior = record(run.runnerProfileJson?.[OWNER_KEY]); + const recordedOwner = record(action.evidence?.owner); + const stop = record(action.evidence?.workspaceOwnerStop); + const receipt = { runId: run.id, resultId: result.id, status: "queued" as const }; + if (run.runnerProfileJson?.[OWNER_KEY] == null && record(stop.owner).token === input.ownerToken + && stop.resultId === result.id) return { receipt, publication: null }; + if (typeof prior.hostname !== "string" || !prior.hostname + || !Number.isInteger(prior.pid) || Number(prior.pid) <= 0 + || typeof prior.processStartedAt !== "string" || !Number.isFinite(Date.parse(prior.processStartedAt)) + || prior.token !== input.ownerToken || recordedOwner.token !== input.ownerToken + || prior.hostname !== recordedOwner.hostname || prior.pid !== recordedOwner.pid + || prior.processStartedAt !== recordedOwner.processStartedAt + || prior.controllerBootId !== recordedOwner.controllerBootId) throw changed(); + // Reject a confirmation contradicted by a locally observable live owner, + // even if that owner's advisory connection has already disconnected. + if (prior.hostname === os.hostname()) { + const takeover = await evaluateNativeControllerTakeover({ owner: { + leaseOwner: input.ownerToken, leaseExpiresAt: new Date(0), + controllerPid: typeof prior.pid === "number" ? prior.pid : null, + controllerProcessStartedAt: typeof prior.processStartedAt === "string" ? new Date(prior.processStartedAt) : null, + }, now: new Date() }); + if (!takeover.allowed) throw conflict("The previous workspace controller is still active or its stop cannot be verified."); + } + const now = new Date(); + const proof = { owner: prior, resultId: result.id, actorId: input.actorId, + stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() }; + await tx.update(heartbeatRuns).set({ + runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now, + }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id), + sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${input.ownerToken}`)); + await tx.update(issueRecoveryActions).set({ evidence: { ...action.evidence, workspaceOwnerStop: proof }, + nextAction: "Workspace finalization is queued for the saved result. The agent will not repeat its work.", + wakePolicy: null, monitorPolicy: null, updatedAt: now, + }).where(eq(issueRecoveryActions.id, action.id)); + const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId, + actorType: "user", actorId: input.actorId, action: "issue.workspace_owner_stop_confirmed", + entityType: "issue", entityId: input.issueId, runId: input.runId, + details: { recoveryActionId: input.actionId, ...proof }, + }); + // Leave the result, coordinator, source lease, task status and wait paths + // untouched. The normal reconciler resumes copyback and arbitrates it. + return { receipt, publication }; + }); + if (admitted.publication) publishActivity(admitted.publication.publication); + return admitted.receipt; +} diff --git a/server/src/services/native-runtime/native-workspace-platform-proof.test.ts b/server/src/services/native-runtime/native-workspace-platform-proof.test.ts new file mode 100644 index 00000000000..fb3297ca80b --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-platform-proof.test.ts @@ -0,0 +1,34 @@ +import { generateKeyPairSync, sign } from "node:crypto"; +import { describe, expect, it, vi } from "vitest"; +import { verifyWorkspaceWorkerStopReceipt, requestWorkspaceWorkerStopProof, currentWorkspaceWorker } from "./native-workspace-platform-proof.js"; +const { privateKey, publicKey } = generateKeyPairSync("ed25519"); +const env = { PAPERCLIP_CLOUD_STACK_ID: "stack", PAPERCLIP_CLOUD_API_ORIGIN: "https://cloud.example", PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "fixture-token", PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [{ ...publicKey.export({ format: "jwk" }), kid: "test", alg: "EdDSA", use: "sig" }] }) }; +const worker = { provider: "railway" as const, projectId: "project", environmentId: "environment", serviceId: "service", deploymentId: "old-deployment" }; +const binding = { runId: "run", ownerToken: "owner", worker }; +const now = new Date("2099-01-01T00:00:00Z"); +function receipt(overrides = {}) { + const header = Buffer.from(JSON.stringify({ alg: "EdDSA", kid: "test", typ: "paperclip-workspace-worker-stop+jwt" })).toString("base64url"); + const payload = Buffer.from(JSON.stringify({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", ...binding, stopped: true, iat: now.getTime()/1000, exp: now.getTime()/1000+60, ...overrides })).toString("base64url"); + return `${header}.${payload}.${sign(null, Buffer.from(`${header}.${payload}`), privateKey).toString("base64url")}`; +} +describe("workspace platform stop proof", () => { + it("accepts only signed proof for the exact stack, owner, run and full stopped deployment", () => { + expect(verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, now)).toMatchObject({ stopped: true, worker }); + for (const invalid of [{ sub: "other-stack" }, { runId: "other-run" }, { ownerToken: "other-owner" }, { stopped: false }, { exp: now.getTime()/1000-1 }, { worker: { ...worker, deploymentId: "other" } }, { aud: "paperclip-cloud-control/v1" }]) { + expect(verifyWorkspaceWorkerStopReceipt(receipt(invalid), binding, env, now)).toBeNull(); + } + expect(verifyWorkspaceWorkerStopReceipt(receipt().slice(0,-5)+"wrong", binding, env, now)).toBeNull(); + }); + it("fails closed on live, unknown, unreachable and unsigned platform responses", async () => { + for (const response of [{ stopped: false }, { stopped: true }, { receipt: "forged" }]) { + const fetchImpl = vi.fn().mockResolvedValue(Response.json(response)); + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl })).toBeNull(); + } + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockRejectedValue(new Error("offline")) })).toBeNull(); + expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockResolvedValue(Response.json({ receipt: receipt() })) })).toMatchObject({ stopped: true }); + }); + it("records a complete platform identity, and does not guess legacy ownership", () => { + expect(currentWorkspaceWorker({ RAILWAY_PROJECT_ID: "project", RAILWAY_ENVIRONMENT_ID: "environment", RAILWAY_SERVICE_ID: "service", RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toEqual(worker); + expect(currentWorkspaceWorker({ RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toBeUndefined(); + }); +}); diff --git a/server/src/services/native-runtime/native-workspace-platform-proof.ts b/server/src/services/native-runtime/native-workspace-platform-proof.ts new file mode 100644 index 00000000000..04b452d1d76 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-platform-proof.ts @@ -0,0 +1,65 @@ +import { verify } from "node:crypto"; +import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js"; +import { getCloudStackContext } from "../cloud-instance.js"; + +/** Deployment identity covers the controller and every orphaned tar/Git child. */ +export type WorkspaceWorker = { + provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string; +}; +export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker }; +export type WorkspaceWorkerStopProof = WorkspaceStopBinding & { + v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string; + stopped: true; iat: number; exp: number; +}; +function validWorker(worker: unknown): worker is WorkspaceWorker { + if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false; + const value = worker as Record; + return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key => + typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200); +} +export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined { + const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID]; + if (ids.some(id => !id?.trim())) return undefined; + return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! }; +} +export function verifyWorkspaceWorkerStopReceipt( + receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(), +): WorkspaceWorkerStopProof | null { + try { + if (receipt.length > 16_000 || !validWorker(binding.worker)) return null; + const parts = receipt.split("."); + if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null; + const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8")); + if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null; + if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null; + const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof; + const seconds = Math.floor(now.getTime()/1000); + if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1" + || !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true + || proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken + || !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30 + || proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60 + || !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null; + return proof; + } catch { return null; } +} +/** Unsupported platforms, legacy receipts and unavailable evidence never authorize takeover. */ +export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: { + env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date; +} = {}): Promise { + const env = options.env ?? process.env; + const context = getCloudStackContext(env); + const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim(); + if (!context?.stackId || !context.cloudOrigin || !token || !validWorker(binding.worker)) return null; + try { + const origin = new URL(context.cloudOrigin); + if (origin.protocol !== "https:" || origin.username || origin.password) return null; + const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, { + method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" }, + body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) return null; + const payload = await response.json() as { receipt?: unknown }; + return typeof payload.receipt === "string" ? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null; + } catch { return null; } +} diff --git a/server/src/services/native-runtime/native-workspace-recovery-binding.ts b/server/src/services/native-runtime/native-workspace-recovery-binding.ts new file mode 100644 index 00000000000..20c4ee6861f --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-recovery-binding.ts @@ -0,0 +1,36 @@ +import { and, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { getNativeReviewAssignment } from "./native-review-participant.js"; + +/** Both operator and platform recovery must still own the task's accepted work. + * Call with coordinator -> issue -> run rows locked in a short transaction. */ +export async function currentNativeWorkspaceRecoveryResult(db: Db, input: { + companyId: string; + issue: typeof issues.$inferSelect; + run: typeof heartbeatRuns.$inferSelect; + coordinator: typeof nativeRunFinalizations.$inferSelect; +}) { + const { companyId, issue, run, coordinator } = input; + if (issue.companyId !== companyId || run.companyId !== companyId || coordinator.companyId !== companyId + || run.nativeIssueId !== issue.id || coordinator.issueId !== issue.id || coordinator.runId !== run.id + || run.runtimeMode !== "native" || run.status !== "running" + || !["result_accepted", "workspace_finalizing", "retryable_failure"].includes(coordinator.phase) + || !coordinator.resultId || ["done", "cancelled"].includes(issue.status) + || issue.executionRunId !== run.id || (issue.checkoutRunId && issue.checkoutRunId !== run.id)) return null; + if (issue.assigneeAgentId !== run.agentId && !await getNativeReviewAssignment(db, { + companyId, issueId: issue.id, agentId: run.agentId, + contextSnapshot: run.contextSnapshot, actingRunId: run.id, issueExecutionRunId: issue.executionRunId, + allowResolvedByRunId: run.id, + })) return null; + const [result] = await db.select().from(nativeRunResults).where(and( + eq(nativeRunResults.id, coordinator.resultId), eq(nativeRunResults.companyId, companyId), + eq(nativeRunResults.issueId, issue.id), eq(nativeRunResults.runId, run.id), eq(nativeRunResults.schemaStatus, "accepted"), + )); + if (!result || result.completionContractId !== run.completionContractId) return null; + const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, run.id), + or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), + or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])), + )).limit(1); + return newer ? null : result; +} diff --git a/ui/src/api/issues.ts b/ui/src/api/issues.ts index 314064ea27a..f3a56c5ef63 100644 --- a/ui/src/api/issues.ts +++ b/ui/src/api/issues.ts @@ -283,6 +283,8 @@ export const issuesApi = { `/issues/${id}/stalled-review-decision`, data, ), + resumeWorkspaceFinalization: (id: string, data: { actionId: string; runId: string; ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string }) => + api.post<{ runId: string; resultId: string; status: "queued" }>(`/issues/${id}/recovery-actions/resume-workspace-finalization`, data), retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) => api.post<{ runId: string; resultId: string; leaseId: string; status: "queued" }>(`/issues/${id}/recovery-actions/retry-workspace-export`, data), resolveRecoveryAction: ( diff --git a/ui/src/components/IssueRecoveryActionCard.test.tsx b/ui/src/components/IssueRecoveryActionCard.test.tsx index 88e6cc7b133..cd513226984 100644 --- a/ui/src/components/IssueRecoveryActionCard.test.tsx +++ b/ui/src/components/IssueRecoveryActionCard.test.tsx @@ -1161,3 +1161,10 @@ it("requires export retry for ordinary restoration while keeping explicit board expect(document.body.textContent).not.toContain("Send for review"); expect(document.body.textContent).toContain("False positive, done"); }); + +it("requires owner-stop recovery even for board false-positive dispositions", () => { + const node = render( {}} canFalsePositive />); + expect(node.querySelector("[data-testid='recovery-action-resolve-trigger']")).toBeNull(); + expect(node.textContent).not.toContain("False positive"); +}); diff --git a/ui/src/components/IssueRecoveryActionCard.tsx b/ui/src/components/IssueRecoveryActionCard.tsx index beb84076e1b..e61adbbb966 100644 --- a/ui/src/components/IssueRecoveryActionCard.tsx +++ b/ui/src/components/IssueRecoveryActionCard.tsx @@ -1067,6 +1067,7 @@ export function IssueRecoveryActionCard({ } satisfies Record)[cardState]; const visibleResolveOptions = RESOLVE_OPTIONS.filter((option) => { + if (action.cause === "native_workspace_finalization_owner_unverified") return false; if (isNativeWorkspaceExportRepairCause(action.cause) && ["todo", "done", "in_review"].includes(option.outcome)) return false; if (option.outcome === "todo" && requiresExecutionReconciliation(action.cause)) return false; if (option.boardOnly && !canFalsePositive) return false; diff --git a/ui/src/components/WorkspaceExportRecovery.test.tsx b/ui/src/components/WorkspaceExportRecovery.test.tsx index 10f4c243eac..28a2409d2f1 100644 --- a/ui/src/components/WorkspaceExportRecovery.test.tsx +++ b/ui/src/components/WorkspaceExportRecovery.test.tsx @@ -1,72 +1,44 @@ // @vitest-environment jsdom import { act } from "react"; import { createRoot, type Root } from "react-dom/client"; -import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { IssueRecoveryAction } from "@paperclipai/shared"; import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery"; -const retry = vi.hoisted(() => vi.fn()); -vi.mock("../api/issues", () => ({ issuesApi: { retryWorkspaceExport: retry } })); -const action = { id: "action", cause: "native_workspace_sync_out_retry_exhausted", status: "active", updatedAt: "2026-01-01T00:00:00Z", ownerType: "board", evidence: { runId: "run" }, wakePolicy: null } as unknown as IssueRecoveryAction; +const action = { cause: "native_workspace_finalization_owner_unverified", status: "escalated", ownerType: "board", evidence: { runId: "run", owner: { token: "owner", hostname: "secret-host", pid: 2 } }, wakePolicy: null } as unknown as IssueRecoveryAction; describe("WorkspaceExportRecovery", () => { - let root: Root, container: HTMLDivElement, client: QueryClient; - const onQueued = vi.fn(); - beforeEach(() => { - retry.mockReset().mockResolvedValue({ status: "queued" }); onQueued.mockReset(); - container = document.createElement("div"); document.body.append(container); root = createRoot(container); - client = new QueryClient({ defaultOptions: { mutations: { retry: false } } }); - }); - afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); }); - async function mount(overrides: Partial[0]> = {}) { - await act(async () => root.render()); + let root: Root, container: HTMLDivElement; + beforeEach(() => { container = document.createElement("div"); document.body.append(container); root = createRoot(container); }); + afterEach(async () => { await act(async () => root.unmount()); container.remove(); }); + async function mount(value: IssueRecoveryAction | null) { + await act(async () => root.render()); + expect(container.querySelector("textarea, button, details, input")).toBeNull(); + expect(container.textContent).not.toContain("secret-host"); } - async function enterNote() { - const textarea = container.querySelector("textarea")!; - await act(async () => { - Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(textarea, "Restored provider connectivity while preserving all saved files."); - textarea.dispatchEvent(new Event("input", { bubbles: true })); - }); - } - async function submit() { - await act(async () => container.querySelector("button")!.click()); - await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); }); - } - it("requires a repair note and submits only the exact recorded run", async () => { - await mount(); expect(container.querySelector("button")!.disabled).toBe(true); - await enterNote(); await submit(); - expect(retry).toHaveBeenCalledWith("issue", { actionId: "action", runId: "run", repairNote: "Restored provider connectivity while preserving all saved files." }); - expect(onQueued).toHaveBeenCalledOnce(); - expect(container.querySelector('[role="status"]')?.textContent).toContain("agent will not repeat its work"); + it("shows a quiet status while the harness restores work", async () => { + await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerRecovery: { state: "resuming" } } }); + expect(container.querySelector('[role="status"]')?.textContent).toBe("Restoring saved work…"); }); - it("offers repair again when a queued export publishes a new permanent failure", async () => { - await mount(); await enterNote(); await submit(); - expect(container.querySelector("button")).toBeNull(); - await mount({ action: { ...action, updatedAt: "2026-01-01T00:01:00Z", wakePolicy: null } }); - expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); + it("does not mistake platform identity for confirmed recovery", async () => { + await mount({ ...action, evidence: { ...action.evidence, owner: { ...(action.evidence.owner as object), worker: { provider: "railway", deploymentId: "old" } } } }); + expect(container.textContent).toContain("operator assistance"); }); - it("keeps unavailable-sandbox errors actionable inline", async () => { - retry.mockRejectedValueOnce(new Error("Resume the retained sandbox before retrying.")); await mount(); - await enterNote(); await submit(); - expect(container.querySelector('[role="alert"]')?.textContent).toContain("Resume the retained sandbox"); + it("states that an operator is needed for legacy receipts", async () => { + await mount(action); + expect(container.textContent).toContain("Workspace recovery needs operator assistance"); }); - it("does not offer the action without runtime access", async () => { - await mount({ canManage: false }); expect(container.querySelector("button")).toBeNull(); + it("keeps export repair out of the task", async () => { + const cause = "native_workspace_sync_out_retry_exhausted"; + await mount({ ...action, cause }); + expect(container.textContent).toContain("operator assistance"); }); - it("offers saved-result export after transient retries are exhausted without claiming an unsafe link", async () => { - await mount({ action: { ...action, cause: "native_workspace_sync_out_retry_exhausted" } }); - expect(container.querySelector("button")?.textContent).toBe("Retry workspace export"); - expect(container.textContent).toContain("export failure"); - expect(container.textContent).not.toContain("unsafe link"); - await enterNote(); await submit(); - expect(retry).toHaveBeenCalledWith("issue", expect.objectContaining({ runId: "run" })); + it("removes the status when recovery resolves, including updates in place", async () => { + await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerStop: {} } }); + expect(container.textContent).toBe("Restoring saved work…"); + await mount({ ...action, status: "resolved" }); expect(container.textContent).toBe(""); + await mount(null); expect(container.textContent).toBe(""); }); - it("never asks users to repair historical unsafe exports", async () => { - await mount({ action: { ...action, cause: "native_workspace_sync_out_unsafe_archive" } }); - expect(container.textContent).toBe(""); - expect(retry).not.toHaveBeenCalled(); - }); - it("does not offer a second retry while the same export is queued", async () => { - await mount({ action: { ...action, wakePolicy: { kind: "resume_native_run" } } }); - expect(container.querySelector('[role="status"]')?.textContent).toContain("Export is queued"); expect(container.querySelector("button")).toBeNull(); + it("does not show unrelated or historical unsafe-export notices", async () => { + await mount({ ...action, cause: "no_live_path", evidence: {} }); expect(container.textContent).toBe(""); + await mount({ ...action, cause: "native_workspace_sync_out_unsafe_archive" }); expect(container.textContent).toBe(""); }); }); diff --git a/ui/src/components/WorkspaceExportRecovery.tsx b/ui/src/components/WorkspaceExportRecovery.tsx index 98376816817..a5b3dcb45e8 100644 --- a/ui/src/components/WorkspaceExportRecovery.tsx +++ b/ui/src/components/WorkspaceExportRecovery.tsx @@ -1,45 +1,17 @@ -import { useId, useState } from "react"; -import { useMutation } from "@tanstack/react-query"; import type { IssueRecoveryAction } from "@paperclipai/shared"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; -import { issuesApi } from "../api/issues"; -import { cn } from "../lib/utils"; -import { Button } from "./ui/button"; -import { Label } from "./ui/label"; -import { Textarea } from "./ui/textarea"; +import { asObject, cn } from "../lib/utils"; -/** Copyback repair preserves the accepted result and never wakes the provider. */ -export function WorkspaceExportRecovery({ issueId, action, canManage, onQueued, className }: { - className?: string; - issueId: string; action: IssueRecoveryAction | null; canManage: boolean; onQueued: () => void; +/** Infrastructure repair belongs to the harness/operator, never a task form. */ +export function WorkspaceExportRecovery({ action, className }: { + className?: string; action: IssueRecoveryAction | null; }) { - const noteId = useId(); - const [repairNote, setRepairNote] = useState(""); - const [queuedActionVersion, setQueuedActionVersion] = useState(null); - const retry = useMutation({ - mutationFn: () => issuesApi.retryWorkspaceExport(issueId, { - actionId: action!.id, runId: action!.evidence.runId as string, repairNote: repairNote.trim(), - }), - onSuccess: () => { setQueuedActionVersion(String(action!.updatedAt)); onQueued(); }, - }); - if (!action || !isNativeWorkspaceExportRepairCause(action.cause) || action.ownerType !== "board" + const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified"; + if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board" || !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null; - const queued = queuedActionVersion === String(action.updatedAt) || action.wakePolicy?.kind === "resume_native_run"; - return
-

Workspace export needs repair

- {queued ?

Export is queued for the saved result. The agent will not repeat its work.

: <> -

{"Automatic workspace export retries stopped. Inspect the export failure, restore provider or destination availability, and preserve the saved files in the retained sandbox. Retry export here when the cause is resolved."}

- {canManage ? <> - -