Merge master recovery updates into Codex defaults

Preserve current recovery and controller preparation behavior, both Storybook histories, and the reviewed Codex onboarding source. Targeted controls and direct TypeScript checks pass; exact-head hosted and live qualification remain required.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-13 10:33:51 -05:00
commit 448db859e3
38 files changed
+1505 -193

No files matched your search

+84 -2
View File
@@ -35,6 +35,46 @@ leave tar/Git children alive, so incomplete copyback requires operator stop
verification even on the same host. PID reuse is checked against its recorded
start time rather than trusted by PID alone.
## Automatic cloud recovery
New Railway copyback receipts record the exact project, environment, service and
deployment ID. On takeover, the existing finalization reconciler asks the cloud
harness to inspect that deployment. The harness resolves the stack's owned
resources independently and issues a short-lived Ed25519 receipt only when the
exact deployment is `REMOVED` with `deploymentStopped: true`. This covers all
processes in the deployment, including orphaned copyback children. Missing,
truncated, live or unavailable provider evidence leaves the physical fence intact.
The tenant verifies the configured harness key, disjoint receipt audience, stack,
run, owner token, deployment identity and expiry. It records proof and an activity
entry when replacing only the exact owner under the existing advisory lock. The
normal reconciler resumes the accepted result; it does not create a provider turn
or change approval/monitor waits. A changed owner during verification fails closed.
Stop proof alone does not authorize old work to overwrite a current task. After
the provider read, automatic recovery uses the operator path's shared binding
checks under short coordinator, issue and run row locks, atomically with the
owner claim. The task must still be open, assigned to the run or its authoritative
review participant, and bound to that running native execution and accepted
result. A different checkout, newer execution or queued/running legacy execution
prevents takeover. Publication checks repeat this validation; row locks are never
held across physical copyback I/O.
The harness endpoint is `POST /v1/tenant/workspace-worker-stop`. The fleet-wide
server token authorizes a read request, never stop evidence: responses must be
signed by the harness. This endpoint requires the companion cloud harness change.
Deploying only the application keeps unknown owners fenced. No cloud provider
credentials are delivered to tenant applications.
The task view shows only “Restoring saved work…” after the server confirms recovery
or an operator queues it,
then removes the notice when commitment resolves recovery. It exposes no process
identifiers, shutdown form or manual resume control. Legacy receipts without
platform identity and unsupported platforms say operator assistance is needed;
operators use the documented exact-owner API below after independent stop proof.
The three diagnosed historical receipts do not gain a deployment identity merely
by upgrading. Never infer one from elapsed time or a hostname.
## Unverified copyback after controller replacement
Generic orphan recovery does not turn a missing provider PID into `process_lost`
@@ -66,6 +106,48 @@ exact prior controller **and its copyback subprocesses** have stopped. Retain th
sandbox, accepted native result, and workspace descriptor. Do not run a new
provider turn, delete workspace contents, or relax archive confinement.
For legacy receipts, an authorized company owner, company administrator or
instance administrator records independently verified platform evidence through
the operator API. The task has no shutdown form or resume control. The API is
`POST /api/issues/:id/recovery-actions/resume-workspace-finalization` with:
```json
{
"actionId": "<recovery-action-id>",
"runId": "<run-id>",
"ownerToken": "<owner-token-from-recovery-evidence>",
"controllerAndCopybackStopped": true,
"stopEvidence": "<deployment-platform evidence for this controller and its children>"
}
```
This is an operator attestation, not an automatic deployment-platform probe.
The separate `runtime:confirm_owner_stop` authorization action limits this
attestation to active company owners or administrators, the trusted local board,
or an instance administrator. Ordinary runtime managers and agents cannot release
the physical owner receipt. Missing role data keeps the client notice read-only.
A
missing advisory connection alone is insufficient. Admission rejects an active
copyback lock, an observably live local controller, a changed owner, a newer run,
or an unbound result. It records the exact owner and operator evidence atomically
with releasing the receipt. The accepted result, workspace descriptor, source
lease, task status, monitors, and approval waits are unchanged. The ordinary
reconciler performs copyback and applies the saved result through the status
arbiter. No provider wake is enqueued. The recovery action remains visible until
finalization commits. Admission supports saved-result copyback from `result_accepted`,
`workspace_finalizing`, and `retryable_failure`; an abandoned retry does not need
a replacement run. The existing retry schedule remains unchanged.
Repeating the same confirmation before reconciliation is
idempotent; it cannot release a newer owner. Generic recovery resolution, including
false-positive dispositions, cannot dismiss this hold or bypass saved-result
finalization.
Repeated sweeps preserve these copyback-specific instructions, including for
older actions whose recovery budget was already exhausted. They must never
recommend starting a replacement provider run for this hold.
For older servers without this action, the maintenance fallback follows.
After that platform verification, use a database maintenance transaction to
release only the exact receipt shown by the recovery action. Replace the four
placeholders with the action's company, run, token, and source issue. The advisory
@@ -138,8 +220,8 @@ A board member with runtime management access can complete the saved result:
the export failure while preserving all user files. Restore transport access
or other provider prerequisites before retrying. Extending the sandbox
retention window can be necessary during this repair.
3. In the task's **Workspace export needs repair** notice, describe the repair and
choose **Retry workspace export**. The equivalent board API is
3. Use the operator API to record the repair and request export-only retry. The
task shows operator assistance is needed, without an infrastructure form. The board API is
`POST /api/issues/:id/recovery-actions/retry-workspace-export` with
`{ "actionId": "<recovery-action-id>", "runId": "<run-id>", "repairNote": "<repair and preservation evidence>" }`.
4. Confirm the same run commits, the saved result determines the task outcome,
+18
View File
@@ -161,6 +161,24 @@ This document also holds three local instrumentation contracts: native runner
traces, sandbox startup traces, and sandbox duplex transport instrumentation.
Those sections follow below.
## Controller preparation phases
With an operator OTLP endpoint, `paperclip.heartbeat-preparation` emits active
`heartbeat.prepare.<phase>` spans around environment selection, execution
configuration, session preparation, workspace preparation, GitHub environment
and launcher preparation, run output initialization, workspace validation,
runtime services, instruction files, and runtime selection. Database and HTTP
instrumentation can attach child spans during each awaited operation.
These spans exist before provider dispatch, including setup failures. Each has
only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal
characters of the run UUID's SHA-256 hash. Failed operations set error status
without recording exception messages, paths, inputs, or returned content.
Search by that hash without a root-name filter to include these preparation
traces; the provider lifecycle trace is created later and can have a different
trace ID. Without an endpoint, tracing remains a no-op. These spans add no
first-party Telemetry events or database run-log writes.
## Native Runner Trace Spans
Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is
@@ -1660,6 +1660,12 @@ The sanitized report is uploaded and independently download-verified as attachme
The human requested that #15683 be finished and merged normally. A separate incremental $100 ceiling and as many bounded messages as needed are explicitly approved. The previous four-message/$25 campaign and original$250 ledger retain all unknown-charge and cleanup holds; they are not refunded or reassigned. Reuse local3136 and owned staging stack7998; preserve their data and receipts. Root owns actual-master reconciliation, exact-head checks/review, ordinary Mac/managed onboarding, checked task/follow-up and normal protected merge. Claude owner retains dependent branches and separate allowance. No production deployment or provisioning-default promotion is authorized. No timer is recreated.
Current-master integration includes `833769ab557255b11dd8f650f43ed7333653f5e4`, then `e3029c632b26b68595adc1a704a387833b3cefcb`. The ordinary merge retains the published Codex history and prerequisite squash history. Its audited logical source delta preserves master fixes, the prepared managed Connect qualification, fresh scoped discovery, and existing first-task eval assertions. The remaining merge conflicts concern Storybook indexes and dated reviews, not execution code. Independent review of both source compositions found no concrete P1/P2 regression; source review does not establish live acceptance.
On the integrated checkout, two focused Codex runs pass 261 tests each across onboarding, runner selection, setup, editing, imports and readiness. Server and UI direct TypeScript checks pass after JavaScript-only shared/DB/adapter-utils/SDK preparation. All token gates pass. New master recovery controls initially report a missing declared trace test dependency and a sandbox-denied PostgreSQL listener; the four unaffected suites pass 74 cases. Isolated installation of the exact declared trace dependencies and an authorized temporary listener allow the two affected suites to pass all 60 cases. No tests are disabled, no lockfile or production policy is changed, and no local native/Docker build occurs. Full canonical workspace checks remain assigned to fresh hosted CI.
Fresh-source remote package qualification, ordinary Apple Silicon onboarding and immutable managed staging onboarding remain pending. The previous canary60bc and stagingd67 receipts stay attributed to those sources. The new budget record reserves resources60/provider30/cleanup10 dollars and retains all prior unknown holds. The supported private Cloud skill and CLI are read from exact private master `4f3a63ad6f3f84c87e0e129da35909d8ccc7ffb0`; a deliberate pre-merge Core preview is separate from actual private Cloud distribution. Freeze the final reviewed product source before building; deploy only the owned pinned QA stack, with `set_default=false`.
Current published Codex head1a0ca662 is draft with green historical checks. Actual master833769ab is newer than merged prerequisite d67 and release repair60bc. Prepared private onboarding/Storybook work will be reconciled by source, without claiming its private/historical checks as final qualification. Native Connect correction is already in the Codex slice. Final managed defaults require the saved immutable OCI environment probe owned by ordinary Connect. Existing suites/fixtures/assertions/reporting are reused; no duplicate grader or silent Legacy fallback is added. All heavy builds stay remote, Codex payloads remain external official npm dependencies, and saved agents retain runners.
Next action: preserve/commit current evidence, integrate actual master plus reviewed prepared corrections, run targeted checks, freeze/push candidate, require fresh fullCI/review, then qualify ordinary creation/onboarding on Mac and the one staging target before protected squash merge.
+23 -6
View File
@@ -129,23 +129,40 @@ Review: `baseline`.
API: `present` — representative operation IDs:
- `POST /api/issues/{id}/recovery-actions/retry-workspace-export`.
- `POST /api/issues/{id}/recovery-actions/resume-workspace-finalization`.
- `POST /api/execution-workspaces/{id}/reconcile-branch`.
Agent access: `restricted` — Board/runtime management APIs; agent reads diagnosis and
performs permitted workspace work.
Policy: Task/run visibility, run ownership and source/action policy; workspace export
retry and some operator controls require board.
Policy: Export retry and branch reconciliation require board runtime management.
Physical-owner stop confirmation uses the separate `runtime:confirm_owner_stop`
action. It requires active company owner/admin membership, trusted local board
access, or instance administration. Agents and ordinary runtime managers are denied.
The exact source task, running native run, accepted result and owner receipt must
match; active copyback locks, live local writers, and newer runs prevent admission.
An authorized administrator must first verify through the deployment platform that
the exact controller and all copyback children stopped, then attest that evidence.
Elapsed time, an expired coordinator lease or a lost advisory connection is not proof.
Only the exact owner receipt is released; normal reconciliation performs export and
commitment without another provider turn or bypassing existing task waits.
Evidence: `source-only` — [implementation](../../server/src/routes/issues.ts#L9478) at
the reviewed commit above.
Evidence: `source-only` — [route](../../server/src/routes/issues.ts),
[administrator authorization](../../server/src/services/authorization.ts), and
[owner admission](../../server/src/services/native-runtime/native-workspace-owner-recovery.ts)
reviewed for this change. Local authorization, route, UI and finalization regressions
cover member/agent denial, exact-owner admission and the real reconciler preserving
monitor and approval waits. This is not live provider-stop proof.
Verification: `not-run` — assert the documented agent denial with an owned test fixture;
Verification: `not-run` for live agent/deployment execution — assert the documented agent denial with an owned test fixture;
verify any listed agent request/result path separately. A delegated operation needs its
own policy and access review.
Gaps: Export retry and branch reconciliation require board runtime:manage at
their HTTP routes. Native `call_api` preserves these denials.
their HTTP routes. The exact-owner operator fallback additionally requires
runtime:confirm_owner_stop. Native call_api preserves these agent denials.
Automatic managed-cloud recovery uses independently verified, signed platform
proof through the finalization reconciler; agents cannot attest owner shutdown.
## `unknown-outcome`
+27 -1
View File
@@ -119,7 +119,22 @@ Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNoti
tests pending/error/gated states and waiting for the real retry acknowledgement.
The recovery-card suite covers workspace divergence, reissue, reconcile, and
confirmation. [Workspace export](../../ui/src/components/WorkspaceExportRecovery.test.tsx)
covers its separate repair surface.
covers quiet restoration, legacy/operator assistance and resolved states.
For an abandoned native copyback owner, the harness verifies the exact stopped
deployment and the reconciler restores saved work automatically. Task users see
no process details or shutdown form. The documented operator API handles older
receipts after independent platform verification, through a separate administrator
gate. The [authorization regressions](../../server/src/__tests__/authorization-service.test.ts)
and [route regressions](../../server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts)
cover administrator admission, ordinary-member and agent denial, suspended and
foreign membership, missing role data, and stale cloud admin rows.
The [native workspace finalizer suite](../../server/src/services/native-runtime/native-workspace-finalizer.test.ts)
runs the real reconciler with a valid saved yield from `result_accepted`,
`workspace_finalizing`, and `retryable_failure`. It verifies physical fencing
before verified stop, automatic and operator recovery, successful export and commitment, hold resolution,
unchanged saved results, retained approval/monitor waits, and no new provider wake.
These local regressions do not prove a deployment platform stopped a foreign writer.
Manual: reproduce an exhausted disposable action. Click once, verify pending
state prevents repeated clicks, and inspect a successful continuation or an
@@ -128,6 +143,17 @@ repair, record the compared branches/revisions and preserve uncommitted work;
confirm the repair's actual result before resuming. Check the denied and stale
action paths. A changed card label is insufficient proof of workspace integrity.
For an abandoned native controller, follow the [stop-verification procedure](../../doc/native-workspace-finalization-recovery.md)
with a disposable deployment. A new platform-bound owner recovers automatically
only after the harness signs proof that the exact deployment fully stopped.
Legacy owners require independently verified stop evidence through the operator
API; the task has no shutdown form. Confirm
the original accepted result commits, the recovery hold resolves, saved files
remain intact, and existing approval waits or monitor checks remain. No new agent
turn should start. Repeat with a retry-phase owner and an ordinary runtime manager;
the first can resume through automatic proof or authorized operator confirmation,
while the second cannot use the operator API.
### `post-stop-message`
Automated: [legacy continuation](../../server/src/services/recovery/legacy-continuation.test.ts),
+6 -4
View File
@@ -4,7 +4,7 @@
[Contract and maintenance](README.md) · [Review findings](REVIEW.md)
224 story files; 2233 indexed stories. Counts describe inventory, not passed tests or complete feature coverage.
226 story files; 2242 indexed stories. Counts describe inventory, not passed tests or complete feature coverage.
Status describes product alignment; implementation describes the rendered subject; boundary describes simulated data. Sources and limitations are in catalog.json. Related capabilities cross-list one canonical story file.
@@ -336,10 +336,12 @@ Stopped work, workspace repair, bounded continuation, and reconnect.
| [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | current / production / fixtures | 9 / 3 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) | `recovery/bounded-continuation` | unreviewed / production / fixtures | 1 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite. |
| [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) | `recovery/diagnosis`, `recovery/source-action` | unreviewed / production / fixtures | 5 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 8 / 4 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. |
| [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) | `recovery/workspace` | unreviewed / production / fixtures | 2 / 0 | Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Master reconciliation inspected assignment-history recipient suggestions and manual-selection retention; existing stories do not establish the new assignment-history behavior. Expanded upstream verification recipes were inspected; the existing fixture scenarios do not prove the added backend handoff, workspace, recovery or authorization contracts. |
| [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) | `recovery/workspace`, `recovery/source-action` | current / production / fixtures | 6 / 2 | Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests. |
| [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) | `recovery/workspace`, `recovery/source-action` | unreviewed / mixed / fixtures | 3 / 0 | Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance. |
## Apps, channels, and extensions
@@ -668,9 +670,9 @@ References are navigation links, not proof of parity. An absent link means no ca
- `execution-environments/images`: No cataloged story.
- `recovery/diagnosis`: [Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards](../../ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx) (unreviewed), [Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab](../../ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice](../../ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx) (unreviewed), [Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page](../../ui/storybook/stories/secrets/credential-access/02-task-page.stories.tsx) (unreviewed), [Projects and execution/Recovery/Task chat/Recovery notice](../../ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Tasks/Execution recovery](../../ui/storybook/stories/recovery/execution-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue/Blocked notice](../../ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx) (unreviewed), [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff](../../ui/storybook/stories/tasks/successful-run-handoff.stories.tsx) (unreviewed)
- `recovery/bounded-continuation`: [Projects and execution/Recovery/Tasks/Model capacity retry](../../ui/storybook/stories/recovery/model-capacity-retry.stories.tsx) (unreviewed), [Projects and execution/Recovery/Product/Issue Scheduled retry surfaces](../../ui/storybook/stories/recovery/scheduled-retry.stories.tsx) (unreviewed)
- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed)
- `recovery/source-action`: [Projects and execution/Recovery/Design previews/Recovery notice](../../ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx) (current), [Projects and execution/Recovery/Paperclip/Source Issue Recovery](../../ui/storybook/stories/recovery/source-issue-recovery.stories.tsx) (unreviewed), [Projects and execution/Recovery/Paperclip/Source-resolved Fold](../../ui/storybook/stories/recovery/source-resolved-fold.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed)
- `recovery/exhaustion`: No cataloged story.
- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed)
- `recovery/workspace`: [Projects and execution/Recovery/Workspace branch recovery/01 Repair card](../../ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace branch recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx) (unreviewed), [Projects and execution/Recovery/Workspace finalization recovery/01 Notice](../../ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx) (current), [Projects and execution/Recovery/Workspace finalization recovery/02 Task journey](../../ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx) (unreviewed)
- `recovery/unknown-outcome`: No cataloged story.
- `recovery/user-continuation`: No cataloged story.
- `recovery/reconnect`: [Projects and execution/Recovery/App/Connection recovery](../../ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx) (unreviewed)
+103
View File
@@ -274,6 +274,35 @@ Reviewed the same twelve workspace-recipe-dependent entries after extending the
Reviewed the twelve unchanged workspace recipe subjects and fixtures after correcting implicit default selection. Task creation and controller classification now share configured-policy, primary, creation-time and ID precedence. Explicit choices and retained denial markers still suppress controller hooks. Project strategy fixtures show configuration, while runtime status and file/browser/artifact/service fixtures do not execute source scripts or prove this backend ordering. Refreshed only their reviewed recipe fingerprints and retained all classifications and stable IDs. No browser render, play, screenshot, Storybook build or live runtime acceptance was performed.
## Workspace finalization UI refinement — 2026-10-13
Source base: `f07efa2802`, followed by scoped edits to the production recovery notice and its two story suites. Reviewed `recovery/workspace` and `recovery/source-action`; stable story IDs and administrator admission remain unchanged. The notice leads with saved work, discloses controller metadata, explains evidence/confirmation requirements, and distinguishes queue acknowledgement from completion. A stale controller error preserves the note and clears confirmation so the operator must verify again. The task composition uses viewer identity for human bubbles and omits a routine status badge.
MacOS in-app browser inspection passed for the task journey at desktop and 390px mobile widths: controller disclosure with Enter, evidence without confirmation remaining disabled, explicit confirmation enabling submission, and a simulated queued result in place. The `owner-changed` story's play assertions passed; the friendly error was visible, the note remained, confirmation cleared and the action was disabled. The `long-controller-name` story rendered at 390px without horizontal overflow. The mobile task shell showed its bottom navigation; evidence text computed to 16px and Tab reached the enabled action. No physical mobile keyboard, Linux screenshot baseline, live deployment stop, real API admission or provider copyback acceptance is claimed. Other story states received source review, not a full browser execution pass.
Refreshed the two reviewed catalog entries only. Global catalog validation already reported unrelated source-fingerprint drift before this work; those entries require their own review.
## Automatic workspace recovery revision — 2026-10-13
Supersedes the task shutdown form reviewed above. The production task now renders
only a quiet restoration status or an honest operator-assistance fallback. It
contains no process metadata, evidence field, confirmation checkbox or resume
button. The task thread omits the duplicate technical recovery card. Obsolete
form stories and their simulated API fixture were removed; suite IDs remain
stable. The composed task journey includes a saved state without the notice.
Observed in the macOS in-app browser: automatic status in the production shell,
completed example without the status, retained approval-wait message, and mobile
bottom navigation with equal content and viewport widths (no horizontal overflow).
These are fixture UI observations, not live provider shutdown or task commitment.
The production component test verifies removal on a resolved action update. The
real reconciler's database tests separately preserve accepted-result identity,
monitor and approval waits, and create no new run or wake request.
Only the two reviewed suites have refreshed fingerprints. Global validation has
71 other source-fingerprint errors, including recovery-recipe dependents after
the operator instructions changed. It is not globally green. No unrelated review
fingerprints or visual baselines were accepted.
## Inherited activity, inbox and recovery recipes — 2026-10-12
@@ -534,3 +563,77 @@ capability IDs. These establish inventory, associations, references and review
fingerprints. No new Storybook build, rendered `play`, built-index comparison,
visual baseline, live authorization, cleanup, persistence or provider execution
is claimed by this reconciliation. Those checks remain with final qualification.
### Current-master reconciliation
Merged master `5e8e992df2`. The newer catalog supplies the baseline associations
and reviews. Compared every declared input against that master: only seven
inputs changed for this feature (recovery recipe, recovery card, task page, quiet
status, two stories and their fixture). Reviewed those changes and the unchanged
subjects/fixtures they affect; refreshed the 26 dependent fingerprints while
retaining their classifications and simulation boundaries. The task-page change
adds the quiet status and suppresses only the owner-unverified duplicate card;
other task variants retain their behavior. The recovery-card change blocks
unsafe generic dismissal. Existing card regressions cover that denial.
Five remaining errors have unchanged source bytes and metadata inherited from
master: shared setup, decision-card, decisions-desk, search and what-needs-me.
These require a separate review; no hashes or visual baselines were accepted for
them. The earlier 71-error count above describes the pre-merge baseline.
### Inherited index metadata follow-up
The current master's personal-toast recipe also left shared setup and four inbox
suites with stale metadata. Reviewed the recipe's added toast capability and
verification limits: decision-card, decisions-desk and search still demonstrate
their existing isolated subjects; what-needs-me's toast fixture does not prove
viewer eligibility. Their subject and fixture bytes are unchanged from master;
retained all unreviewed classifications and fixture boundaries. The generator
adds the missing toast capability and description to the existing inbox feature,
without changing feature order or IDs. Reviewed that generated diff and refreshed
these four entries, the shared setup and Overview fingerprints. No toast behavior,
new browser acceptance or visual baseline is claimed. Global catalog validation
now passes. This supersedes the five inherited-error count above.
### Confirmed recovery status
Reviewed the shipped notice, server evidence contract and both recovery fixtures.
Platform identity alone no longer means work is restoring: the server marks the
exact recovery receipt resuming only after verified stop proof and a successful
owner claim. Missing proof remains operator-required. The new unavailable-proof
story covers a platform-bound receipt without progress; confirmed and queued
stories retain the quiet status. Updated only the two affected fingerprints.
This is component/fixture evidence, not deployed harness acceptance.
### Latest master reconciliation
Retained master’s task-creation visibility scenarios, source-review notes and
unreviewed classifications. Seven overlapping entries differ from master only
in the already reviewed recovery recipe or task-detail recovery notice wiring.
Refreshed those combined-source fingerprints and regenerated the index. No new
creation-visibility acceptance or visual baseline is claimed.
## Codex and workspace-recovery catalog merge — 2026-10-13
Reconciled Codex head `7956056b2f793aa1043f4a4454630cdedd90185c` with current
master `e3029c632b26b68595adc1a704a387833b3cefcb`. Both dated review histories above
retain their original evidence scope. The current master catalog supplies the
new workspace-finalization recovery suites and their source classifications;
Codex creation, onboarding and Legacy/native editing states remain additive.
Existing association, sidebar, home and simulation-boundary metadata is preserved.
The authored source ID union is intact: 2,224 IDs from current master plus 2,233
from the Codex head produce 2,242 stories across 226 suites, with no missing or
unexpected IDs. All nine newly merged workspace-recovery states, the five focused
Codex preparation states, twenty existing settings states and 35 generated feature
Overview IDs remain. The recovery notice is a production component; the task
journey remains an explicitly mixed local composition, not a live task-page test.
Canonical regeneration and validation pass. The existing catalog/generator/map
tests pass 27/27; map validation passes for 35 areas and 175 capability IDs.
Inspected the combined Codex source dependencies and generated harness-description
change before refreshing 79 affected entry fingerprints and the shared setup.
The incoming recovery entry metadata is retained. No classifications or visual
baselines were promoted. These results prove source inventory, references and
structural contracts; no new rendering, `play`, provider, permission, cleanup or
production-continuity acceptance is claimed for the merged source.
+83 -36
View File
@@ -305,7 +305,7 @@
"ui/storybook/stories/agent-chat/agent-chat-sidebar.stories.tsx",
"ui/src/adapters/adapter-display-registry.ts"
],
"fingerprint": "d95a1e9a7a4cf997b6d11c72cd26dce0ffd1c5d976b189e572fb4d716d943a51",
"fingerprint": "2973cbd7b859db2b2535a0e3f80402c3755801627d4efa3ff44de380050a2a55",
"stableId": "design-explorations-agent-chat-sidebar-pages",
"sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat sidebar/Pages"
},
@@ -366,7 +366,7 @@
"ui/storybook/stories/agent-chat/agent-chat.stories.tsx",
"ui/src/adapters/adapter-display-registry.ts"
],
"fingerprint": "991e34e4a986a8d85f59ba28a71b875236c2bdd24a15872448963801d1aa674a",
"fingerprint": "052bff02117a14b972e48ee4b773f8258f05bbc1dd481d36b535d3240b8b992f",
"stableId": "design-explorations-agent-chat",
"sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Agent chat"
},
@@ -521,7 +521,7 @@
"ui/storybook/stories/agents/agent-persona-pages.stories.tsx",
"ui/src/adapters/adapter-display-registry.ts"
],
"fingerprint": "b8756ae1b4b6c713793d979e540167a641451261de455cceb4395ca4a06b4fc9",
"fingerprint": "11e416bdddb44ece283d934eaf87f30eb85b04e5270c06d7bbba815c89484a76",
"stableId": "agents-personas-full-pages",
"sidebarTitle": "Agents and reusable capabilities/Hiring, configuration, and organization/Agents/Personas/Full pages"
},
@@ -1207,7 +1207,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/tasks/assigned-backlog-safeguards.stories.tsx"
],
"fingerprint": "cdb63642517b31c68df9ca8d62e60039f9f4d69d95d1a96dc96235fb7db1487d",
"fingerprint": "d6e735a2c56092274f139709aba0c1c555f3f3cb0fbf9820d50ee02874907b79",
"stableId": "paperclip-assigned-backlog-safeguards",
"sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Assigned Backlog Safeguards"
},
@@ -1231,7 +1231,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/inbox-search/blocked-inbox.stories.tsx"
],
"fingerprint": "bcb6c0a3cb34fcd06e7b056521ad84fb968f73ba46a7ba00eda2a86bb060b11b",
"fingerprint": "70b54f5ce7c083b8e64c28501d66247d4753adabd732b377f7ad18309dc90ffe",
"stableId": "product-inbox-blocked-tab",
"sidebarTitle": "Tasks and conversations/Inbox, decisions, and search/Product/Inbox/Blocked tab"
},
@@ -1470,7 +1470,7 @@
"ui/storybook/stories/agent-chat/chat-entry.stories.tsx",
"ui/src/adapters/adapter-display-registry.ts"
],
"fingerprint": "4919a8431260abef52e018a03d607655484d728377bad92ae658aa4c0eff063e",
"fingerprint": "1063df44a6ff6e8972e15c4fa4718ea12d1edbe1ae5816055ab12a9964f007a8",
"replacement": "ui/storybook/stories/agents/primary-agent/05-chat.stories.tsx",
"stableId": "design-explorations-chat-entry",
"sidebarTitle": "Tasks and conversations/Agent conversations and project handoff/Design explorations/Chat entry"
@@ -1513,13 +1513,13 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/CloudAccessGate.tsx",
"ui/storybook/stories/recovery/cloud-access-recovery.stories.tsx"
],
"fingerprint": "bbec224b127ec6898cf95dfae922e6747483c0d0d4ec55ff7d1560c0c4a5c041",
"fingerprint": "64b2422bdabe2b07b563354481000171455841e49ccbf6a72270066d4e27e9e0",
"stableId": "app-connection-recovery",
"sidebarTitle": "Projects and execution/Recovery/App/Connection recovery"
},
@@ -1843,14 +1843,14 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"maps/feature-map/secrets.md",
"ui/src/components/task-chat/TaskChatMarker.tsx",
"ui/storybook/stories/secrets/credential-access/01-notice.stories.tsx"
],
"fingerprint": "2eba045db9127e5cd8d8bf7c9664e4598a5c84be117ae6159ce7171b24f2005d",
"fingerprint": "14757d026311d143e9301d6294d2f46f40dad25af8ea3f8d5cf1893b260f82db",
"stableId": "task-credential-access-01-notice",
"sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/01 Notice"
},
@@ -1877,7 +1877,7 @@
"ui/storybook/support/credential-access/CredentialTaskPage.tsx",
"ui/src/adapters/adapter-display-registry.ts"
],
"fingerprint": "2fe1a1618237eb46648432c017fd19a5798a68b1ba29b293fa025b3ed293bdc0",
"fingerprint": "0c7ad2d8456115e0907edeab6925234516918387b9a6a7cb809464612a6cb01c",
"stableId": "task-credential-access-02-task-page",
"sidebarTitle": "Apps, channels, and extensions/Secrets and proposals/Task credential access/02 Task page"
},
@@ -2227,13 +2227,13 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/ExecutionBlockerNotice.tsx",
"ui/storybook/stories/recovery/execution-blocker-notice.stories.tsx"
],
"fingerprint": "bee366ba1266e6692c8f0ee027c449fdda6b51762369ba431871a5af5ee018c6",
"fingerprint": "018d8d687921ea109cb98f587abd59b94722f1f148940852326417ce8a457b04",
"stableId": "task-chat-recovery-notice",
"sidebarTitle": "Projects and execution/Recovery/Task chat/Recovery notice"
},
@@ -2247,7 +2247,7 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/ActiveAgentsPanel.tsx",
@@ -2257,7 +2257,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/recovery/execution-recovery.stories.tsx"
],
"fingerprint": "6f8b245b169d95046bdb52513dfaf4937a96f807b72470c198863471f59c8149",
"fingerprint": "82ddff0cf6d98e558f9038d59960758347ad39f3c7a9aaf18f29214a8ad81756",
"stableId": "tasks-execution-recovery",
"sidebarTitle": "Projects and execution/Recovery/Tasks/Execution recovery"
},
@@ -2881,7 +2881,7 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"maps/feature-map/task-coordination.md",
@@ -2889,7 +2889,7 @@
"ui/src/components/task-chat/TaskChatBlockerLinks.tsx",
"ui/storybook/stories/recovery/issue-blocked-notice.stories.tsx"
],
"fingerprint": "6a84d7f60318d33a1835a668873095aaad1c215a83bbbb4735823fdcb3291140",
"fingerprint": "52f861389de245bda4d9c582581b49ce0b9442d60ee7eff950adc6801aaca8a4",
"stableId": "product-issue-blocked-notice",
"sidebarTitle": "Projects and execution/Recovery/Product/Issue/Blocked notice"
},
@@ -3113,13 +3113,13 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/IssueScheduledRetryCard.tsx",
"ui/storybook/stories/recovery/model-capacity-retry.stories.tsx"
],
"fingerprint": "5e08f681f3f96cb7be32e316d3f4066e5239dcf0f4393cb9ec09d599734f201e",
"fingerprint": "1db367a2b77b8c747ab5c81c1fe3317da02bf294308e9e09af827bcd12ee0b9f",
"stableId": "tasks-model-capacity-retry",
"sidebarTitle": "Projects and execution/Recovery/Tasks/Model capacity retry"
},
@@ -3383,7 +3383,7 @@
"ui/storybook/prototypes/originating-tasks/OriginatingTasks.tsx",
"ui/storybook/stories/task-coordination/originating-tasks.stories.tsx"
],
"fingerprint": "59123e73a0286fd11a48376ee152a974d730c59bd5421fd273cfca3a185f6e08",
"fingerprint": "32c9981ce024550b26636b97d7337f7b34a1207e8bcdd5759a11d9a7ddb204fe",
"stableId": "ux-labs-tasks-created-from-a-task",
"sidebarTitle": "Tasks and conversations/Delegation, dependencies, and signoff/UX Labs/Tasks Created From a Task"
},
@@ -3825,7 +3825,7 @@
"ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx",
"ui/storybook/support/private-tasks/PrivacyStory.tsx"
],
"fingerprint": "d4b5ba6bce33a9657d859d3278b1f84bbc9b77cffd24e370214a0fce24c0a02b",
"fingerprint": "ebd9132a1f0228f775b1f382fc30dcae68e65284c8118c7b8a2eee3219c741ed",
"stableId": "private-tasks-01-creation",
"sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/01 Creation"
},
@@ -3971,7 +3971,7 @@
"ui/storybook/stories/access/private-tasks/06-full-product-pages.stories.tsx",
"ui/storybook/support/private-tasks/PrivacyStory.tsx"
],
"fingerprint": "5fb218d88c3b6815839732e4226a27f4b40a2df46c8cbad6709cb665546b4248",
"fingerprint": "6069116fffc3a661df1fb10245f3f6d8045086efd71c886a8064a27eed09ac42",
"stableId": "private-tasks-06-full-product-pages",
"sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/06 Full product pages"
},
@@ -4018,7 +4018,7 @@
"ui/storybook/stories/access/private-tasks/07-user-stories.stories.tsx",
"ui/storybook/support/private-tasks/PrivacyStory.tsx"
],
"fingerprint": "5fbe5c61f8754bc0c1db1c6a990b9c44bc5d941b7b61de77fac76300e6c42a22",
"fingerprint": "483a8bfae265844bfa10ebd8529572f43c23c8474d414b45ba0b663efc280dbd",
"stableId": "private-tasks-07-user-stories",
"sidebarTitle": "Getting started and access/Login, invitations, and access/Private tasks/07 User stories"
},
@@ -4477,7 +4477,7 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/DispositionRecoveryNotice.tsx",
@@ -4486,7 +4486,7 @@
"ui/storybook/prototypes/recovery-notice/RecoveryNoticePreview.tsx",
"ui/storybook/stories/recovery/recovery-notice-preview.stories.tsx"
],
"fingerprint": "cfd7345545112ea9ecc13035ebd187451117de5ed010f439cfb8c6962461fb3f",
"fingerprint": "e83bcaf099227bfcd17d571f680121b5b51d5f605d76479165000a575229a884",
"stableId": "design-previews-recovery-notice",
"sidebarTitle": "Projects and execution/Recovery/Design previews/Recovery notice"
},
@@ -4737,7 +4737,7 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/IssueProperties.tsx",
@@ -4745,7 +4745,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/recovery/scheduled-retry.stories.tsx"
],
"fingerprint": "43ef1917079251d558b3f52e5af750dfb69a54d3326d31494dbf037baeefe193",
"fingerprint": "f360f4207ef2b1cef18eee2307c50a7bce3a5f5bbf8966b24cd86b3ec50d21b4",
"stableId": "product-issue-scheduled-retry-surfaces",
"sidebarTitle": "Projects and execution/Recovery/Product/Issue Scheduled retry surfaces"
},
@@ -5049,8 +5049,8 @@
"status": "unreviewed",
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed. Source review confirms ordinary recovery fixtures remain applicable after the native-owner resolution gate; native-owner stop verification has its own suite.",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/IssueBlockedNotice.tsx",
@@ -5059,7 +5059,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/recovery/source-issue-recovery.stories.tsx"
],
"fingerprint": "1b6e44d4fb4ab7e2e63fb8816aa31fbe3267e10647f238ed82e6591a82d2e918",
"fingerprint": "949814263788ae0e3501b67bbd26118f150db219fc04e5cf653f18dd345ec8d5",
"stableId": "paperclip-source-issue-recovery",
"sidebarTitle": "Projects and execution/Recovery/Paperclip/Source Issue Recovery"
},
@@ -5074,14 +5074,14 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/SourceResolvedFoldBadge.tsx",
"ui/src/components/SourceResolvedFoldCallout.tsx",
"ui/storybook/stories/recovery/source-resolved-fold.stories.tsx"
],
"fingerprint": "3af3daff618b8c278bcbf10ba3e5d081d87240c680758ccb3078eccdf9283122",
"fingerprint": "70cab38574df19267e35ab13c10c4a3eea444bb3b6e72c81f739f21cfc519df0",
"stableId": "paperclip-source-resolved-fold",
"sidebarTitle": "Projects and execution/Recovery/Paperclip/Source-resolved Fold"
},
@@ -5162,7 +5162,7 @@
"ui/storybook/fixtures/paperclipData.ts",
"ui/storybook/stories/tasks/successful-run-handoff.stories.tsx"
],
"fingerprint": "2a7db108d46f35f37dd9346d2fc039e61dff8c688e09d9ed7b0e777e7f79af0d",
"fingerprint": "27ad7c3c2b09acf5ac07163d44026511bd0c2b806387b591fef5fca343b0c63b",
"stableId": "paperclip-successful-run-handoff",
"sidebarTitle": "Tasks and conversations/Task creation and lifecycle/Paperclip/Successful Run Handoff"
},
@@ -5615,14 +5615,14 @@
"implementation": "production",
"boundary": "fixtures",
"notes": "Shipped subjects identified through imports and local wrappers; full state, fixture-contract and route parity remain unreviewed.",
"reviewedAt": "2026-10-12",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/WorkspaceBaseRefRecoveryNotice.tsx",
"ui/storybook/stories/recovery/workspace-branch-recovery/01-component.stories.tsx",
"ui/storybook/support/workspace-branch-recovery/fixtures.ts"
],
"fingerprint": "77f708cc660f6829cadaffa6139930a102109976c41f7e24e2f662ed687d2412",
"fingerprint": "eeaa7044a1d5a2b5355a8ac2375820e3611c71920e88c04d87e5d7ce64ed929a",
"stableId": "workspace-branch-recovery-01-repair-card",
"sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/01 Repair card"
},
@@ -5647,7 +5647,7 @@
"ui/storybook/stories/recovery/workspace-branch-recovery/02-task-journey.stories.tsx",
"ui/storybook/support/workspace-branch-recovery/fixtures.ts"
],
"fingerprint": "5eedfcb5738c125d3fa8ed7dc4b3d20a300b216de51f1ae5e58ede98d38fd75c",
"fingerprint": "f8a090cfdbadfe77d8ee064b99fc11b8f5e097d4e72173df2496280ef52152d1",
"stableId": "workspace-branch-recovery-02-task-journey",
"sidebarTitle": "Projects and execution/Recovery/Workspace branch recovery/02 Task journey"
},
@@ -5712,6 +5712,53 @@
"stableId": "workspaces-service-control-bar",
"sidebarTitle": "Projects and execution/Workspaces, services, and files/Workspaces/Service control bar"
},
{
"file": "ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx",
"capabilities": [
"recovery/workspace",
"recovery/source-action"
],
"purpose": "Workspace finalization recovery/01 Notice",
"status": "current",
"implementation": "production",
"boundary": "fixtures",
"notes": "Production quiet status; automatic restoration, legacy operator fallback, queued and resolved states. No task repair form, process details or API simulation. Provider proof is verified by separate server and harness tests.",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/WorkspaceExportRecovery.tsx",
"ui/storybook/stories/recovery/workspace-finalization-recovery/01-notice.stories.tsx",
"ui/storybook/support/workspace-finalization-recovery/fixtures.tsx"
],
"fingerprint": "45bf8c793c91362634b7ce32ed0ec2007861b7fefbfd0e2da692c3dc804a53c9",
"stableId": "workspace-finalization-recovery-01-notice",
"sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/01 Notice"
},
{
"file": "ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx",
"capabilities": [
"recovery/workspace",
"recovery/source-action"
],
"purpose": "Workspace finalization recovery/02 Task journey",
"status": "unreviewed",
"implementation": "mixed",
"boundary": "fixtures",
"notes": "Production shell, status, bubbles and composer in a local conversation composition; shows restoration and saved work waiting for confirmation. Fixture messages; not the shipped task page or live provider acceptance.",
"reviewedAt": "2026-10-13",
"sources": [
"maps/feature-map/recovery.md",
"ui/src/components/Layout.tsx",
"ui/src/components/WorkspaceExportRecovery.tsx",
"ui/src/components/task-chat/TaskChatBubble.tsx",
"ui/src/components/task-chat/TaskChatComposer.tsx",
"ui/storybook/stories/recovery/workspace-finalization-recovery/02-task-journey.stories.tsx",
"ui/storybook/support/workspace-finalization-recovery/fixtures.tsx"
],
"fingerprint": "aa76cd28dde8d50abd208b34283249170f3a85435ba82fff169a1c33433dfca1",
"stableId": "workspace-finalization-recovery-02-task-journey",
"sidebarTitle": "Projects and execution/Recovery/Workspace finalization recovery/02 Task journey"
},
{
"file": "ui/storybook/stories/agents/codex-runner/creation.stories.tsx",
"capabilities": [
+1
View File
@@ -1999,6 +1999,7 @@ export {
resolveIssueRecoveryActionSchema,
isValidExistingBranchName,
retryWorkspaceExportSchema,
resumeWorkspaceFinalizationSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
checkoutIssueSchema,
+1
View File
@@ -441,6 +441,7 @@ export {
resolveIssueRecoveryActionSchema,
isValidExistingBranchName,
retryWorkspaceExportSchema,
resumeWorkspaceFinalizationSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
checkoutIssueSchema,
+8
View File
@@ -557,6 +557,14 @@ const RESOLVE_ISSUE_RECOVERY_ACTION_OUTCOMES = [
"cancelled",
] as const;
export const resumeWorkspaceFinalizationSchema = z.object({
actionId: z.string().guid(),
runId: z.string().guid(),
ownerToken: z.string().guid(),
controllerAndCopybackStopped: z.literal(true),
stopEvidence: z.string().trim().min(20).max(12000),
}).strict();
export const retryWorkspaceExportSchema = z.object({
actionId: z.string().guid(),
runId: z.string().guid(),
+2
View File
@@ -125,6 +125,8 @@
"acpx"
],
"devDependencies": {
"@opentelemetry/context-async-hooks": "2.10.0",
"@opentelemetry/sdk-trace": "2.10.0",
"@paperclipai/paperclip-runner": "workspace:*",
"@types/compression": "^1.8.1",
"@types/express": "^5.0.0",
@@ -1810,6 +1810,41 @@ describeEmbeddedPostgres("authorization service", () => {
expect(await auth.decide({ actor, action: "agent:wake", resource: { ...resource, companyId: otherCompany.id } })).toMatchObject({ allowed: false });
});
it.each(["session", "cloud_tenant"] as const)("restricts %s workspace-owner stop confirmation to administrators", async source => {
const company = await createCompany(db, "OwnerStop");
const otherCompany = await createCompany(db, "OtherOwnerStop");
const userId = await createUser(db);
await db.insert(companyMemberships).values({ companyId: company.id,
principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
const actor = { type: "board" as const, source, userId, companyIds: [company.id] };
const auth = authorizationService(db);
const request = { actor, action: "runtime:confirm_owner_stop" as const,
resource: { type: "company" as const, companyId: company.id } };
for (const membershipRole of ["owner", "admin", "operator", "member", "viewer", null]) {
await db.update(companyMemberships).set({ membershipRole }).where(eq(companyMemberships.principalId, userId));
expect((await auth.decide(request)).allowed, `role ${membershipRole}`).toBe(membershipRole === "owner" || membershipRole === "admin");
}
await db.update(companyMemberships).set({ membershipRole: "admin", status: "suspended" }).where(eq(companyMemberships.principalId, userId));
expect((await auth.decide(request)).allowed).toBe(false);
await db.update(companyMemberships).set({ status: "active" }).where(eq(companyMemberships.principalId, userId));
expect((await auth.decide({ ...request, resource: { type: "company", companyId: otherCompany.id } })).allowed).toBe(false);
});
it("does not allow agent delegation or stale cloud admin rows to confirm an owner stop", async () => {
const company = await createCompany(db, "OwnerStopDelegation");
const userId = await createUser(db);
const agent = await createAgent(db, company.id, { role: "ceo" });
await db.insert(companyMemberships).values({ companyId: company.id,
principalType: "user", principalId: userId, status: "active", membershipRole: "operator" });
await db.insert(instanceUserRoles).values({ userId, role: "instance_admin" });
const auth = authorizationService(db);
const request = { action: "runtime:confirm_owner_stop" as const, resource: { type: "company" as const, companyId: company.id } };
expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: false } })).allowed).toBe(false);
expect((await auth.decide({ ...request, actor: { type: "agent", agentId: agent.id, companyId: company.id, onBehalfOfUserId: userId } })).allowed).toBe(false);
expect((await auth.decide({ ...request, actor: { type: "board", userId, source: "cloud_tenant", isInstanceAdmin: true } })).allowed).toBe(true);
expect((await auth.decide({ ...request, actor: { type: "board", source: "local_implicit" } })).allowed).toBe(true);
});
it("limits viewer members to read-only visibility actions", async () => {
const company = await createCompany(db, "BoardViewerVisibility");
const userId = `user-${randomUUID()}`;
@@ -162,8 +162,10 @@ const mockLogActivity = vi.hoisted(() => vi.fn(async () => undefined));
const mockObserveCrossIssueInfluence = vi.hoisted(() => vi.fn(async () => null));
const mockRetryWorkspaceExport = vi.hoisted(() => vi.fn());
const mockResumeWorkspaceFinalization = vi.hoisted(() => vi.fn());
function registerRouteMocks() {
vi.doMock("../services/native-runtime/native-workspace-owner-recovery.js", () => ({ resumeNativeWorkspaceAfterOwnerStop: mockResumeWorkspaceFinalization }));
vi.doMock("../services/native-runtime/native-workspace-export-retry.js", () => ({ retryNativeWorkspaceExport: mockRetryWorkspaceExport }));
vi.doMock("@paperclipai/shared/telemetry", () => ({
trackAgentTaskCompleted: vi.fn(),
@@ -2009,6 +2011,52 @@ describe("agent issue mutation checkout ownership", () => {
);
});
it("resumes a verified stopped workspace owner without waking the provider", async () => {
mockAccessService.decide.mockResolvedValue({ allowed: true });
mockResumeWorkspaceFinalization.mockResolvedValue({ runId: ownerRunId, status: "queued" });
const res = await request(await createApp(boardActor()))
.post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`)
.send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId,
controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." });
expect(res.status).toBe(202);
expect(mockAccessService.decide).toHaveBeenCalledWith(expect.objectContaining({ action: "runtime:confirm_owner_stop" }));
expect(mockResumeWorkspaceFinalization).toHaveBeenCalledWith(expect.objectContaining({ companyId, issueId, runId: ownerRunId, ownerToken: ownerRunId, actorId: "board-user" }));
expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled();
});
it.each([{ controllerAndCopybackStopped: false }, { stopEvidence: "" }])("requires explicit stop evidence: %j", async invalid => {
const res = await request(await createApp(boardActor()))
.post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`)
.send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId,
controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited.", ...invalid });
expect(res.status).toBe(400);
expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled();
});
it.each(["agent", "viewer"])("rejects %s workspace-owner release before admission", async kind => {
mockAccessService.decide.mockImplementation(async (input: { action: string }) => ({ allowed: input.action === "issue:read", explanation: "No runtime access" }));
const res = await request(await createApp(kind === "agent" ? ownerActor() : boardActor()))
.post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`)
.send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId,
controllerAndCopybackStopped: true, stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." });
expect(res.status).toBe(403);
expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled();
});
it("rejects an ordinary runtime manager's claimed foreign-owner stop", async () => {
mockAccessService.decide.mockImplementation(async ({ action }: { action: string }) => ({
allowed: action === "issue:read" || action === "runtime:manage",
explanation: "A deployment administrator must verify the controller stopped",
}));
const res = await request(await createApp(boardActor()))
.post(`/api/issues/${issueId}/recovery-actions/resume-workspace-finalization`)
.send({ actionId: recoveryActionId, runId: ownerRunId, ownerToken: ownerRunId,
controllerAndCopybackStopped: true, stopEvidence: "I claim that the foreign controller and all its processes have stopped." });
expect(res.status).toBe(403);
expect(mockResumeWorkspaceFinalization).not.toHaveBeenCalled();
expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled();
});
it("queues board export-only recovery without calling provider wake", async () => {
mockAccessService.decide.mockResolvedValue({ allowed: true });
mockRetryWorkspaceExport.mockResolvedValue({ runId: ownerRunId, status: "queued" });
@@ -2030,7 +2078,25 @@ describe("agent issue mutation checkout ownership", () => {
expect(res.status).toBe(403); expect(mockRetryWorkspaceExport).not.toHaveBeenCalled();
});
it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => {
it.each(["done", "in_review"])("does not bypass unverified copyback via false_positive to %s", async sourceIssueStatus => {
const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId });
mockIssueService.getById.mockResolvedValue(sourceIssue);
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({ ...sourceIssue, ...patch }));
mockIssueRecoveryActionService.getActiveForIssue.mockResolvedValue({
id: recoveryActionId, status: "active", kind: "active_run_watchdog", ownerType: "board",
ownerAgentId: null, returnOwnerAgentId: ownerAgentId,
cause: "native_workspace_finalization_owner_unverified", evidence: { runId: ownerRunId },
});
const res = await request(await createApp(boardActor()))
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "false_positive", sourceIssueStatus });
expect(res.status).toBe(409);
expect(res.body.details?.code).toBe("workspace_owner_stop_required");
expect(mockIssueService.update).not.toHaveBeenCalled();
expect(mockHeartbeatService.wakeup).not.toHaveBeenCalled();
});
it.each(["todo", "done", "in_review"].flatMap(sourceIssueStatus => ["native_workspace_sync_out_unsafe_archive", "native_workspace_sync_out_retry_exhausted", "native_workspace_finalization_owner_unverified"].map(cause => ({ sourceIssueStatus, cause }))))("does not resolve accepted export recovery through an ordinary $sourceIssueStatus transition: $cause", async ({ sourceIssueStatus, cause }) => {
const sourceIssue = makeIssue({ status: "blocked", assigneeAgentId: ownerAgentId });
mockIssueService.getById.mockResolvedValue(sourceIssue);
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({ ...sourceIssue, ...patch }));
@@ -2043,7 +2109,7 @@ describe("agent issue mutation checkout ownership", () => {
.post(`/api/issues/${issueId}/recovery-actions/resolve`)
.send({ actionId: recoveryActionId, outcome: "restored", sourceIssueStatus });
expect.soft(res.status).toBe(409);
expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : "workspace_export_retry_required");
expect.soft(res.body.details?.code).toBe(cause === "native_workspace_sync_out_unsafe_archive" ? "workspace_export_automatic_recovery" : cause === "native_workspace_finalization_owner_unverified" ? "workspace_owner_stop_required" : "workspace_export_retry_required");
expect.soft(mockIssueService.update).not.toHaveBeenCalled();
expect.soft(mockHeartbeatService.wakeup).not.toHaveBeenCalled();
});
+17
View File
@@ -17,6 +17,7 @@ import { setIssueTitleSchema } from "@paperclipai/shared";
import { resolveConfirmationFromComment } from "../services/confirmation-comment-resolution.js";
import { createIssueReadTiming } from "../services/issue-read-timing.js";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js";
import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js";
import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js";
import { deliverConversationComments, isConversation } from "../services/agent-conversations.js";
@@ -96,6 +97,7 @@ import {
resolveCreateIssueStatusDefault,
resolveIssueRecoveryActionSchema,
retryWorkspaceExportSchema,
resumeWorkspaceFinalizationSchema,
runnerGoalActionRequestSchema,
feedbackTargetTypeSchema,
feedbackTraceStatusSchema,
@@ -9511,6 +9513,17 @@ export function issueRoutes(
});
});
router.post("/issues/:id/recovery-actions/resume-workspace-finalization", validate(resumeWorkspaceFinalizationSchema), async (req, res) => {
assertBoard(req);
const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found");
if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return;
const decision = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", resource: { type: "company", companyId: issue.companyId } });
if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
const receipt = await resumeNativeWorkspaceAfterOwnerStop({ db, companyId: issue.companyId, issueId: issue.id,
...req.body, actorId: getActorInfo(req).actorId });
res.status(202).json(receipt);
});
router.post("/issues/:id/recovery-actions/retry-workspace-export", validate(retryWorkspaceExportSchema), async (req, res) => {
assertBoard(req);
const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found");
@@ -9757,6 +9770,10 @@ export function issueRoutes(
throw conflict("Unsafe workspace export recovers automatically without another provider turn.", { code: "workspace_export_automatic_recovery" });
}
if ((outcome === "restored" || outcome === "false_positive") && activeRecoveryAction.cause === "native_workspace_finalization_owner_unverified") {
throw conflict("Verify the previous controller and its copyback processes stopped, then use Resume saved result.", { code: "workspace_owner_stop_required" });
}
if (outcome === "restored" && isNativeWorkspaceExportRepairCause(activeRecoveryAction.cause)) {
throw conflict("Repair the retained sandbox, then use Retry workspace export to finish the accepted result without another provider turn.", { code: "workspace_export_retry_required" });
}
+10
View File
@@ -240,6 +240,7 @@ import {
createAcceptedPlanDecompositionSchema,
resolveIssueRecoveryActionSchema,
retryWorkspaceExportSchema,
resumeWorkspaceFinalizationSchema,
cancelIssueThreadInteractionSchema,
// Secret provider configs and remote import
createSecretProviderConfigSchema,
@@ -10635,6 +10636,15 @@ registerCurrentRoute({
summary: "List issue recovery actions",
});
registerCurrentRoute({
method: "post",
path: "/api/issues/{id}/recovery-actions/resume-workspace-finalization",
tags: ["issues"],
summary: "Resume a saved native result after operator verification of the prior workspace owner's stop",
body: resumeWorkspaceFinalizationSchema,
responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict },
});
registerCurrentRoute({
method: "post",
path: "/api/issues/{id}/recovery-actions/retry-workspace-export",
+16
View File
@@ -94,6 +94,7 @@ export type AuthorizationAction =
| "project:read"
| "project:write_workspace"
| "runtime:manage"
| "runtime:confirm_owner_stop"
| "secrets:read"
| "secrets:propose";
@@ -184,6 +185,7 @@ function permissionForAction(action: AuthorizationAction): PermissionKey | null
action === "issue:read" ||
action === "project:read" ||
action === "runtime:manage" ||
action === "runtime:confirm_owner_stop" ||
action === "secrets:read" ||
action === "secrets:propose"
) {
@@ -1858,6 +1860,11 @@ export function authorizationService(db: Db | DbTransaction) {
});
}
if (input.action === "runtime:confirm_owner_stop" && input.actor.type !== "board") {
return deny({ action: input.action, reason: "deny_unsupported_action",
explanation: "Only a deployment administrator can confirm a workspace controller has stopped." });
}
if (input.actor.type === "board") {
let taskAssignmentPolicyEffect: AssignmentPolicyEffect | null = null;
if (input.actor.source === "local_implicit") {
@@ -1886,6 +1893,15 @@ export function authorizationService(db: Db | DbTransaction) {
explanation: "Allowed because the actor is an instance admin.",
});
}
if (input.action === "runtime:confirm_owner_stop") {
const membership = await getActiveMembership(companyId, "user", input.actor.userId ?? "");
if (membership?.membershipRole === "owner" || membership?.membershipRole === "admin") {
return allow({ action: input.action, reason: "allow_role_default",
explanation: "Allowed by the active company owner or administrator role." });
}
return deny({ action: input.action, reason: "deny_missing_grant",
explanation: "Only a company owner, company administrator, or instance administrator can confirm a workspace controller has stopped." });
}
// What instance-admin elevation used to give cloud tenant users is
// replaced by company-scoped visibility: an active membership in the
// resource company grants the same read surface a same-company agent
@@ -231,7 +231,7 @@ function decodeJsonPart(part: string, label: string): Record<string, unknown> {
return parsed as Record<string, unknown>;
}
function publicKeyForKid(env: NodeJS.ProcessEnv, kid: string) {
export function cloudRuntimePublicKeyForKid(env: NodeJS.ProcessEnv, kid: string) {
const raw = nonEmpty(env.PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS);
if (!raw) throw new Error("PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS is not configured");
let parsed: unknown;
@@ -276,7 +276,7 @@ function verifyClaims(input: {
) {
throw new Error("Cloud runtime identity protected header is invalid");
}
const key = publicKeyForKid(input.env, header.kid);
const key = cloudRuntimePublicKeyForKid(input.env, header.kid);
const signature = Buffer.from(encodedSignature, "base64url");
const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii");
if (!verify(null, signingInput, key, signature)) {
@@ -523,7 +523,7 @@ export function verifyCloudControlAssertion(input: {
) {
throw new Error("Cloud control protected header is invalid");
}
const key = publicKeyForKid(env, header.kid);
const key = cloudRuntimePublicKeyForKid(env, header.kid);
const signature = Buffer.from(encodedSignature, "base64url");
const signingInput = Buffer.from(`${encodedHeader}.${encodedPayload}`, "ascii");
if (!verify(null, signingInput, key, signature)) {
+22 -20
View File
@@ -1,3 +1,4 @@
import { createHeartbeatPreparationTiming } from "./heartbeat/preparation-timing.js";
import { createHeartbeatInstructionCopyRelease, createHeartbeatInstructionWorkingCopy } from "./heartbeat/instruction-working-copy.js";
import { prepareHeartbeatRuntimeServiceIntents, startHeartbeatRuntimeServices, recordHeartbeatAdapterRuntimeServices } from "./heartbeat/runtime-services.js";
import { prepareHeartbeatExecutionConfiguration } from "./heartbeat/execution-configuration.js";
@@ -1918,6 +1919,7 @@ export function heartbeatService(
if (run.runtimeMode === "legacy" && run.controllerBootId &&
run.controllerBootId !== legacyControllerBootId) return;
activeRunExecutions.add(run.id);
const timePreparation = createHeartbeatPreparationTiming(run.id);
const executionControl = createAdapterExecutionControl();
// This coarse scope also covers host finalization after the adapter returns.
// Nested scopes refine the label without making any termination claim.
@@ -2613,7 +2615,7 @@ export function heartbeatService(
materializeNativeChatWorktree,
explicitTaskDirectory,
useIsolatedTaskDirectory,
} = await selectHeartbeatEnvironment(db, {
} = await timePreparation("environment_selection", () => selectHeartbeatEnvironment(db, {
run, agent,
task: { issueRef, issueContext, issueId, context, executionProjectId, workspaceProjectContext, boundSourceWorkspace,
hasSelectedWorkspaceSource: Boolean(selectedWorkspaceSource), taskWorkspaceIntent: taskWorkspaceIntentRow?.intent ?? null, responsibleUserId },
@@ -2623,9 +2625,9 @@ export function heartbeatService(
},
config: { config, sourceWorkspaceConfig, issueAssigneeOverrides },
services: { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder },
});
}));
let nativeChatExpectedCwd = selectedNativeChatExpectedCwd;
const executionConfiguration = await prepareHeartbeatExecutionConfiguration(db, {
const executionConfiguration = await timePreparation("execution_configuration", () => prepareHeartbeatExecutionConfiguration(db, {
run, agent,
task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext },
config: { sourceWorkspaceConfig, aiBinding },
@@ -2635,7 +2637,7 @@ export function heartbeatService(
onFailureSecretsResolved: (readSecrets) => { readFailureReportSecrets = readSecrets; },
onManagedAiRuntimeAcquired: (runtime) => { managedAiRuntime = runtime; },
onIdentityRedactorPrepared: (redactor) => { identityRedactor = redactor; },
});
}));
if (!executionConfiguration.prepared) return;
const {
mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub,
@@ -2649,7 +2651,7 @@ export function heartbeatService(
...identityRedactor.values,
...collectRunFailureSecretValues(runtimeConfig.env, secretKeys),
];
const preparedSession = await prepareHeartbeatSession(db, {
const preparedSession = await timePreparation("session", () => prepareHeartbeatSession(db, {
run, agent,
task: { context, issueRef, issueContext, taskProjectContext, workspaceProjectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision },
session: { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec },
@@ -2663,7 +2665,7 @@ export function heartbeatService(
},
environment: { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig },
getLatestAgentConfigRevision,
});
}));
const { sessionConfigMetadata, configuredModel, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams } = preparedSession;
const {
resolvedWorkspace,
@@ -2685,7 +2687,7 @@ export function heartbeatService(
workspaceRealization,
executionTarget,
workspaceRestoreSource,
} = await prepareHeartbeatWorkspace(db, {
} = await timePreparation("workspace", () => prepareHeartbeatWorkspace(db, {
run, agent, controllerLease, nativeRunnerPreparationSpans,
task: {
issueRef,
@@ -2737,7 +2739,7 @@ export function heartbeatService(
resolveReusedGitWorkspaceAnchor,
appendRunEvent,
},
});
}));
const preparedInstructions = instructionWorkingCopy = createHeartbeatInstructionWorkingCopy({
db, instructionCopies, readRun: () => run!, agent, context,
session: { taskSession, taskSessionForRun, sessionConfigMetadata },
@@ -2901,7 +2903,7 @@ export function heartbeatService(
} else {
delete context.paperclipScratch;
}
const gitExecutionEnv = await prepareGitHubExecutionEnvironment({
const gitExecutionEnv = await timePreparation("github_environment", () => prepareGitHubExecutionEnvironment({
target: executionTarget,
cwd: executionWorkspace.cwd,
env: Object.fromEntries(
@@ -2915,12 +2917,12 @@ export function heartbeatService(
networkAccess:
trustPreset.kind === "standard" &&
process.env.PAPERCLIP_RUNNER_NETWORK_ACCESS !== "disabled",
});
}));
runtimeConfig = { ...runtimeConfig, env: gitExecutionEnv };
for (const key of MANAGED_GITHUB_TOKEN_KEYS) secretKeys.add(key);
context.githubAuthenticationMode = useHostGitHub ? "host" : "managed";
if (!useHostGitHub) {
const githubLaunchers = await prepareHeartbeatGitHubLaunchers({
const githubLaunchers = await timePreparation("github_launchers", () => prepareHeartbeatGitHubLaunchers({
native: agent.adapterType === "paperclip_runner",
githubConfigured: githubSelection.configured,
agentId: agent.id,
@@ -2936,7 +2938,7 @@ export function heartbeatService(
responsibleUserId: responsibleUserId ?? "",
scope: "github_credentials",
})?.token ?? "",
});
}));
githubLauncherLocation = githubLaunchers.cleanupLocation;
runtimeConfig = { ...runtimeConfig, env: githubLaunchers.env };
secretKeys.add("PAPERCLIP_GITHUB_BROKER_TOKEN");
@@ -3167,7 +3169,7 @@ export function heartbeatService(
});
const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent } =
await runOutput.begin(currentRun);
await timePreparation("run_output", () => runOutput.begin(currentRun));
if (runScopedMentionedSkillKeys.length > 0) {
await onLog(
"stdout",
@@ -3178,7 +3180,7 @@ export function heartbeatService(
const logEntry = formatRuntimeWorkspaceWarningLog(warning);
await onLog(logEntry.stream, logEntry.chunk);
}
await assertGitSensitiveAdapterWorkspaceValid({
await timePreparation("workspace_validation", () => assertGitSensitiveAdapterWorkspaceValid({
adapterType: agent.adapterType,
agentId: agent.id,
issue: issueRef
@@ -3195,17 +3197,17 @@ export function heartbeatService(
executionTarget,
environmentDriver: selectedEnvironment.driver,
leaseMetadata: activeEnvironmentLease.lease.metadata,
});
}));
const adapterEnv = Object.fromEntries(
Object.entries(parseObject(runtimeConfig.env)).filter(
(entry): entry is [string, string] =>
typeof entry[0] === "string" && typeof entry[1] === "string",
),
);
const runtimeServices = await startHeartbeatRuntimeServices(db, {
const runtimeServices = await timePreparation("runtime_services", () => startHeartbeatRuntimeServices(db, {
run, agent, issueId, issueRef, executionWorkspace, context, issuesSvc, onLog,
persistedExecutionWorkspace, hostExecutionWorkspaceConfig, skipHostProvision, adapterEnv, workspaceOperationRecorder,
});
}));
const adapter = getServerAdapter(agent.adapterType);
const durableGoalControlRun =
readNonEmptyString(context.goalControlRequestId) !== null ||
@@ -3251,8 +3253,8 @@ export function heartbeatService(
const hasInstructionFilesystem = nativeRuntimeResolution.kind !== "native"
? adapter.supportsInstructionsBundle === true
: !["claude_managed_agents_api", "aws_agentcore_harness_api", "openai_dot_mcp"].includes(nativeRuntimeResolution.profile.backend);
runtimeConfig = await preparedInstructions.prepare({ runtimeConfig, nativeRuntimeResolution, hasInstructionFilesystem });
const runtimeSelection = await selectHeartbeatRuntime(db, {
runtimeConfig = await timePreparation("instructions", () => preparedInstructions.prepare({ runtimeConfig, nativeRuntimeResolution, hasInstructionFilesystem }));
const runtimeSelection = await timePreparation("runtime_selection", () => selectHeartbeatRuntime(db, {
run,
agent,
nativeRuntimeResolution,
@@ -3319,7 +3321,7 @@ export function heartbeatService(
nativeOwnershipHeld = true;
},
stopControllerLease: () => controllerLease.stop(),
});
}));
if (!runtimeSelection.selected) return;
const { nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff } = runtimeSelection;
nativeWorkspaceSync = runtimeSelection.nativeWorkspaceSync;
@@ -0,0 +1,53 @@
import { createHash } from "node:crypto";
import { setImmediate } from "node:timers/promises";
import { context, trace } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { TracerProvider, InMemorySpanExporter, SimpleSpanProcessor } from "@opentelemetry/sdk-trace";
import { expect, it } from "vitest";
import { createHeartbeatPreparationTiming } from "./preparation-timing.js";
it("parents child spans to their own preparation phase across overlapping awaited runs", async () => {
const manager = new AsyncLocalStorageContextManager().enable();
const exporter = new InMemorySpanExporter();
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
expect(context.setGlobalContextManager(manager)).toBe(true);
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
try {
const gates = [0, 1].map(() => {
let resolve!: () => void;
const promise = new Promise<void>(release => { resolve = release; });
return { promise, resolve };
});
const runs = ["first-run", "second-run"];
const pending = runs.map((run, index) => createHeartbeatPreparationTiming(run)("workspace", async () => {
await gates[index].promise;
await setImmediate();
const child = trace.getTracer("preparation-context-test").startSpan(`child.${run}`);
await setImmediate();
child.end();
}));
expect(trace.getActiveSpan()).toBeUndefined();
gates[1].resolve();
gates[0].resolve();
await Promise.all(pending);
await provider.forceFlush();
const spans = exporter.getFinishedSpans();
expect(spans).toHaveLength(4);
const phases = runs.map(run => spans.find(span => span.name === "heartbeat.prepare.workspace"
&& span.attributes["paperclip.task.run.run_id"] === createHash("sha256").update(run).digest("hex").slice(0, 12))!);
expect(phases.every(Boolean)).toBe(true);
expect(phases[0].spanContext().traceId).not.toBe(phases[1].spanContext().traceId);
for (const [index, run] of runs.entries()) {
const child = spans.find(span => span.name === `child.${run}`)!;
expect(child.parentSpanContext?.spanId).toBe(phases[index].spanContext().spanId);
expect(child.spanContext().traceId).toBe(phases[index].spanContext().traceId);
expect(phases[index].parentSpanContext).toBeUndefined();
}
expect(trace.getActiveSpan()).toBeUndefined();
} finally {
trace.disable();
context.disable();
manager.disable();
await provider.shutdown();
}
});
@@ -0,0 +1,49 @@
import { createHash } from "node:crypto";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { createHeartbeatPreparationTiming } from "./preparation-timing.js";
const handles = vi.hoisted(() => ({ spans: [] as Array<{ setStatus: ReturnType<typeof vi.fn>; end: ReturnType<typeof vi.fn> }> }));
const startActiveSpan = vi.hoisted(() => vi.fn((_name, _options, callback) => {
const span = { setStatus: vi.fn(), end: vi.fn() };
handles.spans.push(span);
return callback(span);
}));
vi.mock("@opentelemetry/api", () => ({
SpanStatusCode: { ERROR: 2 },
trace: { getTracer: () => ({ startActiveSpan }) },
}));
describe("heartbeat preparation timing", () => {
beforeEach(() => { vi.clearAllMocks(); handles.spans.length = 0; });
it("correlates preparation with the existing run hash without recording content", async () => {
const runId = "private-run-id";
const time = createHeartbeatPreparationTiming(runId);
const result = { secret: "private result" };
await expect(time("instructions", async () => result)).resolves.toBe(result);
expect(startActiveSpan).toHaveBeenCalledExactlyOnceWith("heartbeat.prepare.instructions", {
attributes: { "paperclip.task.run.run_id": createHash("sha256").update(runId).digest("hex").slice(0, 12) },
}, expect.any(Function));
expect(handles.spans[0].end).toHaveBeenCalledOnce();
expect(handles.spans[0].setStatus).not.toHaveBeenCalled();
});
it("ends a failed setup phase and preserves its exact error without recording its text", async () => {
const error = new Error("private filesystem or credential detail");
await expect(createHeartbeatPreparationTiming("run")("workspace", async () => { throw error; })).rejects.toBe(error);
expect(handles.spans[0].setStatus).toHaveBeenCalledExactlyOnceWith({ code: 2 });
expect(handles.spans[0].end).toHaveBeenCalledOnce();
});
it("keeps overlapping operations open until their own work settles", async () => {
const time = createHeartbeatPreparationTiming("run");
let release!: () => void;
const waiting = time("workspace", () => new Promise<void>(resolve => { release = resolve; }));
await time("runtime_services", async () => undefined);
expect(handles.spans[0].end).not.toHaveBeenCalled();
expect(handles.spans[1].end).toHaveBeenCalledOnce();
release();
await waiting;
expect(handles.spans[0].end).toHaveBeenCalledOnce();
});
});
@@ -0,0 +1,25 @@
import { createHash } from "node:crypto";
import { SpanStatusCode, trace } from "@opentelemetry/api";
type PreparationPhase = "environment_selection" | "execution_configuration" | "session"
| "workspace" | "github_environment" | "github_launchers" | "run_output"
| "workspace_validation" | "runtime_services" | "instructions" | "runtime_selection";
export function createHeartbeatPreparationTiming(runId: string) {
const runIdHash = createHash("sha256").update(runId).digest("hex").slice(0, 12);
return <T>(phase: PreparationPhase, work: () => Promise<T>): Promise<T> =>
trace.getTracer("paperclip.heartbeat-preparation").startActiveSpan(
`heartbeat.prepare.${phase}`,
{ attributes: { "paperclip.task.run.run_id": runIdHash } },
async (span) => {
try {
return await work();
} catch (error) {
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
},
);
}
+15 -2
View File
@@ -375,6 +375,18 @@ export function issueRecoveryActionService(db: Db) {
// beyond the advertised cap. A distinct identity can still supersede the
// exhausted action through the branch above.
if (isRecoveryBudgetExhausted(existing.evidence ?? {})) {
// Older sweeps replaced this operator-only hold with instructions to
// start another run. Restore the exact copyback recovery instructions
// without reopening its budget or erasing a recorded stop confirmation.
if (existing.cause === "native_workspace_finalization_owner_unverified"
&& input.cause === existing.cause && input.fingerprint === existing.fingerprint
&& !existing.evidence?.workspaceOwnerStop && existing.nextAction !== input.nextAction) {
const [updated] = await db.update(issueRecoveryActions).set({ nextAction: input.nextAction, updatedAt: now })
.where(and(eq(issueRecoveryActions.id, existing.id),
inArray(issueRecoveryActions.status, [...ACTIVE_RECOVERY_ACTION_STATUSES]),
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`)).returning();
if (updated) return toReadModel(updated);
}
return existing;
}
const nextAttemptCount =
@@ -417,8 +429,9 @@ export function issueRecoveryActionService(db: Db) {
fingerprint: existing.fingerprint,
},
},
nextAction:
`Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` +
nextAction: existing.cause === "native_workspace_finalization_owner_unverified"
? input.nextAction
: `Automatic recovery exhausted after ${attemptsUsed}/${effectiveMaxAttempts} attempts. ` +
"Review the infrastructure failure and explicitly choose a replacement run or provider configuration.",
wakePolicy: null,
monitorPolicy: null,
@@ -1,15 +1,19 @@
import { logActivity } from "../activity-log.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { nativeSha256 } from "./canonical.js";
import { NATIVE_WORKSPACE_OWNER_UNVERIFIED } from "./native-workspace-finalization-recovery-state.js";
import { randomUUID } from "node:crypto";
import os from "node:os";
import { and, eq, sql } from "drizzle-orm";
import { heartbeatRuns, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { and, eq, inArray, sql } from "drizzle-orm";
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, workspaceOperations, withDedicatedDbConnection, type Db } from "@paperclipai/db";
import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js";
import {
currentNativeControllerIdentity,
evaluateNativeControllerTakeover,
} from "./native-restart-recovery.js";
import { currentWorkspaceWorker, requestWorkspaceWorkerStopProof, type WorkspaceWorker, type WorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js";
const OWNER_KEY = "nativeWorkspaceFinalizationOwner";
const activeOwnerTokens = new Set<string>();
// Positive same-boot evidence, retained only when durable cleanup could not be
@@ -21,6 +25,7 @@ type Owner = {
hostname: string;
pid: number;
processStartedAt: string;
worker?: WorkspaceWorker;
};
export type NativeWorkspaceFinalizationOwnership = {
token: string;
@@ -58,7 +63,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
const identity = await currentNativeControllerIdentity();
const owner: Owner = {
token: randomUUID(), controllerBootId: identity.bootId, hostname: os.hostname(), pid: identity.pid,
processStartedAt: identity.processStartedAt.toISOString(),
processStartedAt: identity.processStartedAt.toISOString(), worker: currentWorkspaceWorker(),
};
const scope = and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.runtimeMode, "native"));
const owned = and(scope, sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${owner.token}`);
@@ -73,13 +78,42 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
if (!run) throw new Error("native_workspace_finalization_binding_missing");
const rawPrior = run.profile?.[OWNER_KEY];
const prior = readOwner(rawPrior);
let automaticResultId: string | undefined;
const currentRecoveryResult = async (db: Db) => {
if (!run.issueId) return null;
// Use the arbiter's lock order and release row locks before copyback I/O.
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId),
eq(nativeRunFinalizations.issueId, run.issueId),
)).for("update");
const [issue] = await db.select().from(issues).where(and(
eq(issues.companyId, input.companyId), eq(issues.id, run.issueId),
)).for("update");
const [currentRun] = await db.select().from(heartbeatRuns).where(scope).for("update");
return coordinator && issue && currentRun ? currentNativeWorkspaceRecoveryResult(db, {
companyId: input.companyId, coordinator, issue, run: currentRun,
}) : null;
};
let platformProof: WorkspaceWorkerStopProof | null = null;
if (rawPrior != null) {
const requireStopVerification = async () => {
const fingerprint = nativeSha256({ runId: input.runId, owner: rawPrior });
const nextAction = "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn.";
const recovery = issueRecoveryActionService(input.db);
const existing = run.issueId ? await recovery.getActiveForIssue(input.companyId, run.issueId) : null;
// Rechecking the same physical owner is not a new recovery attempt.
if (existing?.cause === NATIVE_WORKSPACE_OWNER_UNVERIFIED && existing.fingerprint === fingerprint) {
if (!existing.evidence?.workspaceOwnerStop) {
await input.db.update(issueRecoveryActions).set({ nextAction, updatedAt: new Date(),
evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"operator_required"}'::jsonb)`,
}).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, existing.id),
eq(issueRecoveryActions.fingerprint, fingerprint), eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`${issueRecoveryActions.evidence}->'workspaceOwnerStop' is null`,
sql`(${issueRecoveryActions.nextAction} is distinct from ${nextAction} or ${issueRecoveryActions.evidence}#>>'{workspaceOwnerRecovery,state}' is distinct from 'operator_required')`,
));
}
return { acquired: false } as const;
}
if (run.issueId) await recovery.upsertSourceScoped({
@@ -87,18 +121,21 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: run.agentId,
cause: NATIVE_WORKSPACE_OWNER_UNVERIFIED,
fingerprint,
evidence: { runId: input.runId, owner: rawPrior },
nextAction: "Verify the previous controller and its workspace-copyback processes have stopped, then release only this exact workspace owner receipt using doc/native-workspace-finalization-recovery.md. Resume workspace finalization without another provider turn.",
evidence: { runId: input.runId, owner: rawPrior, workspaceOwnerRecovery: { state: "operator_required" } },
nextAction,
wakePolicy: null, maxAttempts: 1, supersedeOnIdentityChange: true,
});
return { acquired: false } as const;
};
// A foreign host or malformed receipt cannot prove the old physical owner stopped.
if (!prior || prior.hostname !== owner.hostname) return requireStopVerification();
// The harness checks the exact owned deployment, including orphaned children.
// A live deployment, unsigned answer or older receipt stays fenced.
if (prior?.worker) platformProof = await requestWorkspaceWorkerStopProof({ runId: input.runId, ownerToken: prior.token, worker: prior.worker });
if (!prior) return requireStopVerification();
if (!platformProof && prior.hostname !== owner.hostname) return requireStopVerification();
const knownJoinedHere = prior.controllerBootId === identity.bootId
&& prior.pid === identity.pid && joinedOwnerTokens.has(prior.token)
&& !activeOwnerTokens.has(prior.token);
if (!knownJoinedHere) {
if (!platformProof && !knownJoinedHere) {
const takeover = await evaluateNativeControllerTakeover({
owner: { leaseOwner: prior.token, leaseExpiresAt: new Date(0), controllerPid: prior.pid,
controllerProcessStartedAt: new Date(prior.processStartedAt) }, now: new Date(),
@@ -119,15 +156,32 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
}
// Claim before the potentially ambiguous write so finally can remove
// only our token even if the server committed but its response was lost.
activeOwnerTokens.add(owner.token);
claimed = true;
const [claimedRun] = await input.db.update(heartbeatRuns).set({
runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb)`,
}).where(and(scope, prior
? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${prior.token}`
: sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null`
)).returning({ id: heartbeatRuns.id });
const claimOwner = async (db: Db) => {
activeOwnerTokens.add(owner.token);
claimed = true;
const [claimedRun] = await db.update(heartbeatRuns).set({
runnerProfileJson: sql`jsonb_set(coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb), array[${OWNER_KEY}], ${JSON.stringify(owner)}::jsonb) || ${JSON.stringify(platformProof ? { nativeWorkspaceOwnerStopProof: platformProof } : {})}::jsonb`,
}).where(and(scope, prior
? sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY} = ${JSON.stringify(rawPrior)}::jsonb`
: sql`${heartbeatRuns.runnerProfileJson}->>${OWNER_KEY} is null`
)).returning({ id: heartbeatRuns.id });
return claimedRun;
};
// A stop receipt proves the old worker exited, not that its task is still
// current. Revalidate after the provider read and atomically with the CAS.
const claimedRun = platformProof ? await input.db.transaction(async tx => {
const result = await currentRecoveryResult(tx as unknown as Db);
if (!result) return null;
const claimedRun = await claimOwner(tx as unknown as Db);
if (claimedRun) automaticResultId = result.id;
return claimedRun;
}) : await claimOwner(input.db);
if (!claimedRun) return { acquired: false } as const;
if (platformProof && run.issueId) await logActivity(input.db, {
companyId: input.companyId, actorType: "system", actorId: "workspace-recovery",
action: "issue.workspace_owner_stop_verified", entityType: "issue", entityId: run.issueId,
runId: input.runId, details: { proof: platformProof },
});
if (prior) joinedOwnerTokens.delete(prior.token);
const ownership: NativeWorkspaceFinalizationOwnership = {
token: owner.token,
@@ -137,12 +191,25 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
await lock.execute(sql`select 1`);
const [current] = await input.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(owned).limit(1);
if (!current) throw new Error("owner_replaced");
if (automaticResultId) {
const result = await input.db.transaction(tx => currentRecoveryResult(tx as unknown as Db));
if (result?.id !== automaticResultId) throw new Error("task_replaced");
}
} catch (cause) {
throw new NativeWorkspaceFinalizationOwnershipLostError({ cause });
}
},
};
await ownership.assertHeld();
if (platformProof && run.issueId) await input.db.update(issueRecoveryActions).set({
evidence: sql`jsonb_set(coalesce(${issueRecoveryActions.evidence}, '{}'::jsonb), '{workspaceOwnerRecovery}', '{"state":"resuming"}'::jsonb)`,
updatedAt: new Date(),
}).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, run.issueId),
eq(issueRecoveryActions.cause, NATIVE_WORKSPACE_OWNER_UNVERIFIED),
eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: input.runId, owner: rawPrior })),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
));
actionPromise = action(ownership);
const value = await actionPromise;
await ownership.assertHeld();
@@ -1,3 +1,8 @@
import * as platformProof from "./native-workspace-platform-proof.js";
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
import { nativeSha256 } from "./canonical.js";
import { resumeNativeWorkspaceAfterOwnerStop } from "./native-workspace-owner-recovery.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { spawn } from "node:child_process";
import { once } from "node:events";
import { readProcessStartedAt } from "../hot-restart.js";
@@ -9,6 +14,8 @@ import path from "node:path";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { and, desc, eq, sql } from "drizzle-orm";
import {
activityLog,
agentWakeupRequests,
agents,
companies,
completionContracts,
@@ -19,11 +26,15 @@ import {
nativeRunFinalizations,
nativeRunResults,
issueRecoveryActions,
issueThreadInteractions,
projects,
workspaceOperations,
workAssessments,
statusDecisions,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js";
import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js";
describe("native workspace finalization recovery", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -159,6 +170,225 @@ describe("native workspace finalization recovery", () => {
}
});
async function seedAbandonedCopyback() {
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Accepted yield, abandoned copyback" });
const owner = { token: randomUUID(), hostname: "replaced-controller", pid: 2,
processStartedAt: "2026-10-10T00:02:55.560Z", controllerBootId: randomUUID() };
await db.update(heartbeatRuns).set({ status: "running", nativePhase: "workspace_finalizing",
runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" },
}).where(eq(heartbeatRuns.id, seeded.runId));
await db.update(issues).set({ executionRunId: seeded.runId }).where(eq(issues.id, seeded.issueId));
await db.update(nativeRunFinalizations).set({ phase: "workspace_finalizing" }).where(eq(nativeRunFinalizations.runId, seeded.runId));
const recovery = issueRecoveryActionService(db);
const hold = { companyId, sourceIssueId: seeded.issueId, kind: "active_run_watchdog" as const,
ownerType: "board" as const, cause: "native_workspace_finalization_owner_unverified",
fingerprint: nativeSha256({ runId: seeded.runId, owner }), evidence: { runId: seeded.runId, owner }, maxAttempts: 1,
nextAction: "Verify the previous controller and its workspace-copyback processes have stopped. Resume workspace finalization without another provider turn." };
const action = await recovery.upsertSourceScoped(hold);
await recovery.upsertSourceScoped(hold);
return { ...seeded, owner, actionId: action.id, companyId, db, ownerToken: owner.token,
actorId: "operator", controllerAndCopybackStopped: true as const,
stopEvidence: "Deployment platform confirms the exact old container and all its processes exited." };
}
it("retains copyback-specific instructions after repeated recovery sweeps", async () => {
const seed = await seedAbandonedCopyback();
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId));
expect(action.status).toBe("escalated");
expect(action.nextAction).toContain("workspace-copyback processes have stopped");
expect(action.nextAction).not.toContain("replacement run");
await db.update(issueRecoveryActions).set({ nextAction: "Start a replacement run" }).where(eq(issueRecoveryActions.id, seed.actionId));
await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected");
const [repaired] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId));
expect(repaired.nextAction).toContain("workspace-copyback processes have stopped");
expect(repaired.attemptCount).toBe(action.attemptCount);
expect(repaired.evidence?.recoveryBudget).toEqual(action.evidence?.recoveryBudget);
});
it("releases only the verified owner and resumes the saved result without changing its disposition", async () => {
const seed = await seedAbandonedCopyback();
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false });
const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed);
expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" });
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
expect(after).toEqual(before);
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson).toEqual({ preserved: "workspace descriptor" });
expect(run.status).toBe("running");
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId));
expect(action.evidence?.workspaceOwnerStop).toMatchObject({ owner: seed.owner, actorId: "operator", resultId: before.resultId, stopEvidence: seed.stopEvidence });
expect(action.status).toBe("escalated"); // Only commitment resolves the hold.
expect(action.wakePolicy).toBeNull();
expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1);
const file = path.join(workspaceRoot, seed.runId);
expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => {
await fs.writeFile(file, "accepted work");
return "resumed";
})).toEqual({ acquired: true, value: "resumed" });
expect(await fs.readFile(file, "utf8")).toBe("accepted work");
// A duplicate confirmation neither clears a newer owner nor writes another audit.
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toEqual(receipt);
expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(1);
});
it.each(["expired", "live", "unknown"] as const)("checks the coordinator lease before verified recovery: %s", async state => {
const seed = await seedAbandonedCopyback();
const leaseExpiresAt = state === "unknown" ? null : new Date(Date.now() + (state === "expired" ? -60_000 : 60_000));
await db.update(nativeRunFinalizations).set({ leaseOwner: "previous-reconciler", leaseExpiresAt }).where(eq(nativeRunFinalizations.runId, seed.runId));
if (state === "expired") {
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].leaseOwner).toBe("previous-reconciler");
} else {
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0].runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(seed.owner);
}
});
it.each(["pending", "resolved", "superseded", "unassigned"] as const)("requires authoritative reviewer recovery: %s", async state => {
const seed = await seedAbandonedCopyback();
const reviewerId = randomUUID(), sourceRunId = randomUUID(), sourceResultId = randomUUID(), assessmentId = randomUUID();
const decisionId = randomUUID(), interactionId = randomUUID();
await db.insert(agents).values({ id: reviewerId, companyId, name: "Review agent", adapterType: "paperclip_runner", status: "running" });
const [actingRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, nativeIssueId: seed.issueId, runtimeMode: "native", status: "succeeded", completionContractId: actingRun.completionContractId, createdAt: new Date(actingRun.createdAt.getTime() - 60_000) });
await db.insert(nativeRunResults).values({ id: sourceResultId, companyId, issueId: seed.issueId, runId: sourceRunId, completionContractId: actingRun.completionContractId!, serverFingerprint: sourceResultId, schemaStatus: "accepted", resultJson: {}, canonicalSha256: sourceResultId });
await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId: seed.issueId, runId: sourceRunId, contractId: actingRun.completionContractId!, resultId: sourceResultId, triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, policyVersion: "fixture", assessmentJson: {}, inputDigest: assessmentId });
const [issue] = await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, seed.issueId)).returning();
await db.insert(statusDecisions).values({ id: decisionId, companyId, issueId: seed.issueId, runId: sourceRunId, assessmentId, decisionVersion: 1, policyVersion: "fixture", fromStatus: "in_progress", toStatus: "in_review", reasonCode: "explicit_review", decisionJson: { projectedStatusVersion: issue.statusVersion }, decisionDigest: decisionId, applicationState: "applied" });
await db.update(issues).set({ lastStatusDecisionId: state === "superseded" ? null : decisionId }).where(eq(issues.id, seed.issueId));
await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seed.issueId, kind: "request_confirmation", sourceRunId, addresseeAgentId: state === "unassigned" ? agentId : reviewerId, createdByAgentId: agentId, continuationPolicy: "wake_assignee", status: state === "resolved" ? "accepted" : "pending", ...(state === "resolved" ? { resolvedByRunId: seed.runId, resolvedByAgentId: reviewerId } : {}), payload: { version: 1, prompt: "Review the completed work.", target: { type: "custom", key: "native_completion_review", revisionId: decisionId } } });
await db.update(heartbeatRuns).set({ agentId: reviewerId, contextSnapshot: { issueId: seed.issueId, nativeReviewInteractionId: interactionId, nativeReviewDecisionId: decisionId } }).where(eq(heartbeatRuns.id, seed.runId));
if (state === "pending" || state === "resolved") {
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).resolves.toMatchObject({ status: "queued" });
expect((await db.select().from(issues).where(eq(issues.id, seed.issueId)))[0].assigneeAgentId).toBe(agentId);
} else {
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toMatchObject({ status: 409 });
}
});
it.each(([
["workspace_finalizing", "monitor"], ["workspace_finalizing", "approval"],
["retryable_failure", "monitor"], ["retryable_failure", "approval"],
["result_accepted", "monitor"], ["result_accepted", "approval"],
] as const).flatMap(([phase, wait]) => (["operator", "automatic"] as const).map(mode => [phase, wait, mode] as const)))("commits the saved yield from %s preserving its %s wait via %s recovery", async (phase, wait, mode) => {
const seed = await seedAbandonedCopyback();
await db.update(nativeRunFinalizations).set({ phase,
...(phase === "retryable_failure" ? { attempt: 1, nextAttemptAt: new Date(Date.now() - 60_000),
failureCode: "native_workspace_sync_out_failed" } : {}),
}).where(eq(nativeRunFinalizations.runId, seed.runId));
await db.update(heartbeatRuns).set({ nativePhase: phase }).where(eq(heartbeatRuns.id, seed.runId));
const workspaceId = randomUUID();
const cwd = path.join(workspaceRoot, seed.runId);
await fs.mkdir(cwd);
await fs.writeFile(path.join(cwd, "saved.txt"), "accepted work");
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId,
mode: "local", strategyType: "local_directory", name: "Retained workspace", cwd });
const nextCheckAt = new Date(Date.now() - 60_000); // Already due, like the staging runs.
const policy = { monitor: { nextCheckAt: nextCheckAt.toISOString(), notes: "Check the existing PR." } };
await db.update(issues).set({ executionWorkspaceId: workspaceId,
...(wait === "monitor" ? { executionPolicy: policy, monitorNextCheckAt: nextCheckAt } : {}),
}).where(eq(issues.id, seed.issueId));
const interactionId = randomUUID();
if (wait === "approval") await db.insert(issueThreadInteractions).values({
id: interactionId, companyId, issueId: seed.issueId, sourceRunId: seed.runId,
createdByAgentId: agentId, kind: "request_confirmation", status: "pending",
payload: { version: 1, prompt: "Approve the plan?", target: { type: "issue_document",
issueId: seed.issueId, key: "plan", revisionId: randomUUID() } },
});
const resultJson = {
result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded",
summary: "Wait for the existing continuation.",
completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [],
remainingWork: [{ description: "Wait for the check or plan confirmation.", blocksCompletion: true }] },
continuation: { kind: wait === "monitor" ? "monitor" : "response_wake",
summary: "Wait for the existing continuation.", idempotencyKey: "saved-wait" },
evidence: [], verification: [], attentionRequests: [], artifacts: [] },
terminal: { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed",
runTerminalState: "succeeded", reportedWorkDisposition: "yielded" },
};
await db.update(nativeRunResults).set({ resultJson }).where(eq(nativeRunResults.runId, seed.runId));
const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId));
// The real reconciler must remain fenced before operator confirmation.
await reconcileNativeFinalizations(db, [seed.runId]);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe(phase);
const [beforeConfirmation] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
if (mode === "operator") {
await resumeNativeWorkspaceAfterOwnerStop(seed);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toEqual(beforeConfirmation);
await reconcileNativeFinalizations(db, [seed.runId]);
} else {
const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" };
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceFinalizationOwner: { ...seed.owner, worker } } }).where(eq(heartbeatRuns.id, seed.runId));
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: seed.owner.token, worker, stopped: true, iat: 1, exp: 61 });
try { await reconcileNativeFinalizations(db, [seed.runId]); } finally { probe.mockRestore(); }
}
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0]).toMatchObject({ phase: "committed", resultId: accepted.id });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId)))[0]).toMatchObject({ status: "succeeded", nativePhase: "committed" });
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, seed.runId))).toEqual([accepted]);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId)))[0].status).toBe("resolved");
const [issue] = await db.select().from(issues).where(eq(issues.id, seed.issueId));
expect(issue.status).toBe(wait === "monitor" ? "in_progress" : "in_review");
if (wait === "monitor") {
expect(issue.monitorNextCheckAt).toEqual(nextCheckAt);
expect(issue.executionPolicy).toEqual(policy);
} else {
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0].status).toBe("pending");
}
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, seed.issueId))).toHaveLength(1);
expect(await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.payload}->>'issueId' = ${seed.issueId}`)).toHaveLength(0);
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, seed.runId),
eq(workspaceOperations.phase, "workspace_finalize"), eq(workspaceOperations.status, "succeeded")))).toHaveLength(1);
expect(await fs.readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("accepted work");
});
it.each(["token", "company", "result", "newer owner", "missing confirmation", "malformed owner"])("rejects stale or unverified copyback recovery: %s", async changed => {
const seed = await seedAbandonedCopyback();
if (changed === "malformed owner") {
const owner = { token: seed.ownerToken };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId));
}
if (changed === "token") seed.ownerToken = randomUUID();
if (changed === "company") seed.companyId = foreignCompanyId;
if (changed === "result") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId));
if (changed === "newer owner") await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: { ...seed.owner, token: randomUUID() } } }).where(eq(heartbeatRuns.id, seed.runId));
if (changed === "missing confirmation") Object.assign(seed, { controllerAndCopybackStopped: false });
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeDefined();
expect(await db.select().from(activityLog).where(eq(activityLog.runId, seed.runId))).toHaveLength(0);
});
it("rejects a locally live owner even after its advisory connection is gone", async () => {
const seed = await seedAbandonedCopyback();
const identity = await currentNativeControllerIdentity();
const owner = { ...seed.owner, hostname: os.hostname(), pid: identity.pid,
processStartedAt: identity.processStartedAt.toISOString(), controllerBootId: identity.bootId };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
await db.update(issueRecoveryActions).set({ evidence: { runId: seed.runId, owner } }).where(eq(issueRecoveryActions.id, seed.actionId));
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active");
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner);
});
it("does not reopen an old result after a newer task execution exists", async () => {
const seed = await seedAbandonedCopyback();
await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", nativeIssueId: seed.issueId });
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("no longer current");
});
it("rejects stop confirmation while another controller holds the physical lock", async () => {
const seed = await seedAbandonedCopyback();
await db.transaction(async tx => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${companyId}:${seed.runId}`}, 0))`);
await expect(resumeNativeWorkspaceAfterOwnerStop(seed)).rejects.toThrow("still active");
});
});
it("reuses a successful export even if an old controller wrote a later failed barrier", async () => {
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Late failed barrier" });
const [success] = await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: seeded.runId,
@@ -311,6 +541,137 @@ describe("native workspace finalization recovery", () => {
.toEqual({ acquired: true, value: "recovered" });
});
it("automatically adopts a proven stopped deployment without another agent turn", async () => {
const seed = await seedAbandonedCopyback();
const worker = { provider: "railway" as const, projectId: "fixture-project", environmentId: "fixture-env", serviceId: "fixture-service", deploymentId: "fixture-old" };
const owner = { ...seed.owner, worker };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner, preserved: "workspace descriptor" } }).where(eq(heartbeatRuns.id, seed.runId));
await db.update(issueRecoveryActions).set({ fingerprint: nativeSha256({ runId: seed.runId, owner }), evidence: { runId: seed.runId, owner, preserved: "recovery evidence" } }).where(eq(issueRecoveryActions.id, seed.actionId));
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
const proof = { v: 1 as const, iss: "paperclip-cloud" as const, aud: "paperclip-workspace-worker-stop/v1" as const, sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true as const, iat: 1, exp: 61 };
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(proof);
try {
const work = vi.fn(async () => {
const [recovery] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, seed.actionId));
expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "resuming" });
expect(recovery.evidence?.preserved).toBe("recovery evidence");
return "saved workspace copied";
});
expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: true, value: "saved workspace copied" });
expect(probe).toHaveBeenCalledWith({ runId: seed.runId, ownerToken: owner.token, worker });
expect(work).toHaveBeenCalledOnce();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined();
expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toEqual(proof);
expect(run.runnerProfileJson?.preserved).toBe("workspace descriptor");
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
expect(after).toEqual(before);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toHaveLength(0);
} finally { probe.mockRestore(); }
});
it("retains the fence for unknown platform evidence and owner changes during verification", async () => {
const seed = await seedAbandonedCopyback();
const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" };
const owner = { ...seed.owner, worker };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue(null);
const work = vi.fn();
try {
expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false });
const [recovery] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, seed.issueId), eq(issueRecoveryActions.fingerprint, nativeSha256({ runId: seed.runId, owner }))));
expect(recovery.evidence?.workspaceOwnerRecovery).toEqual({ state: "operator_required" });
const replacement = { ...owner, token: randomUUID() };
probe.mockImplementation(async () => {
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: replacement } }).where(eq(heartbeatRuns.id, seed.runId));
return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 };
});
expect(await withNativeWorkspaceFinalizationOwnership(seed, work)).toEqual({ acquired: false });
expect(work).not.toHaveBeenCalled();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(replacement);
} finally { probe.mockRestore(); }
});
it.each(["done", "cancelled", "execution replaced", "checkout replaced", "newer run", "queued legacy run", "reassigned", "result unbound", "result rejected", "run stopped"])("does not copy saved files after the task moves on during platform verification: %s", async changed => {
const seed = await seedAbandonedCopyback();
const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" };
const owner = { ...seed.owner, worker };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
const file = path.join(workspaceRoot, `${seed.runId}.txt`);
await fs.writeFile(file, "new task work");
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockImplementation(async () => {
if (changed === "done" || changed === "cancelled") await db.update(issues).set({ status: changed }).where(eq(issues.id, seed.issueId));
if (changed === "execution replaced") await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, seed.issueId));
if (changed === "checkout replaced") {
const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded" }).returning();
await db.update(issues).set({ checkoutRunId: other.id }).where(eq(issues.id, seed.issueId));
}
if (changed === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, seed.issueId));
if (changed === "result unbound") await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, seed.runId));
if (changed === "result rejected") await db.update(nativeRunResults).set({ schemaStatus: "rejected" }).where(eq(nativeRunResults.runId, seed.runId));
if (changed === "run stopped") await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, seed.runId));
if (changed === "newer run" || changed === "queued legacy run") {
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
await db.insert(heartbeatRuns).values({ companyId, agentId,
runtimeMode: changed === "newer run" ? "native" : "legacy",
nativeIssueId: changed === "newer run" ? seed.issueId : null,
contextSnapshot: { issueId: seed.issueId }, status: changed === "newer run" ? "succeeded" : "queued",
createdAt: new Date(run.createdAt.getTime() + (changed === "newer run" ? 60_000 : -60_000)),
});
}
return { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 };
});
const copyback = vi.fn(async () => fs.writeFile(file, "old sandbox work"));
try {
expect(await withNativeWorkspaceFinalizationOwnership(seed, copyback)).toEqual({ acquired: false });
expect(copyback).not.toHaveBeenCalled();
expect(await fs.readFile(file, "utf8")).toBe("new task work");
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.runId));
expect(run.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner);
expect(run.runnerProfileJson?.nativeWorkspaceOwnerStopProof).toBeUndefined();
} finally { probe.mockRestore(); }
});
it("rechecks automatic task ownership at the copyback publication boundary", async () => {
const seed = await seedAbandonedCopyback();
const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" };
const owner = { ...seed.owner, worker };
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 });
const write = vi.fn();
try {
await expect(withNativeWorkspaceFinalizationOwnership(seed, async ownership => {
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, seed.issueId));
await ownership.assertHeld();
write();
})).rejects.toThrow("native_workspace_finalization_ownership_lost");
expect(write).not.toHaveBeenCalled();
} finally { probe.mockRestore(); }
});
it.each(["closed", "superseded"])("does not start a workspace operation when the real reconciler selects a %s task", async changed => {
const seed = await seedAbandonedCopyback();
const worker = { provider: "railway" as const, projectId: "p", environmentId: "e", serviceId: "s", deploymentId: "d" };
const owner = { ...seed.owner, worker };
const workspaceId = randomUUID(), cwd = path.join(workspaceRoot, seed.runId);
await fs.mkdir(cwd);
await fs.writeFile(path.join(cwd, "current.txt"), "new task work");
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, projectId, mode: "local", strategyType: "local_directory", name: "Current workspace", cwd });
await db.update(issues).set({ executionWorkspaceId: workspaceId,
...(changed === "closed" ? { status: "done" } : { executionRunId: null }),
}).where(eq(issues.id, seed.issueId));
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }).where(eq(heartbeatRuns.id, seed.runId));
const probe = vi.spyOn(platformProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", runId: seed.runId, ownerToken: owner.token, worker, stopped: true, iat: 1, exp: 61 });
try {
await reconcileNativeFinalizations(db, [seed.runId]);
expect(probe).toHaveBeenCalledOnce();
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seed.runId))).toHaveLength(0);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)))[0].phase).toBe("workspace_finalizing");
expect(await fs.readFile(path.join(cwd, "current.txt"), "utf8")).toBe("new task work");
} finally { probe.mockRestore(); }
});
it("surfaces an unverifiable foreign controller as board recovery without physical writes", async () => {
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Foreign copyback controller" });
const owner = { token: randomUUID(), hostname: "other-controller.invalid", pid: 42, processStartedAt: new Date().toISOString() };
@@ -0,0 +1,87 @@
import os from "node:os";
import { and, eq, sql } from "drizzle-orm";
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { resumeWorkspaceFinalizationSchema } from "@paperclipai/shared";
import { conflict } from "../../errors.js";
import { persistActivity, publishActivity } from "../activity-log.js";
import { currentNativeWorkspaceRecoveryResult } from "./native-workspace-recovery-binding.js";
import { evaluateNativeControllerTakeover } from "./native-restart-recovery.js";
const OWNER_KEY = "nativeWorkspaceFinalizationOwner";
const record = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
const changed = () => conflict("The recorded workspace owner is no longer current. Refresh the task and inspect its run.", { code: "workspace_owner_recovery_stale" });
/** Board-only admission after deployment-platform verification. A missing DB
* lock or an expired lease alone never proves physical copyback has stopped. */
export async function resumeNativeWorkspaceAfterOwnerStop(input: {
db: Db; companyId: string; issueId: string; actionId: string; runId: string;
ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string; actorId: string;
}) {
const confirmation = resumeWorkspaceFinalizationSchema.parse({
actionId: input.actionId, runId: input.runId, ownerToken: input.ownerToken,
controllerAndCopybackStopped: input.controllerAndCopybackStopped, stopEvidence: input.stopEvidence,
});
const admitted = await input.db.transaction(async tx => {
// Share the physical copyback lock. Do not wait behind an active exporter
// and then apply a confirmation that described an earlier owner.
const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`);
if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery.");
// Match the status arbiter's coordinator -> issue -> run lock order.
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.id, input.runId))).for("update");
const [action] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update");
if (!issue || !run || !coordinator || !action || action.evidence?.runId !== run.id
|| action.cause !== "native_workspace_finalization_owner_unverified" || action.ownerType !== "board"
|| action.kind !== "active_run_watchdog" || !["active", "escalated"].includes(action.status)
|| (coordinator.leaseOwner && (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt.getTime() > Date.now()))) throw changed();
const result = await currentNativeWorkspaceRecoveryResult(tx as unknown as Db, {
companyId: input.companyId, issue, run, coordinator,
});
if (!result) throw changed();
const prior = record(run.runnerProfileJson?.[OWNER_KEY]);
const recordedOwner = record(action.evidence?.owner);
const stop = record(action.evidence?.workspaceOwnerStop);
const receipt = { runId: run.id, resultId: result.id, status: "queued" as const };
if (run.runnerProfileJson?.[OWNER_KEY] == null && record(stop.owner).token === input.ownerToken
&& stop.resultId === result.id) return { receipt, publication: null };
if (typeof prior.hostname !== "string" || !prior.hostname
|| !Number.isInteger(prior.pid) || Number(prior.pid) <= 0
|| typeof prior.processStartedAt !== "string" || !Number.isFinite(Date.parse(prior.processStartedAt))
|| prior.token !== input.ownerToken || recordedOwner.token !== input.ownerToken
|| prior.hostname !== recordedOwner.hostname || prior.pid !== recordedOwner.pid
|| prior.processStartedAt !== recordedOwner.processStartedAt
|| prior.controllerBootId !== recordedOwner.controllerBootId) throw changed();
// Reject a confirmation contradicted by a locally observable live owner,
// even if that owner's advisory connection has already disconnected.
if (prior.hostname === os.hostname()) {
const takeover = await evaluateNativeControllerTakeover({ owner: {
leaseOwner: input.ownerToken, leaseExpiresAt: new Date(0),
controllerPid: typeof prior.pid === "number" ? prior.pid : null,
controllerProcessStartedAt: typeof prior.processStartedAt === "string" ? new Date(prior.processStartedAt) : null,
}, now: new Date() });
if (!takeover.allowed) throw conflict("The previous workspace controller is still active or its stop cannot be verified.");
}
const now = new Date();
const proof = { owner: prior, resultId: result.id, actorId: input.actorId,
stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() };
await tx.update(heartbeatRuns).set({
runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now,
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id),
sql`${heartbeatRuns.runnerProfileJson}->${OWNER_KEY}->>'token' = ${input.ownerToken}`));
await tx.update(issueRecoveryActions).set({ evidence: { ...action.evidence, workspaceOwnerStop: proof },
nextAction: "Workspace finalization is queued for the saved result. The agent will not repeat its work.",
wakePolicy: null, monitorPolicy: null, updatedAt: now,
}).where(eq(issueRecoveryActions.id, action.id));
const publication = await persistActivity(tx as unknown as Db, { companyId: input.companyId,
actorType: "user", actorId: input.actorId, action: "issue.workspace_owner_stop_confirmed",
entityType: "issue", entityId: input.issueId, runId: input.runId,
details: { recoveryActionId: input.actionId, ...proof },
});
// Leave the result, coordinator, source lease, task status and wait paths
// untouched. The normal reconciler resumes copyback and arbitrates it.
return { receipt, publication };
});
if (admitted.publication) publishActivity(admitted.publication.publication);
return admitted.receipt;
}
@@ -0,0 +1,34 @@
import { generateKeyPairSync, sign } from "node:crypto";
import { describe, expect, it, vi } from "vitest";
import { verifyWorkspaceWorkerStopReceipt, requestWorkspaceWorkerStopProof, currentWorkspaceWorker } from "./native-workspace-platform-proof.js";
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
const env = { PAPERCLIP_CLOUD_STACK_ID: "stack", PAPERCLIP_CLOUD_API_ORIGIN: "https://cloud.example", PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "fixture-token", PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [{ ...publicKey.export({ format: "jwk" }), kid: "test", alg: "EdDSA", use: "sig" }] }) };
const worker = { provider: "railway" as const, projectId: "project", environmentId: "environment", serviceId: "service", deploymentId: "old-deployment" };
const binding = { runId: "run", ownerToken: "owner", worker };
const now = new Date("2099-01-01T00:00:00Z");
function receipt(overrides = {}) {
const header = Buffer.from(JSON.stringify({ alg: "EdDSA", kid: "test", typ: "paperclip-workspace-worker-stop+jwt" })).toString("base64url");
const payload = Buffer.from(JSON.stringify({ v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1", sub: "stack", ...binding, stopped: true, iat: now.getTime()/1000, exp: now.getTime()/1000+60, ...overrides })).toString("base64url");
return `${header}.${payload}.${sign(null, Buffer.from(`${header}.${payload}`), privateKey).toString("base64url")}`;
}
describe("workspace platform stop proof", () => {
it("accepts only signed proof for the exact stack, owner, run and full stopped deployment", () => {
expect(verifyWorkspaceWorkerStopReceipt(receipt(), binding, env, now)).toMatchObject({ stopped: true, worker });
for (const invalid of [{ sub: "other-stack" }, { runId: "other-run" }, { ownerToken: "other-owner" }, { stopped: false }, { exp: now.getTime()/1000-1 }, { worker: { ...worker, deploymentId: "other" } }, { aud: "paperclip-cloud-control/v1" }]) {
expect(verifyWorkspaceWorkerStopReceipt(receipt(invalid), binding, env, now)).toBeNull();
}
expect(verifyWorkspaceWorkerStopReceipt(receipt().slice(0,-5)+"wrong", binding, env, now)).toBeNull();
});
it("fails closed on live, unknown, unreachable and unsigned platform responses", async () => {
for (const response of [{ stopped: false }, { stopped: true }, { receipt: "forged" }]) {
const fetchImpl = vi.fn().mockResolvedValue(Response.json(response));
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl })).toBeNull();
}
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockRejectedValue(new Error("offline")) })).toBeNull();
expect(await requestWorkspaceWorkerStopProof(binding, { env, now, fetchImpl: vi.fn().mockResolvedValue(Response.json({ receipt: receipt() })) })).toMatchObject({ stopped: true });
});
it("records a complete platform identity, and does not guess legacy ownership", () => {
expect(currentWorkspaceWorker({ RAILWAY_PROJECT_ID: "project", RAILWAY_ENVIRONMENT_ID: "environment", RAILWAY_SERVICE_ID: "service", RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toEqual(worker);
expect(currentWorkspaceWorker({ RAILWAY_DEPLOYMENT_ID: "old-deployment" })).toBeUndefined();
});
});
@@ -0,0 +1,65 @@
import { verify } from "node:crypto";
import { cloudRuntimePublicKeyForKid } from "../cloud-runtime-identity.js";
import { getCloudStackContext } from "../cloud-instance.js";
/** Deployment identity covers the controller and every orphaned tar/Git child. */
export type WorkspaceWorker = {
provider: "railway"; projectId: string; environmentId: string; serviceId: string; deploymentId: string;
};
export type WorkspaceStopBinding = { runId: string; ownerToken: string; worker: WorkspaceWorker };
export type WorkspaceWorkerStopProof = WorkspaceStopBinding & {
v: 1; iss: "paperclip-cloud"; aud: "paperclip-workspace-worker-stop/v1"; sub: string;
stopped: true; iat: number; exp: number;
};
function validWorker(worker: unknown): worker is WorkspaceWorker {
if (!worker || typeof worker !== "object" || Array.isArray(worker)) return false;
const value = worker as Record<string, unknown>;
return value.provider === "railway" && ["projectId", "environmentId", "serviceId", "deploymentId"].every(key =>
typeof value[key] === "string" && value[key].length > 0 && value[key].length <= 200);
}
export function currentWorkspaceWorker(env: NodeJS.ProcessEnv = process.env): WorkspaceWorker | undefined {
const ids = [env.RAILWAY_PROJECT_ID, env.RAILWAY_ENVIRONMENT_ID, env.RAILWAY_SERVICE_ID, env.RAILWAY_DEPLOYMENT_ID];
if (ids.some(id => !id?.trim())) return undefined;
return { provider: "railway", projectId: ids[0]!, environmentId: ids[1]!, serviceId: ids[2]!, deploymentId: ids[3]! };
}
export function verifyWorkspaceWorkerStopReceipt(
receipt: string, binding: WorkspaceStopBinding, env: NodeJS.ProcessEnv = process.env, now = new Date(),
): WorkspaceWorkerStopProof | null {
try {
if (receipt.length > 16_000 || !validWorker(binding.worker)) return null;
const parts = receipt.split(".");
if (parts.length !== 3 || parts.some(part => !/^[A-Za-z0-9_-]+$/.test(part))) return null;
const header = JSON.parse(Buffer.from(parts[0], "base64url").toString("utf8"));
if (header.alg !== "EdDSA" || header.typ !== "paperclip-workspace-worker-stop+jwt" || typeof header.kid !== "string") return null;
if (!verify(null, Buffer.from(`${parts[0]}.${parts[1]}`, "ascii"), cloudRuntimePublicKeyForKid(env, header.kid), Buffer.from(parts[2], "base64url"))) return null;
const proof = JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")) as WorkspaceWorkerStopProof;
const seconds = Math.floor(now.getTime()/1000);
if (proof.v !== 1 || proof.iss !== "paperclip-cloud" || proof.aud !== "paperclip-workspace-worker-stop/v1"
|| !env.PAPERCLIP_CLOUD_STACK_ID || proof.sub !== env.PAPERCLIP_CLOUD_STACK_ID || proof.stopped !== true
|| proof.runId !== binding.runId || proof.ownerToken !== binding.ownerToken
|| !Number.isInteger(proof.iat) || !Number.isInteger(proof.exp) || proof.iat > seconds+30
|| proof.exp <= seconds || proof.exp <= proof.iat || proof.exp-proof.iat > 60
|| !validWorker(proof.worker) || Object.keys(binding.worker).some(key => proof.worker[key as keyof WorkspaceWorker] !== binding.worker[key as keyof WorkspaceWorker])) return null;
return proof;
} catch { return null; }
}
/** Unsupported platforms, legacy receipts and unavailable evidence never authorize takeover. */
export async function requestWorkspaceWorkerStopProof(binding: WorkspaceStopBinding, options: {
env?: NodeJS.ProcessEnv; fetchImpl?: typeof fetch; now?: Date;
} = {}): Promise<WorkspaceWorkerStopProof | null> {
const env = options.env ?? process.env;
const context = getCloudStackContext(env);
const token = env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
if (!context?.stackId || !context.cloudOrigin || !token || !validWorker(binding.worker)) return null;
try {
const origin = new URL(context.cloudOrigin);
if (origin.protocol !== "https:" || origin.username || origin.password) return null;
const response = await (options.fetchImpl ?? fetch)(`${origin.origin}/v1/tenant/workspace-worker-stop`, {
method: "POST", redirect: "error", headers: { authorization: `Bearer ${token}`, "x-paperclip-cloud-stack-id": context.stackId, "content-type": "application/json" },
body: JSON.stringify(binding), signal: AbortSignal.timeout(10_000),
});
if (!response.ok) return null;
const payload = await response.json() as { receipt?: unknown };
return typeof payload.receipt === "string" ? verifyWorkspaceWorkerStopReceipt(payload.receipt, binding, env, options.now) : null;
} catch { return null; }
}
@@ -0,0 +1,36 @@
import { and, eq, gt, inArray, ne, or, sql } from "drizzle-orm";
import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { getNativeReviewAssignment } from "./native-review-participant.js";
/** Both operator and platform recovery must still own the task's accepted work.
* Call with coordinator -> issue -> run rows locked in a short transaction. */
export async function currentNativeWorkspaceRecoveryResult(db: Db, input: {
companyId: string;
issue: typeof issues.$inferSelect;
run: typeof heartbeatRuns.$inferSelect;
coordinator: typeof nativeRunFinalizations.$inferSelect;
}) {
const { companyId, issue, run, coordinator } = input;
if (issue.companyId !== companyId || run.companyId !== companyId || coordinator.companyId !== companyId
|| run.nativeIssueId !== issue.id || coordinator.issueId !== issue.id || coordinator.runId !== run.id
|| run.runtimeMode !== "native" || run.status !== "running"
|| !["result_accepted", "workspace_finalizing", "retryable_failure"].includes(coordinator.phase)
|| !coordinator.resultId || ["done", "cancelled"].includes(issue.status)
|| issue.executionRunId !== run.id || (issue.checkoutRunId && issue.checkoutRunId !== run.id)) return null;
if (issue.assigneeAgentId !== run.agentId && !await getNativeReviewAssignment(db, {
companyId, issueId: issue.id, agentId: run.agentId,
contextSnapshot: run.contextSnapshot, actingRunId: run.id, issueExecutionRunId: issue.executionRunId,
allowResolvedByRunId: run.id,
})) return null;
const [result] = await db.select().from(nativeRunResults).where(and(
eq(nativeRunResults.id, coordinator.resultId), eq(nativeRunResults.companyId, companyId),
eq(nativeRunResults.issueId, issue.id), eq(nativeRunResults.runId, run.id), eq(nativeRunResults.schemaStatus, "accepted"),
));
if (!result || result.completionContractId !== run.completionContractId) return null;
const [newer] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), ne(heartbeatRuns.id, run.id),
or(eq(heartbeatRuns.nativeIssueId, issue.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`),
or(gt(heartbeatRuns.createdAt, run.createdAt), inArray(heartbeatRuns.status, ["queued", "running"])),
)).limit(1);
return newer ? null : result;
}
+2
View File
@@ -283,6 +283,8 @@ export const issuesApi = {
`/issues/${id}/stalled-review-decision`,
data,
),
resumeWorkspaceFinalization: (id: string, data: { actionId: string; runId: string; ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string }) =>
api.post<{ runId: string; resultId: string; status: "queued" }>(`/issues/${id}/recovery-actions/resume-workspace-finalization`, data),
retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) =>
api.post<{ runId: string; resultId: string; leaseId: string; status: "queued" }>(`/issues/${id}/recovery-actions/retry-workspace-export`, data),
resolveRecoveryAction: (
@@ -1161,3 +1161,10 @@ it("requires export retry for ordinary restoration while keeping explicit board
expect(document.body.textContent).not.toContain("Send for review");
expect(document.body.textContent).toContain("False positive, done");
});
it("requires owner-stop recovery even for board false-positive dispositions", () => {
const node = render(<IssueRecoveryActionCard action={buildAction({ kind: "active_run_watchdog", ownerType: "board",
cause: "native_workspace_finalization_owner_unverified" })} onResolve={() => {}} canFalsePositive />);
expect(node.querySelector("[data-testid='recovery-action-resolve-trigger']")).toBeNull();
expect(node.textContent).not.toContain("False positive");
});
@@ -1067,6 +1067,7 @@ export function IssueRecoveryActionCard({
} satisfies Record<RecoveryCardCardState, string>)[cardState];
const visibleResolveOptions = RESOLVE_OPTIONS.filter((option) => {
if (action.cause === "native_workspace_finalization_owner_unverified") return false;
if (isNativeWorkspaceExportRepairCause(action.cause) && ["todo", "done", "in_review"].includes(option.outcome)) return false;
if (option.outcome === "todo" && requiresExecutionReconciliation(action.cause)) return false;
if (option.boardOnly && !canFalsePositive) return false;
@@ -1,72 +1,44 @@
// @vitest-environment jsdom
import { act } from "react";
import { createRoot, type Root } from "react-dom/client";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import type { IssueRecoveryAction } from "@paperclipai/shared";
import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery";
const retry = vi.hoisted(() => vi.fn());
vi.mock("../api/issues", () => ({ issuesApi: { retryWorkspaceExport: retry } }));
const action = { id: "action", cause: "native_workspace_sync_out_retry_exhausted", status: "active", updatedAt: "2026-01-01T00:00:00Z", ownerType: "board", evidence: { runId: "run" }, wakePolicy: null } as unknown as IssueRecoveryAction;
const action = { cause: "native_workspace_finalization_owner_unverified", status: "escalated", ownerType: "board", evidence: { runId: "run", owner: { token: "owner", hostname: "secret-host", pid: 2 } }, wakePolicy: null } as unknown as IssueRecoveryAction;
describe("WorkspaceExportRecovery", () => {
let root: Root, container: HTMLDivElement, client: QueryClient;
const onQueued = vi.fn();
beforeEach(() => {
retry.mockReset().mockResolvedValue({ status: "queued" }); onQueued.mockReset();
container = document.createElement("div"); document.body.append(container); root = createRoot(container);
client = new QueryClient({ defaultOptions: { mutations: { retry: false } } });
});
afterEach(async () => { await act(async () => root.unmount()); client.clear(); container.remove(); });
async function mount(overrides: Partial<Parameters<typeof WorkspaceExportRecovery>[0]> = {}) {
await act(async () => root.render(<QueryClientProvider client={client}><WorkspaceExportRecovery issueId="issue" action={action} canManage onQueued={onQueued} {...overrides} /></QueryClientProvider>));
let root: Root, container: HTMLDivElement;
beforeEach(() => { container = document.createElement("div"); document.body.append(container); root = createRoot(container); });
afterEach(async () => { await act(async () => root.unmount()); container.remove(); });
async function mount(value: IssueRecoveryAction | null) {
await act(async () => root.render(<WorkspaceExportRecovery action={value} />));
expect(container.querySelector("textarea, button, details, input")).toBeNull();
expect(container.textContent).not.toContain("secret-host");
}
async function enterNote() {
const textarea = container.querySelector("textarea")!;
await act(async () => {
Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(textarea, "Restored provider connectivity while preserving all saved files.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
}
async function submit() {
await act(async () => container.querySelector("button")!.click());
await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); });
}
it("requires a repair note and submits only the exact recorded run", async () => {
await mount(); expect(container.querySelector("button")!.disabled).toBe(true);
await enterNote(); await submit();
expect(retry).toHaveBeenCalledWith("issue", { actionId: "action", runId: "run", repairNote: "Restored provider connectivity while preserving all saved files." });
expect(onQueued).toHaveBeenCalledOnce();
expect(container.querySelector('[role="status"]')?.textContent).toContain("agent will not repeat its work");
it("shows a quiet status while the harness restores work", async () => {
await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerRecovery: { state: "resuming" } } });
expect(container.querySelector('[role="status"]')?.textContent).toBe("Restoring saved work…");
});
it("offers repair again when a queued export publishes a new permanent failure", async () => {
await mount(); await enterNote(); await submit();
expect(container.querySelector("button")).toBeNull();
await mount({ action: { ...action, updatedAt: "2026-01-01T00:01:00Z", wakePolicy: null } });
expect(container.querySelector("button")?.textContent).toBe("Retry workspace export");
it("does not mistake platform identity for confirmed recovery", async () => {
await mount({ ...action, evidence: { ...action.evidence, owner: { ...(action.evidence.owner as object), worker: { provider: "railway", deploymentId: "old" } } } });
expect(container.textContent).toContain("operator assistance");
});
it("keeps unavailable-sandbox errors actionable inline", async () => {
retry.mockRejectedValueOnce(new Error("Resume the retained sandbox before retrying.")); await mount();
await enterNote(); await submit();
expect(container.querySelector('[role="alert"]')?.textContent).toContain("Resume the retained sandbox");
it("states that an operator is needed for legacy receipts", async () => {
await mount(action);
expect(container.textContent).toContain("Workspace recovery needs operator assistance");
});
it("does not offer the action without runtime access", async () => {
await mount({ canManage: false }); expect(container.querySelector("button")).toBeNull();
it("keeps export repair out of the task", async () => {
const cause = "native_workspace_sync_out_retry_exhausted";
await mount({ ...action, cause });
expect(container.textContent).toContain("operator assistance");
});
it("offers saved-result export after transient retries are exhausted without claiming an unsafe link", async () => {
await mount({ action: { ...action, cause: "native_workspace_sync_out_retry_exhausted" } });
expect(container.querySelector("button")?.textContent).toBe("Retry workspace export");
expect(container.textContent).toContain("export failure");
expect(container.textContent).not.toContain("unsafe link");
await enterNote(); await submit();
expect(retry).toHaveBeenCalledWith("issue", expect.objectContaining({ runId: "run" }));
it("removes the status when recovery resolves, including updates in place", async () => {
await mount({ ...action, evidence: { ...action.evidence, workspaceOwnerStop: {} } });
expect(container.textContent).toBe("Restoring saved work…");
await mount({ ...action, status: "resolved" }); expect(container.textContent).toBe("");
await mount(null); expect(container.textContent).toBe("");
});
it("never asks users to repair historical unsafe exports", async () => {
await mount({ action: { ...action, cause: "native_workspace_sync_out_unsafe_archive" } });
expect(container.textContent).toBe("");
expect(retry).not.toHaveBeenCalled();
});
it("does not offer a second retry while the same export is queued", async () => {
await mount({ action: { ...action, wakePolicy: { kind: "resume_native_run" } } });
expect(container.querySelector('[role="status"]')?.textContent).toContain("Export is queued"); expect(container.querySelector("button")).toBeNull();
it("does not show unrelated or historical unsafe-export notices", async () => {
await mount({ ...action, cause: "no_live_path", evidence: {} }); expect(container.textContent).toBe("");
await mount({ ...action, cause: "native_workspace_sync_out_unsafe_archive" }); expect(container.textContent).toBe("");
});
});
+11 -39
View File
@@ -1,45 +1,17 @@
import { useId, useState } from "react";
import { useMutation } from "@tanstack/react-query";
import type { IssueRecoveryAction } from "@paperclipai/shared";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { issuesApi } from "../api/issues";
import { cn } from "../lib/utils";
import { Button } from "./ui/button";
import { Label } from "./ui/label";
import { Textarea } from "./ui/textarea";
import { asObject, cn } from "../lib/utils";
/** Copyback repair preserves the accepted result and never wakes the provider. */
export function WorkspaceExportRecovery({ issueId, action, canManage, onQueued, className }: {
className?: string;
issueId: string; action: IssueRecoveryAction | null; canManage: boolean; onQueued: () => void;
/** Infrastructure repair belongs to the harness/operator, never a task form. */
export function WorkspaceExportRecovery({ action, className }: {
className?: string; action: IssueRecoveryAction | null;
}) {
const noteId = useId();
const [repairNote, setRepairNote] = useState("");
const [queuedActionVersion, setQueuedActionVersion] = useState<string | null>(null);
const retry = useMutation({
mutationFn: () => issuesApi.retryWorkspaceExport(issueId, {
actionId: action!.id, runId: action!.evidence.runId as string, repairNote: repairNote.trim(),
}),
onSuccess: () => { setQueuedActionVersion(String(action!.updatedAt)); onQueued(); },
});
if (!action || !isNativeWorkspaceExportRepairCause(action.cause) || action.ownerType !== "board"
const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified";
if (!action || (!ownerRecovery && !isNativeWorkspaceExportRepairCause(action.cause)) || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || typeof action.evidence.runId !== "string") return null;
const queued = queuedActionVersion === String(action.updatedAt) || action.wakePolicy?.kind === "resume_native_run";
return <section aria-label="Workspace export repair" className={cn("task-context-notice flex flex-col gap-2", className)}>
<p className="font-medium">Workspace export needs repair</p>
{queued ? <p role="status">Export is queued for the saved result. The agent will not repeat its work.</p> : <>
<p className="text-muted-foreground">{"Automatic workspace export retries stopped. Inspect the export failure, restore provider or destination availability, and preserve the saved files in the retained sandbox. Retry export here when the cause is resolved."}</p>
{canManage ? <>
<Label htmlFor={noteId}>Repair performed</Label>
<Textarea id={noteId} value={repairNote} onChange={event => setRepairNote(event.target.value)} maxLength={12_000}
placeholder="Describe the repair and how the saved workspace files were preserved." disabled={retry.isPending} />
<div className="flex justify-end">
<Button onClick={() => retry.mutate()} disabled={retry.isPending || repairNote.trim().length < 20}>
{retry.isPending ? "Queueing export…" : "Retry workspace export"}
</Button>
</div>
</> : <p className="text-muted-foreground">A board member with runtime access can retry this export after repair.</p>}
{retry.isError && <p role="alert" className="text-destructive">{retry.error instanceof Error ? retry.error.message : "Could not queue export. Refresh the task and inspect its run."}</p>}
</>}
</section>;
const queued = !!action.evidence.workspaceOwnerStop || action.wakePolicy?.kind === "resume_native_run";
const automatic = queued || (ownerRecovery && asObject(action.evidence.workspaceOwnerRecovery).state === "resuming");
return <p role="status" className={cn("text-sm text-muted-foreground", className)}>
{automatic ? "Restoring saved work…" : "Workspace recovery needs operator assistance. The agent’s result is saved."}
</p>;
}
+4 -4
View File
@@ -2497,7 +2497,7 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({
blockerAttention={blockerAttention}
successfulRunHandoff={successfulRunHandoff}
scheduledRetry={scheduledRetry}
recoveryAction={recoveryAction ?? null}
recoveryAction={recoveryAction?.cause === "native_workspace_finalization_owner_unverified" ? null : recoveryAction ?? null}
onResolveRecoveryAction={onResolveRecoveryAction}
onReissueIsolatedRecoveryAction={onReissueIsolatedRecoveryAction}
reissueIsolatedRecoveryActionPending={
@@ -7659,9 +7659,9 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
/>
{resolvedDetailTab === "chat" && (
<WorkspaceExportRecovery key={`workspace-export:${issue.activeRecoveryAction?.id ?? issue.id}`} issueId={issue.id}
<WorkspaceExportRecovery key={`workspace-export:${issue.activeRecoveryAction?.id ?? issue.id}`}
className={shellSectionClass}
action={issue.activeRecoveryAction ?? null} canManage={canManageBoardRuntime} onQueued={invalidateIssueDetail} />
action={issue.activeRecoveryAction ?? null} />
)}
{resolvedDetailTab === "chat" && issue.executionBlocker && (
<ExecutionBlockerNotice className={shellSectionClass} companyId={issue.companyId} issueId={issue.id} blocker={issue.executionBlocker} onRetried={invalidateIssueDetail} />
@@ -8046,7 +8046,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
blockerAttention={issue.blockerAttention ?? null}
successfulRunHandoff={issue.successfulRunHandoff ?? null}
scheduledRetry={issue.scheduledRetry ?? null}
recoveryAction={issue.activeRecoveryAction ?? null}
recoveryAction={issue.activeRecoveryAction?.cause === "native_workspace_finalization_owner_unverified" ? null : issue.activeRecoveryAction ?? null}
onResolveRecoveryAction={handleResolveRecoveryAction}
onReissueIsolatedRecoveryAction={
handleReissueIsolatedRecoveryAction
@@ -0,0 +1,29 @@
import type { Meta, StoryObj } from "@storybook/react-vite";
import { expect, within } from "storybook/test";
import { WorkspaceExportRecovery } from "@/components/WorkspaceExportRecovery";
import { automaticAction, queuedAction, recoveryAction } from "../../../support/workspace-finalization-recovery/fixtures";
const meta = {
id: "workspace-finalization-recovery-01-notice",
title: "Projects and execution/Recovery/Workspace finalization recovery/01 Notice",
component: WorkspaceExportRecovery,
args: { action: automaticAction },
parameters: { layout: "fullscreen", docs: { story: { inline: false }, description: { component: "Controller recovery runs in the harness. The task shows a quiet status and removes it when saving finishes. Legacy receipts without platform identity require operator assistance through the documented API; no shutdown form is shown to users." } } },
decorators: [Story => <div className="mx-auto max-w-2xl p-4 sm:p-8"><Story /></div>],
} satisfies Meta<typeof WorkspaceExportRecovery>;
export default meta;
type Story = StoryObj<typeof meta>;
export const RestoringSavedWork: Story = { play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
await expect(canvas.getByRole("status")).toHaveTextContent("Restoring saved work");
await expect(canvas.queryByRole("button")).not.toBeInTheDocument();
await expect(canvas.queryByRole("textbox")).not.toBeInTheDocument();
} };
export const NeedsVerification: Story = { args: { action: recoveryAction } };
export const VerificationUnavailable: Story = { args: { action: {
...automaticAction, evidence: { ...automaticAction.evidence, workspaceOwnerRecovery: { state: "operator_required" } },
} }, play: async ({ canvasElement }) => {
await expect(within(canvasElement).getByRole("status")).toHaveTextContent("operator assistance");
} };
export const AlreadyQueued: Story = { args: { action: queuedAction } };
export const Saved: Story = { args: { action: null } };
export const Mobile: Story = { globals: { viewport: { value: "mobile", isRotated: false } } };
@@ -0,0 +1,43 @@
import { useEffect, useState } from "react";
import type { Meta, StoryObj } from "@storybook/react-vite";
import { Route, Routes } from "@/lib/router";
import { Layout } from "@/components/Layout";
import { TaskChatBubble } from "@/components/task-chat/TaskChatBubble";
import { TaskChatComposer } from "@/components/task-chat/TaskChatComposer";
import { WorkspaceExportRecovery } from "@/components/WorkspaceExportRecovery";
import { PluginLauncherProvider } from "@/plugins/launchers";
import { useBreadcrumbs } from "@/context/BreadcrumbContext";
import { automaticAction } from "../../../support/workspace-finalization-recovery/fixtures";
function TaskConversation({ saved = false }: { saved?: boolean }) {
const [messages, setMessages] = useState<string[]>([]);
const { setBreadcrumbs } = useBreadcrumbs();
useEffect(() => { setBreadcrumbs([{ label: "Tasks", href: "/PAP/issues" }, { label: "Prepare the rollout plan", identifier: "PAP-24" }]); }, [setBreadcrumbs]);
return <main className="mx-auto flex min-h-full w-full max-w-3xl flex-col gap-6 px-4 py-6 sm:px-8">
<header className="flex flex-col gap-2">
<span className="font-mono text-xs text-muted-foreground">PAP-24</span>
<h1 className="text-xl font-semibold">Prepare the rollout plan</h1>
<p className="text-sm text-muted-foreground">Assigned to Jordan Lee</p>
</header>
<TaskChatBubble item={{ id: "request", kind: "message", author: "human", authorName: "Riley Board", authorUserId: "user-board", isCurrentUser: true, text: "Prepare the rollout plan and wait for my confirmation before starting implementation." }} />
<TaskChatBubble item={{ id: "saved-result", kind: "message", author: "agent", authorName: "Jordan Lee", text: "The plan is ready. I’m waiting for your confirmation before continuing." }} />
<WorkspaceExportRecovery action={saved ? null : automaticAction} />
{messages.map((text, index) => <TaskChatBubble key={index} item={{ id: `followup-${index}`, kind: "message", author: "human", authorName: "Riley Board", authorUserId: "user-board", isCurrentUser: true, text }} />)}
<div className="flex-1" />
<TaskChatComposer workMode="standard" onAdd={async body => setMessages(current => [...current, body])} placeholder="Message Jordan Lee…" />
</main>;
}
function Journey({ saved = false }: { saved?: boolean }) {
return <PluginLauncherProvider><Routes><Route path="/:companyPrefix" element={<Layout />}><Route path="issues/:issueId" element={<TaskConversation saved={saved} />} /></Route></Routes></PluginLauncherProvider>;
}
const meta = {
id: "workspace-finalization-recovery-02-task-journey",
title: "Projects and execution/Recovery/Workspace finalization recovery/02 Task journey", component: Journey, args: { saved: false },
parameters: { layout: "fullscreen", initialEntries: ["/PAP/issues/PAP-24"], docs: { description: { component: "A composed task conversation in the production app shell, using the shipped recovery notice, task bubbles and composer. Recovery happens automatically with no form or user action. This is a composed UI preview; finalization and approval resolution are verified separately by server tests." }, story: { inline: false } } },
} satisfies Meta<typeof Journey>;
export default meta;
type Story = StoryObj<typeof meta>;
export const ResumeInContext: Story = {};
export const SavedAndWaitingForConfirmation: Story = { args: { saved: true } };
export const Mobile: Story = { globals: { viewport: { value: "mobile", isRotated: false } } };
@@ -0,0 +1,37 @@
import type { IssueRecoveryAction } from "@paperclipai/shared";
export const issueId = "00000000-0000-4000-8000-000000000824";
export const stopEvidence = "The deployment platform confirms that previous-controller-7d4f and all its workspace-copyback processes exited. The saved workspace is retained.";
export const recoveryAction: IssueRecoveryAction = {
id: "00000000-0000-4000-8000-000000000825",
companyId: "company-storybook", sourceIssueId: issueId, recoveryIssueId: null,
kind: "active_run_watchdog", status: "escalated", ownerType: "board",
ownerAgentId: null, ownerUserId: null, previousOwnerAgentId: null, returnOwnerAgentId: null,
cause: "native_workspace_finalization_owner_unverified", fingerprint: "storybook-saved-result",
evidence: { runId: "00000000-0000-4000-8000-000000000826", owner: {
token: "00000000-0000-4000-8000-000000000827", hostname: "previous-controller-7d4f",
pid: 2, processStartedAt: "2026-10-10T00:02:55.560Z",
controllerBootId: "00000000-0000-4000-8000-000000000828",
} },
nextAction: "Verify the previous controller and its workspace-copyback processes have stopped. Resume workspace finalization without another provider turn.",
wakePolicy: null, monitorPolicy: null, attemptCount: 1, maxAttempts: 1,
timeoutAt: null, lastAttemptAt: "2026-10-10T00:10:00.000Z",
outcome: null, resolutionNote: null, resolvedAt: null,
createdAt: "2026-10-10T00:10:00.000Z", updatedAt: "2026-10-10T00:10:00.000Z",
};
export const queuedAction: IssueRecoveryAction = {
...recoveryAction, evidence: { ...recoveryAction.evidence, workspaceOwnerStop: {
owner: recoveryAction.evidence.owner, stopEvidence,
resultId: "00000000-0000-4000-8000-000000000829", actorId: "user-board",
confirmedAt: "2026-10-10T00:15:00.000Z",
} },
nextAction: "Workspace finalization is queued for the saved result. The agent will not repeat its work.",
};
export const automaticAction: IssueRecoveryAction = {
...recoveryAction, evidence: { ...recoveryAction.evidence, workspaceOwnerRecovery: { state: "resuming" }, owner: { ...(recoveryAction.evidence.owner as object), worker: {
provider: "railway", projectId: "fixture-project", environmentId: "fixture-environment", serviceId: "fixture-service", deploymentId: "fixture-old-deployment",
} } },
};