Files
PaperClipAI/doc/native-workspace-finalization-recovery.md
T
Devin FoleyandPaperclip bf55063567 fix(server): keep chat titles current and space verification retries (#22102)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Agent chats use persistent tasks, and remote runs verify their
provider files before execution.
> - A chat keeps the name that its agent had when the chat was created.
> - A fast gateway failure can consume both verification attempts before
the gateway recovers.
> - This change updates generated chat titles on an agent rename and
waits before a verification retry.
> - Chats show the current name, and a short gateway fault has time to
clear.

## Linked Issues or Issue Description

**What happened?**

Renaming an agent leaves existing generated chat titles unchanged.
Remote provider verification also retries a transport failure
immediately. Two fast HTTP 502 responses can fail startup before the
model starts.

**Expected behavior**

Generated chat titles follow an agent rename. A read-only verification
retry gives the remote gateway time to recover. Verification still
rejects incompatible files and stops after two attempts.

**Steps to reproduce**

1. Open a chat with an agent, then rename that agent. Inspect the task
linked from a run: its title still has the old agent name.
2. Make the remote verification transport return HTTP 502 for one
second. Start verification. Both immediate attempts fail before the
transport recovers.

Related: #22087 covers a provider connection failure during lease
acquisition. This change covers a later provider-file verification
request.

## What Changed

- Update matching generated chat titles in the agent configuration
transaction. Preserve custom chat titles, ordinary task titles, and
other agents' chats.
- Record each changed chat as `issue.updated` and publish its event
after commit so open pages refresh. Roll back the rename and queued
events if audit persistence fails.
- Derive new chat titles from the current agent name under the existing
company transaction lock. Both user-opened and proactive chats use this
path, including a create that read the old name before a rename
committed.
- Acquire that company lock before proactive chat delivery takes chat or
agent row locks, so delivery and renames use a consistent lock order.
- Wait one second before the second remote verification attempt. Keep
the existing attempt limit, transport timeouts, artifact checks, and
failure codes.
- Add route and startup-verification regressions. Document the retry
delay.
- Wait for the connection-review E2E fixture to save its Ask first
permission before it creates a task or navigates away.

## Verification

- Final acceptance on `3fb74ffaf6fa0ee2ecbe5dba3d01d252cb0c986f`: [full
CI
run](https://github.com/paperclipai/paperclip/actions/runs/44208144377)
is green, all 61 reported checks are complete with no failures, Greptile
is 5/5 on this exact commit, all review threads are resolved, and GitHub
reports the branch clean and mergeable.
- Red/green: the rename test retained the old titles before the fix. The
gateway recovery test consumed both attempts before the one-second
recovery point. Both pass with the changes.
- The live-update and concurrent-create tests failed on the first PR
revision: no title events were published, and a paused chat creation
retained the old name. The fixes queue events for publication after
commit and re-read the agent name under the company lock. Both
reproductions now pass.
- `pnpm exec vitest run
server/src/__tests__/agent-configuration-save.test.ts
server/src/__tests__/agent-chat-messages.test.ts
server/src/__tests__/agent-conversations.test.ts
server/src/services/native-runtime/remote-provider-pack.test.ts
server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
191 passed on macOS, including the concurrent rename/message regression.
- `pnpm --filter @paperclipai/server typecheck`: passed, including
source and test layers.
- Earlier CI on `de2a0a4` failed in the release-registry watchdog
self-test ([job
log](https://github.com/paperclipai/paperclip/actions/runs/44202511888/job/134460224014)).
Its 1.5-second watchdog fired during `starting the file's tests`; the
assertion expected `test "waits forever"`. Both reporter files are
unchanged from the base and current head. `node --test
scripts/ci/stalled-file-reporter.test.mjs` passed all 7 tests on macOS
with Node v25.6.1. The current revision also passed the Linux Typecheck
+ Release Registry CI check; no reporter code was changed.
- CI on `c83fcd9` failed the connection-review failure journey ([shard
log](https://github.com/paperclipai/paperclip/actions/runs/44204043415/job/134465938135)).
Navigation aborted the pending permission save, so the fixture tool ran
without creating the expected approval card. A temporary two-second
delay on that save reproduced the failure locally. With the response
wait, the same delayed-save reproduction passed. The temporary delay is
removed from the diff.
- `node scripts/typecheck-test-projects.mjs --reuse-declarations
tests/e2e`: passed using the server declarations from the same
application source revision.
- Before the rebase, the complete connection-review spec passed 4
journeys and skipped the controlled-restart journey under the default
configuration. On `95fb2aa`, rebased onto `9032f55d83e`, the
provider-failure journey, all 190 focused tests, server typecheck, and
E2E test-project typecheck passed again. The dedicated
controlled-restart suite was not run.
- A concurrent rename/message regression reproduced PostgreSQL `40P01:
deadlock detected` on `95fb2aa`. Delivery now takes the shared company
lock before the chat and agent locks. The regression uses observed
PostgreSQL blockers to stage the collision, and both operations commit
with the fix. Concurrent creation also waits behind the same lock and
retains the renamed title.
- The application tree for `3fb74ff` passed all 191 focused tests and
server typecheck. The subsequent rebase onto `96907f5518c` changed only
`.github/CODEOWNERS` compared with that tested tree. The full CI run for
this revision is now green.
- CI jobs on `95fb2aa` were interrupted after their runners received
shutdown signals: [server
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311777),
[workspace
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311921),
[typecheck/release
registry](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474310521),
and [serialized server
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311690).
The typecheck job is labelled failed, but its active step was cancelled.
These interruptions did not supply passing test evidence. The fresh CI
run on `3fb74ff` completed without failures.
- `pnpm verify --explain`: exit 2. The changed recovery document
requires broad CI coverage. Required CI supplies the full test and build
coverage; no full local suite or build was run.
- `git diff --check` and a Gitleaks scan of the diff passed. The diff
was also checked for personal data and private identifiers.

## Risks

- An inconclusive verification takes one extra second before its last
attempt. A persistent outage still fails with the existing named error.
No provider turn or artifact mutation is replayed by this retry.
- Only chat titles that match the previous generated title change.
Earlier stale titles and custom titles need an explicit correction.
- Proactive chat delivery now shares the existing company writer lock
with agent configuration. Concurrent writes in that company wait for the
active transaction to finish.
- No schema, session, checkpoint, launch identity, or pack identity
changes. State saved by the previous release carries forward with the
same verification checks and attempt limit.
- Revert this commit to restore the previous behavior. Existing title
updates are metadata and need no schema rollback.

## Model Used

- OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with source inspection,
browser investigation, and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:59:05 -08:00

46 KiB

Native workspace finalization ownership and recovery

Online legacy workspace import may preserve an unresolved terminal source while unrelated object-backed work continues. This is a file-authority handoff. It does not finish the old result, clear its recovery action, stop a provider, or start another turn. Keep every descriptor, retained allocation, and finalization checkpoint for the recovery owner.

The migration owner must hold the existing per-run physical finalizer advisory locks across capture and publication. It must reject a non-null coordinator owner, a durable physical copyback owner, a live controller lease, or an active adapter control. An expired owner is still an owner. Historical agent PID fields remain unchanged; they do not certify that the provider stopped. Recheck the exact source proof under publication row locks. A changed source or checkpoint keeps local authority. Provider stop verification and any later file export remain the recovery owner's separate obligations.

The same local file handoff can preserve a blocked prepared source after a native session is interrupted with ambiguous state, or after its file export recovery is exhausted. The interruption may precede a provider session. Its recorded session flag must be a boolean; it does not prove remote recovery. The interrupted source must retain its exact task failure block and declared recovery allocations. The exhausted source must retain its accepted result, bound original allocation, exhausted system recovery and blocked file recovery. An empty list of additional recovery allocations does not remove the original prepared allocation binding. The migration checks these records again under the same physical and task locks before publication. It keeps the failed run, accepted result, recovery action and remote file obligation unchanged.

Periodic heartbeat recovery runs one pass at a time in each server process. A tick that arrives during a slow pass reports its age and starts no duplicate. A later tick can start a fresh pass after the current work settles. Shutdown and idle sleep still track the actual recovery promise through cleanup. Direct run operations and scheduled work delivery keep their existing paths. A pass that remains active for three scheduler intervals produces an error log with its age. A waiting timeout does not admit another pass: the actual work must settle before ownership is released. A stuck operation requires investigation and recovery at the operation or provider boundary.

A native provider can fail before it submits a result while its workspace still contains unpublished edits. A prepared workspace reference is not an export receipt. Terminal provider failure pins the exact bound sandbox and records a file-recovery obligation in the same transaction as the failed run. Both setup failure handlers record that obligation before their terminal writer and provider cleanup run. This includes descriptor reads that fail inside runtime selection. Only its exact live recovery claim can settle an owned preparation failure; another owner or a stale claim prevents cleanup. Settling that claim and recording file retention share a transaction.

Modern remoteWorkspace preparation references use the same failure-retention producer as workspace-sync descriptors. A failed native run can also retain an expired awaiting_runner_reattach coordinator after its task execution lock was released. The orphan-lease owner may record a file-only preservation obligation for that historical state only when the exact allocation remains active, unreleased and ephemeral. It rechecks the task, run, coordinator generation, prepared workspace writer, complete allocation history, current controller and cleanup state under short row locks. A reusable allocation, accepted result, competing writer, manual recovery state or ambiguous cleanup stays held. The presence of a Stop-only marker remains a hold even when its value is invalid.

The transaction pins the existing StopAndRetain intent before provider I/O. The ordinary cleanup owner stops only the recorded allocation and rechecks all its current owners before dispatch. Normal checkpoint restart must first publish its task and coordinator claim; that claim prevents preservation intake. After a pin, the released task execution lock prevents an observed run from resuming itself. No row lock spans provider or archive I/O. Conflicting native or cleanup owners produce a nonblocking busy result after transaction rollback.

Only matching source and generation fingerprints plus the exact stopped-and-retained receipt let the native environment settlement owner retire that stale coordinator into manual terminal reconciliation. The existing workspace worker then verifies and saves the original files. The interrupted provider checkpoint and its unknown effects remain unknown; this path admits no provider turn, accepts no result and changes no task status. A changed or malformed modern workspace claim remains a cleanup hold even when it is ineligible for automatic preservation. It cannot fall through to destructive orphan cleanup. Invalid workspace authority is refused without throwing or stopping preservation of the next source in the batch. Already released or mismatched historical sources require their owning reconciliation and are not rearmed by this path.

The reconciliation candidate scan selects the complete workspace-sync record and the native status wake identifier from each saved run. It keeps their JSON types and reads full evidence later where finalization or recovery needs it. Saved execution input, runner profiles, and logs stay in the stored run.

Wake-queue release reads keep complete context and result receipts. Native runs omit runner profiles and log excerpts from this read; cleanup and execution read their own current evidence. Legacy runs retain the full release row so workspace-restore and pre-dispatch proofs keep their existing inputs. The selection is one scoped database query for either runtime.

The recovery card names the original run and requires evidence that its exact allocation stopped and its files were recovered. Recording that evidence does not retry the old turn, certify unknown external actions, change the current task owner or status, or require an accepted native result. Accepted-result export recovery and same-run provider recovery keep their existing paths. An export that has already finalized does not create this file obligation.

Native workspace export and merge acquire a PostgreSQL advisory lock scoped to company and run before the first physical copyback. The live heartbeat and the reconciler share that lock. Recovery skips a busy owner without recording another workspace operation or spending a retry. Recovery also rechecks the coordinator's terminal state and retry time under ownership: an earlier sweep snapshot cannot start another export after live finalization publishes a permanent repair or delay. A completed workspace barrier is reread under ownership before export, and a committed coordinator cannot be overwritten by a late failure receipt.

An unverified workspace owner keeps one board recovery action for its exact run and owner fingerprint. Repeated recovery checks do not consume attempts or rewrite that action. Task completion or cancellation does not prove the old physical writer stopped, so reads and recovery sweeps retain this action while the matching run still carries that owner receipt. After the receipt is cleared or replaced, ordinary stale-action cleanup applies. Cleanup is attributed to the system; historical cleanup from page reads does not add tasks to a person's inbox or refresh issue activity timestamps.

The lock transaction holds no row locks. Ordinary progress and finalization receipts remain visible through the normal database pool. A dedicated connection outside the application pool is reserved for the duration of copyback and closed when it settles; even a one-connection application pool remains available for progress writes. Its run profile carries nativeWorkspaceFinalizationOwner, an exact token, host, PID, and process-start receipt. Losing the lock connection does not prove physical copyback stopped: a contender still refuses a receipt whose controller is alive. The original callback joins before its token is released, and publication checks the lock connection and token. Graceful completion clears the receipt. If that cleanup write fails after the callback joins, only the same exact controller boot retains positive in-process join evidence and may resume after reconnecting; an unknown token or a new boot does not inherit that authority. A controller that has exited on the same host can be recovered automatically only when a durable successful workspace barrier proves its copyback finished. A dead parent can leave tar/Git children alive, so incomplete copyback requires operator stop verification even on the same host. PID reuse is checked against its recorded start time rather than trusted by PID alone.

When PostgreSQL configures an idle-transaction timeout, the owner probes its dedicated lock transaction at one third of that interval, capped at 30 seconds. The probe stops and joins before the transaction is released. This keeps a live copyback from losing its lock only because filesystem work has no database statements. It does not disable the database timeout or reconnect a failed lock session. Publication still requires the physical connection and the exact durable owner token. A disabled timeout adds no periodic probe.

Remote startup cancellation and retained uncertainty

Remote artifact staging can outlast the transport's shutdown preparation wait. The remote launcher keeps cancellation until staging settles and checks it before launch dispatch. A cancelled, undispatched launch starts no process. A stop during an in-flight launch waits for the exact process identity before the existing guarded signal path can run. An accepted signal request does not prove that the process stopped.

Read-only health retries are bounded by three attempts and 30 seconds. An unavailable health RPC, missing identity, or changed process generation retains the allocation and its files. A healthy listener alone is not identity or stop proof. Confirmed predecessor exit can permit bounded transport recovery, but an unrelated listener must remain untouched. A close timeout does not certify cleanup settlement, file restoration, or provider continuation admission.

A runner receipt that its reconnect grace expired ends that incident. The transport recognizes both the coded error and the runner's plain-text message. It must not open another recovery window after that receipt.

Verify file restoration independently of runner connection and provider activity. A file hash receipt proves the restored bytes only. The normal continuation gates must still check current ownership, pause, approval and budget state. Retained unknown effects remain unknown; successful file recovery does not authorize replay of a historical turn.

Automatic cloud recovery

New Railway copyback receipts record the exact project, environment, service and deployment ID. On takeover, the existing finalization reconciler asks the cloud harness to inspect that deployment. The harness resolves the stack's owned resources independently and issues a short-lived Ed25519 receipt only when the exact deployment is REMOVED with deploymentStopped: true. This covers all processes in the deployment, including orphaned copyback children. Missing, truncated, live or unavailable provider evidence leaves the physical fence intact.

The tenant verifies the configured harness key, disjoint receipt audience, stack, run, owner token, deployment identity and expiry. It records proof and an activity entry when replacing only the exact owner under the existing advisory lock. The normal reconciler resumes the accepted result; it does not create a provider turn or change approval/monitor waits. A changed owner during verification fails closed.

Stop proof alone does not authorize old work to overwrite a current task. After the provider read, automatic recovery uses the operator path's shared binding checks under short coordinator, issue and run row locks, atomically with the owner claim. The task must still be open, assigned to the run or its authoritative review participant, and bound to that running native execution and accepted result. A different checkout, newer execution or queued/running legacy execution prevents takeover. A later wake for the same assignee that has never started, such as the answer to the run's question, does not. Its claim waits for this run's execution lock, so it waits for this copyback. Counting it would block both runs. Another agent's run and a recovery-action wake still prevent takeover. Publication checks repeat this validation, so a wake that starts during copyback stops publication. Row locks are never held across physical copyback I/O.

Replaced deployment on a volume-backed service (automatic, no harness call). Railway mounts a service volume into one deployment at a time: it stops the previous deployment before the new one starts, and a service with a volume has no replicas. When this server runs on a Railway service with an attached volume (RAILWAY_VOLUME_ID and RAILWAY_VOLUME_MOUNT_PATH), and the receipt names the same project, environment and service with a different deployment ID, the shared owner-liveness primitive (proveOwnerGone) returns gone with reason deployment_replaced. That deployment's controller and every copyback child in it have exited, on whichever host they ran. The finalizer then claims the exact receipt as it does for a harness receipt, logs issue.workspace_owner_stop_verified with deploymentProof, marks the recovery resuming, and resumes the save with no provider turn. The same deployment ID, another service or environment, or a server without a volume gives no proof; the harness request below still applies. New receipts also record the mounted volume ID, and the proof then requires this server to mount that same volume. Receipts saved by earlier releases already record the deployment, so they take this path after an upgrade. The finalizer reaches this check only after it holds the run's copyback advisory lock, so the previous owner's lock session has also ended. A wait that this rule and the harness cannot settle names the Resume saved result action; its text never asks a person to inspect a server.

The harness endpoint is POST /v1/tenant/workspace-worker-stop. The fleet-wide server token authorizes a read request, never stop evidence: responses must be signed by the harness. This endpoint requires the companion cloud harness change. Deploying only the application keeps unknown owners fenced. No cloud provider credentials are delivered to tenant applications.

The task view shows only “Restoring saved work…” after the server confirms recovery or an operator queues it, then removes the notice when commitment resolves recovery. It exposes no process identifiers, shutdown form or manual resume control. Legacy receipts without platform identity and unsupported platforms say operator assistance is needed; operators use the documented exact-owner API below after independent stop proof. The three diagnosed historical receipts do not gain a deployment identity merely by upgrading. Never infer one from elapsed time or a hostname.

Unverified copyback after controller replacement

Generic orphan recovery does not turn a missing provider PID into process_lost while the native coordinator is workspace_finalizing with an accepted result bound to the same company, issue, and run. The finalizer owns that unfinished suffix, including the interval before it acquires its physical owner receipt. The heartbeat's phase or profile alone does not establish this authority. The conditional orphan write rechecks the durable result and the heartbeat's PostgreSQL tuple version, so a result published while the write waits on its row lock cannot be overwritten using an older statement snapshot. Result publishers lock and update that heartbeat in the same transaction as acceptance.

The running run keeps its source lease while the existing finalizer either completes, records its bounded failure policy, or requires operator stop verification. A new queued run or a healthy child task cannot clear an unverified copyback hold. Verified finalization clears the matching run's hold after commitment. Existing terminal-task dispositions and explicit cancellation remain unchanged; this does not recover a source already destroyed by an older controller or claim that an unfinished export succeeded.

The controller cannot verify a process on a foreign or unknown host, or orphaned copyback children after an abrupt parent death before the success barrier. It surfaces native_workspace_finalization_owner_unverified as board-owned recovery, with no automatic provider wake. This is an intentional limit: elapsed time or a missing database connection never proves the old copyback process stopped.

Same-host restart (automatic). A control-plane restart that kills the controller mid-copyback leaves a receipt whose owner can never confirm its own stop. Each receipt records the controller's process group (processGroupId). Copyback children (tar and Git) are spawned without detached, so they keep that group after their parent dies. A successor on the same host releases the receipt itself only when all of these hold: the receipt names another controller boot, the recorded PID is dead or recycled (process start time), and the recorded process group is empty, or its only remaining members predate the old controller or belong to the successor's own process tree. It logs issue.workspace_owner_stop_verified with that local proof, marks the recovery resuming, and the ordinary finalizer resumes copyback from the accepted result with no provider turn. A still-live group member keeps the fence and is rechecked automatically (local_stop_pending). A receipt without a process group, from another host, or on a platform without ps still needs the board confirmation below. The copyback advisory lock and the coordinator lease are unchanged.

An instance operator must first verify through the deployment platform that the exact prior controller and its copyback subprocesses have stopped. Retain the sandbox, accepted native result, and workspace descriptor. Do not run a new provider turn, delete workspace contents, or relax archive confinement.

For legacy receipts, an authorized company owner, company administrator or instance administrator records independently verified platform evidence through the operator API. The task shows this as its one action: its interruption is waiting_for_user with userAction: "confirm_owner_stop", and the chat offers Confirm the previous server stopped, which asks for the stop evidence and calls the same API. The control is shown only to roles that hold runtime:confirm_owner_stop. The API is POST /api/issues/:id/recovery-actions/resume-workspace-finalization with:

{
  "actionId": "<recovery-action-id>",
  "runId": "<run-id>",
  "ownerToken": "<owner-token-from-recovery-evidence>",
  "controllerAndCopybackStopped": true,
  "stopEvidence": "<deployment-platform evidence for this controller and its children>"
}

This is an operator attestation, not an automatic deployment-platform probe. The separate runtime:confirm_owner_stop authorization action limits this attestation to active company owners or administrators, the trusted local board, or an instance administrator. Ordinary runtime managers and agents cannot release the physical owner receipt. Missing role data keeps the client notice read-only. A missing advisory connection alone is insufficient. Admission rejects an active copyback lock, an observably live local controller, a changed owner, a newer run, or an unbound result. It records the exact owner and operator evidence atomically with releasing the receipt. The accepted result, workspace descriptor, source lease, task status, monitors, and approval waits are unchanged. The ordinary reconciler performs copyback and applies the saved result through the status arbiter. No provider wake is enqueued. The recovery action remains visible until finalization commits. Admission supports saved-result copyback from result_accepted, workspace_finalizing, and retryable_failure; an abandoned retry does not need a replacement run. The existing retry schedule remains unchanged. Repeating the same confirmation before reconciliation is idempotent; it cannot release a newer owner. Generic recovery resolution, including false-positive dispositions, cannot dismiss this hold or bypass saved-result finalization.

Repeated sweeps preserve these copyback-specific instructions, including for older actions whose recovery budget was already exhausted. They must never recommend starting a replacement provider run for this hold.

For older servers without this action, the maintenance fallback follows.

After that platform verification, use a database maintenance transaction to release only the exact receipt shown by the recovery action. Replace the four placeholders with the action's company, run, token, and source issue. The advisory lock prevents concurrent acquisition during this change; the token comparison prevents clearing a newer owner. A zero-row update means ownership changed and requires fresh inspection. This maintenance operation is for a full-control instance operator, not an agent tool.

BEGIN;
SELECT pg_advisory_xact_lock(hashtextextended(
  'native-workspace-finalization:<company-id>:<run-id>', 0));
UPDATE heartbeat_runs
SET runner_profile_json = runner_profile_json - 'nativeWorkspaceFinalizationOwner'
WHERE company_id = '<company-id>'::uuid
  AND id = '<run-id>'::uuid
  AND native_issue_id = '<issue-id>'::uuid
  AND runtime_mode = 'native'
  AND runner_profile_json->'nativeWorkspaceFinalizationOwner'->>'token' = '<owner-token>'
RETURNING id;
COMMIT;

Resolve the existing board recovery action with a note containing the platform stop evidence. The ordinary reconciliation sweep then resumes workspace finalization from the accepted result. Confirm the run's native phase and resultJson.finalizationPhase are committed, there is no nextAttemptAt, and no workspace operation is still running. The accepted provider result is reused.

The remote workspace save (remote-workspace-snapshot-archive.ts) records symlinks and never follows them. A link whose target resolves inside the workspace, such as node_modules/.bin/tool -> ../tool/cli.js, is saved as a link and restored as the same link. A link that is absolute or leaves the workspace through .., such as a headless Chromium profile's SingletonSocket -> /tmp/... or a link into a sandbox-only cache, has no portable meaning: the save records its path in the archive manifest's excluded list, omits it, and continues. Such a link no longer fails the save with remote_workspace_archive_unsafe_symlink. Restore materializes only the manifest entries, so an omitted link is absent after restore, and restore still refuses any archive whose manifest names an absolute or escaping link. Omitting a tracked link shows as a deletion in git status after restore.

Saves that recovery exhausted before this policy (capture policy version 1) are retried once automatically.

Automatic unsafe archive recovery

An unsafe link in a native workspace must not fail a completed task or require an operator to repair a sandbox. The accepted agent result remains authoritative.

Daytona validates every exported archive before extraction. If validation rejects an archive, export once more using files, directories, and relative symlinks whose resolved targets stay inside the workspace. The fallback does not follow or delete symlinks. It omits unsafe or unresolvable links, stores hard-linked files as ordinary bytes, and preserves directory exclusions and empty directories. The second archive passes the same confinement checks. A fixed informational message records the fallback in the provider log.

If native copyback still rejects the archive or its source confinement check, Paperclip discards that export, records workspace_export_omitted at info level in the local run log, and completes finalization using the original accepted result. It does not create a task warning, recovery card, repair request, or new provider turn. The normal completion policy still enforces ownership and explicit workflow constraints. Lost remote files are an accepted tradeoff.

The live path and restart finalizer use the same policy under workspace finalization ownership. Ownership loss, transport failures, missing sandboxes, and other unrelated errors retain their existing handling. No unsafe archive is extracted. No host path or link target is copied into the informational run event.

Repairing a failed workspace export without rerunning the agent

The background workspace worker runs up to four copy-backs at once, one per task, so a slow sandbox export does not hold every other task's retry. Due work runs oldest first. Work without a saved retry time, such as a copy-back that a restart interrupted, is ordered by when it became pending; it does not go ahead of older retries that are already due.

Transient export failures retry three times, then produce native_workspace_sync_out_retry_exhausted. The accepted result stays saved, and the exact sandbox is stopped and retained for export-only recovery. The reason identifies a transport/copyback failure, not an inferred disk-space cause. New unsafe archives use the automatic policy above and do not create this hold. Historical unsafe-archive holds recover automatically as described below; they do not use this operator flow.

A board member with runtime management access can complete the saved result:

  1. Read the recovery action, run, and environment lease. Verify the company, run, provider sandbox ID, and stopped-provider receipt. Preserve that exact sandbox; do not acquire a replacement or seed the host workspace over it.
  2. Through the provider console or official SDK, resume that sandbox and inspect the export failure while preserving all user files. Restore transport access or other provider prerequisites before retrying. Extending the sandbox retention window can be necessary during this repair.
  3. Use the operator API to record the repair and request export-only retry. The task shows operator assistance is needed, without an infrastructure form. The board API is POST /api/issues/:id/recovery-actions/retry-workspace-export with { "actionId": "<recovery-action-id>", "runId": "<run-id>", "repairNote": "<repair and preservation evidence>" }.
  4. Confirm the same run commits, the saved result determines the task outcome, and the recovery action resolves. No new provider turn or wake is created. A reusable repair sandbox is stopped and retained again. An ephemeral sandbox follows its original destroy-after-turn policy only after the exact result commits and the workspace reference contains a finalized host-copy receipt.

Admission requires the same accepted result, task owner, descriptor, and lease; confirmed prior provider stop; an available repaired sandbox; and no newer task execution or competing current lease. It shares finalization ownership and resumes only the recorded provider lease through its verified lifecycle method. That method drains old activity and verifies the saved workspace identity before reopening the provider's controller admission gate; an external console restart alone does not reopen that gate. Admission rejects a missing or replacement sandbox and revalidates after probing the exact workspace. A changed binding or unavailable sandbox returns 409 without reopening work. A duplicate queued request is idempotent. Unsafe archives use automatic salvage or omission during retry. Generic ordinary recovery resolution cannot retry, mark done, or send the task for review in place of export-only retry. Explicit board false-positive/cancellation dispositions remain deliberate overrides; they do not claim successful copyback.

Before publishing terminal export failure, the finalizer atomically records a stop-only intent with the failed run. This includes ephemeral allocations: ordinary release policy cannot delete their unexported files. A crash before cleanup leaves enough exact lease, result, and plugin authority for restart recovery. The same intent remains while export-only retry is active.

Before resuming the retained sandbox, the controller durably records a stop-only cleanup intent on that exact lease and removes the old stopped receipt. If the resume reply is lost, probing fails, or admission changes, it stops and retains the sandbox. A failed stop remains pending_cleanup; a restarted controller's bounded cleanup sweep retries the verified stop without destroying saved files. An in-flight request holds a 15-minute cleanup claim, so a controller crash may delay that sweep until the claim expires. Neither an unconfirmed stop nor a stale receipt grants admission, and a changed lease or competing sandbox owner prevents cleanup from taking ownership. Retry export after the lease has a new confirmed stopped receipt. No provider turn is created by this recovery. Once recovery has saved the files and queued destruction of the stopped sandbox, the cleanup worker admits its first attempt without the five-minute retry delay. The shared worker spacing and any in-flight claim deadline still apply. A failed first attempt keeps the existing retry delay, attempt limit and ownership checks. Cleanup and its readiness probe use the plugin ID recorded on that lease. Another plugin with the same provider name cannot take over; an unavailable original plugin defers cleanup without consuming an attempt. Stop-only cleanup requires the exact plugin worker to advertise environmentStopLease. Older plugins without that hook receive neither release nor destroy; resume admission also defers until safe compensation is available. New intents use schema v2 with an explicit plugin pin. A v1 intent created before that field existed remains recoverable using only the plugin ID already recorded on its exact lease; an explicit mismatched pin is still rejected.

The opt-in native-workspace-export-resume.live.test.ts is a provider-boundary fault integration, separate from the browser Product E2E. After building the Daytona plugin, run that exact Vitest file with PAPERCLIP_LIVE_EXPORT_RESUME=1, DAYTONA_API_KEY, and PAPERCLIP_LIVE_EXPORT_RESUME_IMAGE set to an immutable image digest. It creates one disposable ephemeral sandbox and database, injects probe and stop transport failures, and verifies a fresh runtime can stop the sandbox while preserving exact nonce bytes. It also injects three transient failures at the production finalizer boundary and verifies retained work plus export-only retry admission. This boundary test does not claim physical copyback or result commitment; historical browser Product E2E supplied that proof before automatic unsafe-export recovery. It deletes only that owned fixture after proof.

Historical unsafe exports

The reconciliation sweep automatically recovers accepted results that an older controller terminalized with native_workspace_sync_out_unsafe_archive. It omits the already-rejected export without accessing the old provider, so a stopped, unavailable, or deleted sandbox cannot require user repair. The omission barrier and result re-admission are atomic under finalization ownership. The normal status arbiter still enforces current ownership, contracts, and gates.

This works with active, incorrectly resolved, or missing old repair actions. It clears the matching stale unsafe notices instead of restoring them. Newer runs, changed task ownership or contracts, another active recovery action, and explicit operator dispositions prevent replay of old results. No provider turn is created. Diagnostics appear only in the local run log; no unsafe-export activity notice, repair card, or recovery chip is created. Existing stop-only allocation intents retain their exact ownership and cleanup protections.

Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.

New ephemeral native allocations receive an acquisition-time workspace sentinel bound to their run and provider allocation. Export-only resume requires that proof and the unchanged durable intent. A legacy ephemeral allocation without this proof remains blocked; recovery never creates a sentinel or accepts a replacement during resume. The reusable workspace identity hash is unchanged.

Disabling auto-delete retains provider storage while repair waits. No automatic lease-age sweep deletes these released, confirmed-stopped allocations. Storage costs can continue until committed copyback applies the original cleanup policy or an operator explicitly deletes the allocation.

Generic stop-only cleanup

An explicit sandbox stop_and_retain also requires the dedicated provider hook when no export repair is involved, including successful per-turn reusable runs and startup cancellation. Before dispatch, the controller stores a separate sandboxStopAndRetain intent bound to the company, run, lease, provider allocation, and original plugin. The initial cleanup and restart sweep both require that plugin to advertise environmentStopLease. Missing capability or failed stop leaves pending cleanup; ordinary release and destroy are never fallback methods. A typed terminal stop failure (PluginEnvironmentStopError with reason resource_unrecoverable or recovery_failed) escalates the lease for operator attention on its first attempt and uses the long retry interval. The lease stays in pending cleanup.

A matching stopped receipt clears the pending intent, leaves the lease released and resumable, and saves sandboxStopAndRetainReceipt. Its request identity, original plugin and method: "environmentStopLease" attest this dispatch route. The accepted-result export intent and its commitment requirements are unchanged.

The built-in fake provider uses its explicit stopLease operation rather than a plugin RPC. Its intent pins the registered built-in provider and its receipt says builtin.stopLease. This provider owns no real process or filesystem; the receipt models its lifecycle. Missing built-in stop support or an unconfirmed receipt also remains pending through restart, without calling release or destroy.

Legacy copy-back after a confirmed stop

A legacy adapter can report failed workspace copy-back after cancellation has already stopped and released its sandbox lease. The host records the exact lease snapshot from before adapter dispatch. For an originally ephemeral source, the recorder can preserve the released allocation for repair only while the current lease is still ephemeral and its company, run, issue, workspace, provider, allocation, acquisition time, plugin, and realized root still match. It requires both the dedicated stop-and-retain receipt and the matching stopped-execution receipt. A competing owner or changed source fails closed. The recorder leaves release status and receipts intact, excludes the allocation from reuse, and creates the existing board-owned file-repair obligation without replaying work or changing a newer run or the task's status.

This late-recording path does not admit reusable leases. Provider resume can precede the database handoff, so an old stop receipt and a database row lock cannot prove that reuse has not started. Existing active-source and previously recorded recovery paths retain their behavior. A missing or rejected original snapshot does not authorize adopting another released lease.

A retained source receipt records control-plane evidence; it does not prove the provider still holds the original bytes or that copy-back succeeded. Operators must verify and recover the exact source separately. Later successful runs and completed tasks do not certify recovery of earlier workspace files.

Quiet task recovery and provider verification

The task conversation does not ask its user to certify recovered infrastructure files. A source-bound recovery with a live native run or an eligible future bounded retry uses the normal working presentation. A missed retry, exhausted budget, unknown action outcome, or unresolved file hold remains visible. The notice gives a brief explanation and a run-inspection link; stored diagnostics and operator guidance remain under Recovery details. Source-only repair remains available through the board-authorized recovery resolution API and its original stop-receipt and evidence checks.

After a remote provider pack passes full integrity verification and the session passes authenticated admission, the controller saves its manifest and runner binary identity in provider-pack-authority.json. The receipt binds the company, agent, workspace, session, runner instance, environment lease, remote working directory, and provider descriptor. It stays in private controller state, outside guest workspace and harness backup directories. Projectless runs use their original environment lease as the workspace identity because each run otherwise has a new workspace placeholder. Real workspace identities still require an exact match. A receipt write failure warns and returns the admitted session to its normal owner; it does not discard the live session handle. Future recovery still fails closed without matching authority.

Exact process reattachment uses that saved release authority. It still verifies the full remote pack, runner protocol and capabilities, exact process identity, and authenticated PRP handshake. A new controller release alone does not require a live provider to change its executable files. Fresh launches and explicit session replacement select current release artifacts. Invalid or mismatched receipts fail closed. Older sessions without a receipt retain the strict current release gate; a sandbox's own manifest cannot supply missing launch authority.

Remote provider-pack verification retries an interrupted read at most once. The retry waits one second so a fast gateway failure has time to recover. Each command retains its 30-second transport timeout. A timeout, signal, absent exit status, or transport exception does not prove artifact incompatibility. Two inconclusive attempts hold the original run with runner_remote_provider_verification_unavailable. Reattachment does not spend a new provider attempt, so it cannot renew this exhausted read budget. A completed nonzero verification, digest mismatch, or wrong OpenCode version remains incompatible and cannot authorize a new provider turn. Recovery retains the exact source workspace and existing ownership gates; these read-only retries do not replay tools or alter a retained pack. Verification errors include bounded, redacted output from both transport streams, including providers that combine stderr into stdout.

Continue after confirmed source destruction

When an old export failed and its exact sandbox has a confirmed destruction receipt, the task can continue from surviving attachments. This is reconstruction, not recovery of the missing edits. The task's recovery notice exposes Continue from saved sources to board operators with runtime:manage. Select the surviving attachments, describe the missing work, and record the observed external action outcomes. A retained or unverified sandbox must use the existing export or file recovery path instead. The source task may still display in_progress after an older controller's run-only Stop. Reconstruction accepts that inactive state only with no execution or checkout owner and the same source, recovery and status-version checks; an operator does not need to manufacture a separate blocker merely to record recovery.

POST /api/issues/{id}/recovery-actions/reconstruct-workspace records this choice. It validates the source run, accepted result, prepared descriptor, exact destroyed lease, current task owner and status version. It rejects active copyback owners, competing recovery, unreconciled newer executions, and attachments from another task or company. The transaction records attachment hashes and a durable fresh-session delivery intent. The old run, result, descriptor and failed finalization remain intact. Repeated submissions adopt the original intent. Normal admission rechecks source identity, attachments, authority, budgets, holds and task eligibility. It preserves the retry count and does not grant a new work window or replay completed actions. The agent must verify the selected attachments before rebuilding files. Queued successors recheck their exact reconstruction receipt, sources and current execution gates when claimed and immediately before provider handoff. A later human block cannot be overwritten by automatic checkout. Bounded retries keep the same reconstruction authority. Dispatch verifies the system wake and predecessor chain back to the approved first run. It rechecks the saved sources, remaining retry limits and current recovery obligations. Unrelated executions and uncertain predecessor outcomes still stop dispatch.

A later native run-only cancellation may coexist with the older loss decision only after its exact Stop acknowledgement, stopped lease receipts, and board file-only repair have been recorded. Its external-action outcome must be not_performed; an unknown or performed action stays held. Reconstruction saves the identities and hashes of those repair decisions. Queue claim and dispatch recheck the same evidence, so a changed receipt or another later run requires a fresh decision. The cancelled run stays cancelled and is never replayed.

Stop also preserves a native workspace prepared before paperclip_finish. The executor pins its files before releasing the environment. Startup and periodic recovery process a bounded batch of historical acknowledged cancellations that missed this step. The existing stop-only cleanup worker retains their exact sandboxes and records provider receipts; this does not reopen a task or start another turn. The normal file-only repair control remains required before an older lost-work reconstruction can proceed. If file retention loses the task lock, teardown keeps the source active for a later pass. Historical sources already stopping or stopped remain eligible under the existing exact-source checks. A stopped source requires the independent stop-and-retain receipts and must not have a competing allocation owner. Active warm allocations keep the same retention retry path. Only historical stopping or stopped sources require the ephemeral policy, because a reusable released allocation may already have a new owner. Daytona stop-only cleanup also accepts a freshly verified archived allocation. It first confirms that provider auto-delete is disabled. It leaves the archive cold, with its files preserved, and records the process as stopped. For a cancelled native run, the file-recovery form also offers an optional confirmation that no external actions were performed. Leave it unchecked unless the run history verifies that fact. The default records files without certifying action outcomes. Changing the selected source clears the evidence and confirmations.

Committed chat response authorization reads current run identity, completion binding, and chat source fields under the existing nonblocking share lock. It excludes saved execution input, runner profiles, and log excerpts. Actual retry admission still reads the complete run and its recovery evidence. These reads preserve stored run data, source revocation checks, and JSON value types. External review-wait discovery selects only chat authorization context fields. It keeps the current endpoint, membership, resource, and question-receipt checks; saved prompts remain stored and do not cross this discovery query.

Legacy continuation recovery rejects native runs before it loads their saved execution input. The persisted disposition check reads only the task binding for a native run. It keeps the same task-binding and native-finalization skip reasons, including the original nonblank binding values. Native input and history stay stored. Real legacy recovery still reads its complete context, profile, excerpts, and required failure evidence.

The native Stop continuation recheck reads current run identity, the full Stop acknowledgement receipt, and process identity. It does not load saved execution input, runner profiles, excerpts, or unrelated result fields. Lease cleanup, remote termination receipts, local process checks, task ownership, and admission checks keep their existing behavior. Stored runs retain all of their data.

One periodic stranded-message pass coalesces successful native release checks for the same company, task, source run, and current source row version. The release path already drains that task's queue through current admission checks. Each saved request still advances its recovery cursor. Rejected attempts, changed source runs, and updated source evidence remain eligible, and the next pass checks current state again. Legacy per-request handling stays unchanged. Pending work is awaited before a source is marked checked.

Retained saved-work recovery for a finished run uses the same owner receipt. When the receipt names a controller that a restart killed and no automatic proof can release it, the recovery worker does not spend an attempt on each recheck. It records the exact receipt on the retained recovery action as workspaceOwnerRecovery.state = "operator_required" and keeps rechecking within the bounded recovery window. A task Retry waiting on that saved result reports workspace_owner_stop_required and does not offer another Retry, because another click cannot release the receipt. The board confirms the stop through the same resume-workspace-finalization route. The task chat shows the same single Confirm the previous server stopped control in place of the retained file recovery controls while this hold is active, with the recorded receipt token, to roles that hold runtime:confirm_owner_stop. The confirmation must name the receipt that is still on the run. It clears only that receipt and returns the existing recovery to the worker (a spent budget gets one new generation). It accepts no result and starts no agent turn. The waiting Retry then continues through its normal path. If the recovery window ends first, the board handoff names the same confirmation instead of advising another Retry.