mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
Project current chat authorization fields for external review-wait discovery instead of saved execution input. Preserve typed source bindings, nested question receipts, candidate eligibility, live revocation checks, and stored context. Validation: 181 native chat integration tests, full build and typecheck, exact-head Greptile 5/5 (medium), full green CI, and no unresolved review threads. The full-context counterfactual fails both direct-chat size regressions. Co-Authored-By: Paperclip <noreply@paperclip.ing>
519 lines
34 KiB
Markdown
519 lines
34 KiB
Markdown
# Native workspace finalization ownership and recovery
|
|
|
|
Periodic heartbeat recovery runs one pass at a time in each server process.
|
|
A tick that arrives during a slow pass reports its age and starts no duplicate.
|
|
A later tick can start a fresh pass after the current work settles. Shutdown
|
|
and idle sleep still track the actual recovery promise through cleanup. Direct
|
|
run operations and scheduled work delivery keep their existing paths. A pass
|
|
that remains active for three scheduler intervals produces an error log with its
|
|
age. A waiting timeout does not admit another pass: the actual work must settle
|
|
before ownership is released. A stuck operation requires investigation and
|
|
recovery at the operation or provider boundary.
|
|
|
|
A native provider can fail before it submits a result while its workspace still
|
|
contains unpublished edits. A prepared workspace reference is not an export
|
|
receipt. Terminal provider failure pins the exact bound sandbox and records a
|
|
file-recovery obligation in the same transaction as the failed run. Both setup
|
|
failure handlers record that obligation before their terminal writer and provider
|
|
cleanup run. This includes descriptor reads that fail inside runtime selection. Only its exact live recovery claim can settle an owned
|
|
preparation failure; another owner or a stale claim prevents cleanup. Settling
|
|
that claim and recording file retention share a transaction.
|
|
|
|
The reconciliation candidate scan selects the complete workspace-sync record and
|
|
the native status wake identifier from each saved run. It keeps their JSON types
|
|
and reads full evidence later where finalization or recovery needs it. Saved
|
|
execution input, runner profiles, and logs stay in the stored run.
|
|
|
|
Wake-queue release reads keep complete context and result receipts. Native runs
|
|
omit runner profiles and log excerpts from this read; cleanup and execution
|
|
read their own current evidence. Legacy runs retain the full release row so
|
|
workspace-restore and pre-dispatch proofs keep their existing inputs. The
|
|
selection is one scoped database query for either runtime.
|
|
|
|
The recovery card names the original run and requires evidence that its exact
|
|
allocation stopped and its files were recovered. Recording that evidence does
|
|
not retry the old turn, certify unknown external actions, change the current task
|
|
owner or status, or require an accepted native result. Accepted-result export
|
|
recovery and same-run provider recovery keep their existing paths. An export
|
|
that has already finalized does not create this file obligation.
|
|
|
|
Native workspace export and merge acquire a PostgreSQL advisory lock scoped to
|
|
company and run before the first physical copyback. The live heartbeat and the
|
|
reconciler share that lock. Recovery skips a busy owner without recording another
|
|
workspace operation or spending a retry. Recovery also rechecks the coordinator's terminal state and retry time under ownership: an earlier sweep snapshot cannot
|
|
start another export after live finalization publishes a permanent repair or delay.
|
|
A completed workspace barrier is reread
|
|
under ownership before export, and a committed coordinator cannot be overwritten
|
|
by a late failure receipt.
|
|
|
|
An unverified workspace owner keeps one board recovery action for its exact run
|
|
and owner fingerprint. Repeated recovery checks do not consume attempts or
|
|
rewrite that action. Task completion or cancellation does not prove the old
|
|
physical writer stopped, so reads and recovery sweeps retain this action while
|
|
the matching run still carries that owner receipt. After the receipt is cleared
|
|
or replaced, ordinary stale-action cleanup applies. Cleanup is attributed to
|
|
the system; historical cleanup from page reads does not add tasks to a person's
|
|
inbox or refresh issue activity timestamps.
|
|
|
|
The lock transaction holds no row locks. Ordinary progress and finalization
|
|
receipts remain visible through the normal database pool. A dedicated connection outside the application pool is
|
|
reserved for the duration of copyback and closed when it settles; even a one-connection application pool remains available for progress writes. Its run profile carries
|
|
`nativeWorkspaceFinalizationOwner`, an exact token, host, PID, and process-start
|
|
receipt. Losing the lock connection does not prove physical copyback stopped:
|
|
a contender still refuses a receipt whose controller is alive. The original
|
|
callback joins before its token is released, and publication checks the lock
|
|
connection and token. Graceful completion clears the receipt. If that cleanup write fails after the
|
|
callback joins, only the same exact controller boot retains positive in-process
|
|
join evidence and may resume after reconnecting; an unknown token or a new boot
|
|
does not inherit that authority. A controller that
|
|
has exited on the same host can be recovered automatically only when a durable
|
|
successful workspace barrier proves its copyback finished. A dead parent can
|
|
leave tar/Git children alive, so incomplete copyback requires operator stop
|
|
verification even on the same host. PID reuse is checked against its recorded
|
|
start time rather than trusted by PID alone.
|
|
|
|
When PostgreSQL configures an idle-transaction timeout, the owner probes its
|
|
dedicated lock transaction at one third of that interval, capped at
|
|
30 seconds. The probe stops and joins before the transaction is released. This
|
|
keeps a live copyback from losing its lock only because filesystem work has no
|
|
database statements. It does not disable the database timeout or reconnect a
|
|
failed lock session. Publication still requires the physical connection and
|
|
the exact durable owner token. A disabled timeout adds no periodic probe.
|
|
|
|
## Automatic cloud recovery
|
|
|
|
New Railway copyback receipts record the exact project, environment, service and
|
|
deployment ID. On takeover, the existing finalization reconciler asks the cloud
|
|
harness to inspect that deployment. The harness resolves the stack's owned
|
|
resources independently and issues a short-lived Ed25519 receipt only when the
|
|
exact deployment is `REMOVED` with `deploymentStopped: true`. This covers all
|
|
processes in the deployment, including orphaned copyback children. Missing,
|
|
truncated, live or unavailable provider evidence leaves the physical fence intact.
|
|
|
|
The tenant verifies the configured harness key, disjoint receipt audience, stack,
|
|
run, owner token, deployment identity and expiry. It records proof and an activity
|
|
entry when replacing only the exact owner under the existing advisory lock. The
|
|
normal reconciler resumes the accepted result; it does not create a provider turn
|
|
or change approval/monitor waits. A changed owner during verification fails closed.
|
|
|
|
Stop proof alone does not authorize old work to overwrite a current task. After
|
|
the provider read, automatic recovery uses the operator path's shared binding
|
|
checks under short coordinator, issue and run row locks, atomically with the
|
|
owner claim. The task must still be open, assigned to the run or its authoritative
|
|
review participant, and bound to that running native execution and accepted
|
|
result. A different checkout, newer execution or queued/running legacy execution
|
|
prevents takeover. Publication checks repeat this validation; row locks are never
|
|
held across physical copyback I/O.
|
|
|
|
The harness endpoint is `POST /v1/tenant/workspace-worker-stop`. The fleet-wide
|
|
server token authorizes a read request, never stop evidence: responses must be
|
|
signed by the harness. This endpoint requires the companion cloud harness change.
|
|
Deploying only the application keeps unknown owners fenced. No cloud provider
|
|
credentials are delivered to tenant applications.
|
|
|
|
The task view shows only “Restoring saved work…” after the server confirms recovery
|
|
or an operator queues it,
|
|
then removes the notice when commitment resolves recovery. It exposes no process
|
|
identifiers, shutdown form or manual resume control. Legacy receipts without
|
|
platform identity and unsupported platforms say operator assistance is needed;
|
|
operators use the documented exact-owner API below after independent stop proof.
|
|
The three diagnosed historical receipts do not gain a deployment identity merely
|
|
by upgrading. Never infer one from elapsed time or a hostname.
|
|
|
|
## Unverified copyback after controller replacement
|
|
|
|
Generic orphan recovery does not turn a missing provider PID into `process_lost`
|
|
while the native coordinator is `workspace_finalizing` with an accepted result
|
|
bound to the same company, issue, and run. The finalizer owns that unfinished
|
|
suffix, including the interval before it acquires its physical owner receipt.
|
|
The heartbeat's phase or profile alone does not establish this authority.
|
|
The conditional orphan write rechecks the durable result and the heartbeat's
|
|
PostgreSQL tuple version, so a result published while the write waits on its row
|
|
lock cannot be overwritten using an older statement snapshot. Result publishers
|
|
lock and update that heartbeat in the same transaction as acceptance.
|
|
|
|
The running run keeps its source lease while the existing finalizer either
|
|
completes, records its bounded failure policy, or requires operator stop
|
|
verification. A new queued run or a healthy child task cannot clear an
|
|
unverified copyback hold. Verified finalization clears the matching run's hold
|
|
after commitment. Existing terminal-task dispositions and explicit cancellation
|
|
remain unchanged; this does not recover a source already destroyed by an older
|
|
controller or claim that an unfinished export succeeded.
|
|
|
|
The controller cannot verify a process on a foreign or unknown host, or orphaned
|
|
copyback children after an abrupt parent death before the success barrier. It surfaces
|
|
`native_workspace_finalization_owner_unverified` as board-owned recovery, with no
|
|
automatic provider wake. This is an intentional limit: elapsed time or a missing
|
|
database connection never proves the old copyback process stopped.
|
|
|
|
An instance operator must first verify through the deployment platform that the
|
|
exact prior controller **and its copyback subprocesses** have stopped. Retain the
|
|
sandbox, accepted native result, and workspace descriptor. Do not run a new
|
|
provider turn, delete workspace contents, or relax archive confinement.
|
|
|
|
For legacy receipts, an authorized company owner, company administrator or
|
|
instance administrator records independently verified platform evidence through
|
|
the operator API. The task has no shutdown form or resume control. The API is
|
|
`POST /api/issues/:id/recovery-actions/resume-workspace-finalization` with:
|
|
|
|
```json
|
|
{
|
|
"actionId": "<recovery-action-id>",
|
|
"runId": "<run-id>",
|
|
"ownerToken": "<owner-token-from-recovery-evidence>",
|
|
"controllerAndCopybackStopped": true,
|
|
"stopEvidence": "<deployment-platform evidence for this controller and its children>"
|
|
}
|
|
```
|
|
|
|
This is an operator attestation, not an automatic deployment-platform probe.
|
|
The separate `runtime:confirm_owner_stop` authorization action limits this
|
|
attestation to active company owners or administrators, the trusted local board,
|
|
or an instance administrator. Ordinary runtime managers and agents cannot release
|
|
the physical owner receipt. Missing role data keeps the client notice read-only.
|
|
A
|
|
missing advisory connection alone is insufficient. Admission rejects an active
|
|
copyback lock, an observably live local controller, a changed owner, a newer run,
|
|
or an unbound result. It records the exact owner and operator evidence atomically
|
|
with releasing the receipt. The accepted result, workspace descriptor, source
|
|
lease, task status, monitors, and approval waits are unchanged. The ordinary
|
|
reconciler performs copyback and applies the saved result through the status
|
|
arbiter. No provider wake is enqueued. The recovery action remains visible until
|
|
finalization commits. Admission supports saved-result copyback from `result_accepted`,
|
|
`workspace_finalizing`, and `retryable_failure`; an abandoned retry does not need
|
|
a replacement run. The existing retry schedule remains unchanged.
|
|
Repeating the same confirmation before reconciliation is
|
|
idempotent; it cannot release a newer owner. Generic recovery resolution, including
|
|
false-positive dispositions, cannot dismiss this hold or bypass saved-result
|
|
finalization.
|
|
|
|
Repeated sweeps preserve these copyback-specific instructions, including for
|
|
older actions whose recovery budget was already exhausted. They must never
|
|
recommend starting a replacement provider run for this hold.
|
|
|
|
For older servers without this action, the maintenance fallback follows.
|
|
|
|
After that platform verification, use a database maintenance transaction to
|
|
release only the exact receipt shown by the recovery action. Replace the four
|
|
placeholders with the action's company, run, token, and source issue. The advisory
|
|
lock prevents concurrent acquisition during this change; the token comparison
|
|
prevents clearing a newer owner. A zero-row update means ownership changed and
|
|
requires fresh inspection. This maintenance operation is for a full-control
|
|
instance operator, not an agent tool.
|
|
|
|
```sql
|
|
BEGIN;
|
|
SELECT pg_advisory_xact_lock(hashtextextended(
|
|
'native-workspace-finalization:<company-id>:<run-id>', 0));
|
|
UPDATE heartbeat_runs
|
|
SET runner_profile_json = runner_profile_json - 'nativeWorkspaceFinalizationOwner'
|
|
WHERE company_id = '<company-id>'::uuid
|
|
AND id = '<run-id>'::uuid
|
|
AND native_issue_id = '<issue-id>'::uuid
|
|
AND runtime_mode = 'native'
|
|
AND runner_profile_json->'nativeWorkspaceFinalizationOwner'->>'token' = '<owner-token>'
|
|
RETURNING id;
|
|
COMMIT;
|
|
```
|
|
|
|
Resolve the existing board recovery action with a note containing the platform
|
|
stop evidence. The ordinary reconciliation sweep then resumes workspace
|
|
finalization from the accepted result. Confirm the run's native phase and
|
|
`resultJson.finalizationPhase` are `committed`, there is no `nextAttemptAt`, and
|
|
no workspace operation is still running. The accepted provider result is reused.
|
|
|
|
## Automatic unsafe archive recovery
|
|
|
|
An unsafe link in a native workspace must not fail a completed task or require
|
|
an operator to repair a sandbox. The accepted agent result remains authoritative.
|
|
|
|
Daytona validates every exported archive before extraction. If validation rejects
|
|
an archive, export once more using files, directories, and relative symlinks
|
|
whose resolved targets stay inside the workspace. The fallback does not follow
|
|
or delete symlinks. It omits unsafe or unresolvable links, stores hard-linked
|
|
files as ordinary bytes, and preserves directory exclusions and empty directories. The second archive passes the same confinement checks.
|
|
A fixed informational message records the fallback in the provider log.
|
|
|
|
If native copyback still rejects the archive or its source confinement check,
|
|
Paperclip discards that export, records `workspace_export_omitted` at info level
|
|
in the local run log, and completes finalization using the original accepted
|
|
result. It does not create a task warning, recovery card, repair request, or new
|
|
provider turn. The normal completion policy still enforces ownership and explicit
|
|
workflow constraints. Lost remote files are an accepted tradeoff.
|
|
|
|
The live path and restart finalizer use the same policy under workspace
|
|
finalization ownership. Ownership loss, transport failures, missing sandboxes,
|
|
and other unrelated errors retain their existing handling. No unsafe archive is
|
|
extracted. No host path or link target is copied into the informational run event.
|
|
|
|
|
|
## Repairing a failed workspace export without rerunning the agent
|
|
|
|
Transient export failures retry three times, then produce
|
|
`native_workspace_sync_out_retry_exhausted`. The accepted result stays saved,
|
|
and the exact sandbox is stopped and retained for export-only recovery. The
|
|
reason identifies a transport/copyback failure, not an inferred disk-space cause.
|
|
New unsafe archives use the automatic policy above and do not create this hold.
|
|
Historical unsafe-archive holds recover automatically as described below; they do not use this operator flow.
|
|
|
|
A board member with runtime management access can complete the saved result:
|
|
|
|
1. Read the recovery action, run, and environment lease. Verify the company, run,
|
|
provider sandbox ID, and stopped-provider receipt. Preserve that exact sandbox;
|
|
do not acquire a replacement or seed the host workspace over it.
|
|
2. Through the provider console or official SDK, resume that sandbox and inspect
|
|
the export failure while preserving all user files. Restore transport access
|
|
or other provider prerequisites before retrying. Extending the sandbox
|
|
retention window can be necessary during this repair.
|
|
3. Use the operator API to record the repair and request export-only retry. The
|
|
task shows operator assistance is needed, without an infrastructure form. The board API is
|
|
`POST /api/issues/:id/recovery-actions/retry-workspace-export` with
|
|
`{ "actionId": "<recovery-action-id>", "runId": "<run-id>", "repairNote": "<repair and preservation evidence>" }`.
|
|
4. Confirm the same run commits, the saved result determines the task outcome,
|
|
and the recovery action resolves. No new provider turn or wake is created.
|
|
A reusable repair sandbox is stopped and retained again. An ephemeral sandbox follows its original destroy-after-turn policy only after the exact result commits and the workspace reference contains a finalized host-copy receipt.
|
|
|
|
Admission requires the same accepted result, task owner, descriptor, and lease;
|
|
confirmed prior provider stop; an available repaired sandbox; and no newer task
|
|
execution or competing current lease. It shares finalization ownership and
|
|
resumes only the recorded provider lease through its verified lifecycle method.
|
|
That method drains old activity and verifies the saved workspace identity before
|
|
reopening the provider's controller admission gate; an external console restart
|
|
alone does not reopen that gate. Admission rejects a missing or replacement
|
|
sandbox and revalidates after probing the exact workspace. A changed binding or unavailable sandbox
|
|
returns `409` without reopening work. A duplicate queued request is idempotent.
|
|
Unsafe archives use automatic salvage or omission during retry. Generic
|
|
ordinary recovery resolution cannot retry, mark done, or send the task for review
|
|
in place of export-only retry. Explicit board false-positive/cancellation
|
|
dispositions remain deliberate overrides; they do not claim successful copyback.
|
|
|
|
Before publishing terminal export failure, the finalizer atomically records a stop-only intent with the failed run. This includes ephemeral allocations: ordinary release policy cannot delete their unexported files. A crash before cleanup leaves enough exact lease, result, and plugin authority for restart recovery. The same intent remains while export-only retry is active.
|
|
|
|
Before resuming the retained sandbox, the controller durably records a stop-only
|
|
cleanup intent on that exact lease and removes the old stopped receipt. If the
|
|
resume reply is lost, probing fails, or admission changes, it stops and retains
|
|
the sandbox. A failed stop remains `pending_cleanup`; a restarted controller's
|
|
bounded cleanup sweep retries the verified stop without destroying saved files.
|
|
An in-flight request holds a 15-minute cleanup claim, so a controller crash may
|
|
delay that sweep until the claim expires. Neither an unconfirmed stop nor a
|
|
stale receipt grants admission, and a changed lease or competing sandbox owner
|
|
prevents cleanup from taking ownership. Retry export after the lease has a new
|
|
confirmed stopped receipt. No provider turn is created by this recovery.
|
|
Cleanup and its readiness probe use the plugin ID recorded on that lease. Another
|
|
plugin with the same provider name cannot take over; an unavailable original
|
|
plugin defers cleanup without consuming an attempt. Stop-only cleanup requires the exact plugin worker to advertise `environmentStopLease`. Older plugins without that hook receive neither release nor destroy; resume admission also defers until safe compensation is available.
|
|
New intents use schema v2 with an explicit plugin pin. A v1 intent created before
|
|
that field existed remains recoverable using only the plugin ID already recorded
|
|
on its exact lease; an explicit mismatched pin is still rejected.
|
|
|
|
The opt-in `native-workspace-export-resume.live.test.ts` is a provider-boundary
|
|
fault integration, separate from the browser Product E2E. After building the
|
|
Daytona plugin, run that exact Vitest file with `PAPERCLIP_LIVE_EXPORT_RESUME=1`,
|
|
`DAYTONA_API_KEY`, and `PAPERCLIP_LIVE_EXPORT_RESUME_IMAGE` set to an immutable
|
|
image digest. It creates one disposable ephemeral sandbox and database, injects probe and stop transport failures, and verifies a fresh runtime can stop the sandbox while preserving exact nonce bytes. It also injects three transient failures at the production finalizer boundary and verifies retained work plus export-only retry admission. This boundary test does not claim physical copyback or result commitment; historical browser Product E2E supplied that proof before automatic unsafe-export recovery. It deletes only that owned fixture after proof.
|
|
|
|
### Historical unsafe exports
|
|
|
|
The reconciliation sweep automatically recovers accepted results that an older
|
|
controller terminalized with `native_workspace_sync_out_unsafe_archive`. It
|
|
omits the already-rejected export without accessing the old provider, so a
|
|
stopped, unavailable, or deleted sandbox cannot require user repair. The
|
|
omission barrier and result re-admission are atomic under finalization ownership.
|
|
The normal status arbiter still enforces current ownership, contracts, and gates.
|
|
|
|
This works with active, incorrectly resolved, or missing old repair actions. It
|
|
clears the matching stale unsafe notices instead of restoring them. Newer runs,
|
|
changed task ownership or contracts, another active recovery action, and explicit
|
|
operator dispositions prevent replay of old results. No provider turn is created.
|
|
Diagnostics appear only in the local run log; no unsafe-export activity notice,
|
|
repair card, or recovery chip is created. Existing stop-only allocation intents
|
|
retain their exact ownership and cleanup protections.
|
|
|
|
Daytona stop-only preservation disables provider auto-delete and refreshes the provider record to confirm the disabled policy before stopping. An unavailable or unconfirmed policy leaves cleanup pending; it never falls back to stop or delete. The original ephemeral destroy policy applies only after exact accepted-result copyback and commitment.
|
|
|
|
New ephemeral native allocations receive an acquisition-time workspace sentinel bound to their run and provider allocation. Export-only resume requires that proof and the unchanged durable intent. A legacy ephemeral allocation without this proof remains blocked; recovery never creates a sentinel or accepts a replacement during resume. The reusable workspace identity hash is unchanged.
|
|
|
|
Disabling auto-delete retains provider storage while repair waits. No automatic lease-age sweep deletes these released, confirmed-stopped allocations. Storage costs can continue until committed copyback applies the original cleanup policy or an operator explicitly deletes the allocation.
|
|
|
|
### Generic stop-only cleanup
|
|
|
|
An explicit sandbox `stop_and_retain` also requires the dedicated provider hook
|
|
when no export repair is involved, including successful per-turn reusable runs
|
|
and startup cancellation. Before dispatch, the controller stores a separate
|
|
`sandboxStopAndRetain` intent bound to the company, run, lease, provider allocation,
|
|
and original plugin. The initial cleanup and restart sweep both require that
|
|
plugin to advertise `environmentStopLease`. Missing capability or failed stop
|
|
leaves pending cleanup; ordinary release and destroy are never fallback methods.
|
|
|
|
A matching stopped receipt clears the pending intent, leaves the lease released
|
|
and resumable, and saves `sandboxStopAndRetainReceipt`. Its request identity,
|
|
original plugin and `method: "environmentStopLease"` attest this dispatch route.
|
|
The accepted-result export intent and its commitment requirements are unchanged.
|
|
|
|
The built-in fake provider uses its explicit `stopLease` operation rather than a
|
|
plugin RPC. Its intent pins the registered built-in provider and its receipt says
|
|
`builtin.stopLease`. This provider owns no real process or filesystem; the receipt
|
|
models its lifecycle. Missing built-in stop support or an unconfirmed receipt
|
|
also remains pending through restart, without calling release or destroy.
|
|
|
|
### Legacy copy-back after a confirmed stop
|
|
|
|
A legacy adapter can report failed workspace copy-back after cancellation has
|
|
already stopped and released its sandbox lease. The host records the exact lease
|
|
snapshot from before adapter dispatch. For an originally ephemeral source, the
|
|
recorder can preserve the released allocation for repair only while the current
|
|
lease is still ephemeral and its company, run, issue, workspace, provider,
|
|
allocation, acquisition time, plugin, and realized root still match. It requires
|
|
both the dedicated stop-and-retain receipt and the matching stopped-execution
|
|
receipt. A competing owner or changed source fails closed. The recorder leaves
|
|
release status and receipts intact, excludes the allocation from reuse, and
|
|
creates the existing board-owned file-repair obligation without replaying work
|
|
or changing a newer run or the task's status.
|
|
|
|
This late-recording path does not admit reusable leases. Provider resume can
|
|
precede the database handoff, so an old stop receipt and a database row lock
|
|
cannot prove that reuse has not started. Existing active-source and previously
|
|
recorded recovery paths retain their behavior. A missing or rejected original
|
|
snapshot does not authorize adopting another released lease.
|
|
|
|
A retained source receipt records control-plane evidence; it does not prove the
|
|
provider still holds the original bytes or that copy-back succeeded. Operators
|
|
must verify and recover the exact source separately. Later successful runs and
|
|
completed tasks do not certify recovery of earlier workspace files.
|
|
|
|
### Quiet task recovery and provider verification
|
|
|
|
The task conversation does not ask its user to certify recovered infrastructure
|
|
files. A source-bound recovery with a live native run or an eligible future
|
|
bounded retry uses the normal working presentation. A missed retry, exhausted
|
|
budget, unknown action outcome, or unresolved file hold remains visible. The
|
|
notice gives a brief explanation and a run-inspection link; stored diagnostics
|
|
and operator guidance remain under Recovery details. Source-only repair remains
|
|
available through the board-authorized recovery resolution API and its original
|
|
stop-receipt and evidence checks.
|
|
|
|
After a remote provider pack passes full integrity verification and the session
|
|
passes authenticated admission, the controller saves its manifest and runner
|
|
binary identity in `provider-pack-authority.json`. The receipt binds the company,
|
|
agent, workspace, session, runner instance, environment lease, remote working
|
|
directory, and provider descriptor. It stays in private controller state, outside
|
|
guest workspace and harness backup directories. Projectless runs use their
|
|
original environment lease as the workspace identity because each run otherwise
|
|
has a new workspace placeholder. Real workspace identities still require an exact
|
|
match. A receipt write failure warns and returns the admitted session to its
|
|
normal owner; it does not discard the live session handle. Future recovery still
|
|
fails closed without matching authority.
|
|
|
|
Exact process reattachment uses that saved release authority. It still verifies
|
|
the full remote pack, runner protocol and capabilities, exact process identity,
|
|
and authenticated PRP handshake. A new controller release alone does not require
|
|
a live provider to change its executable files. Fresh launches and explicit
|
|
session replacement select current release artifacts. Invalid or mismatched
|
|
receipts fail closed. Older sessions without a receipt retain the strict current
|
|
release gate; a sandbox's own manifest cannot supply missing launch authority.
|
|
|
|
Remote provider-pack verification retries an interrupted read at most once.
|
|
Each command retains its 30-second transport timeout. A timeout, signal, absent
|
|
exit status, or transport exception does not prove artifact incompatibility.
|
|
Two inconclusive attempts hold the original run with
|
|
`runner_remote_provider_verification_unavailable`. Reattachment does not spend
|
|
a new provider attempt, so it cannot renew this exhausted read budget. A
|
|
completed nonzero verification, digest mismatch, or
|
|
wrong OpenCode version remains incompatible and cannot authorize a new provider
|
|
turn. Recovery retains the exact source workspace and existing ownership gates;
|
|
these read-only retries do not replay tools or alter a retained pack. Verification
|
|
errors include bounded, redacted output from both transport streams, including
|
|
providers that combine stderr into stdout.
|
|
|
|
### Continue after confirmed source destruction
|
|
|
|
When an old export failed and its exact sandbox has a confirmed destruction
|
|
receipt, the task can continue from surviving attachments. This is reconstruction,
|
|
not recovery of the missing edits. The task's recovery notice exposes **Continue
|
|
from saved sources** to board operators with `runtime:manage`. Select the surviving
|
|
attachments, describe the missing work, and record the observed external action
|
|
outcomes. A retained or unverified sandbox must use the existing export or file
|
|
recovery path instead.
|
|
The source task may still display `in_progress` after an older controller's
|
|
run-only Stop. Reconstruction accepts that inactive state only with no execution
|
|
or checkout owner and the same source, recovery and status-version checks; an
|
|
operator does not need to manufacture a separate blocker merely to record recovery.
|
|
|
|
`POST /api/issues/{id}/recovery-actions/reconstruct-workspace` records this choice.
|
|
It validates the source run, accepted result, prepared descriptor, exact destroyed
|
|
lease, current task owner and status version. It rejects active copyback owners,
|
|
competing recovery, unreconciled newer executions, and attachments from another task or company.
|
|
The transaction records attachment hashes and a durable fresh-session delivery
|
|
intent. The old run, result, descriptor and failed finalization remain intact.
|
|
Repeated submissions adopt the original intent. Normal admission rechecks source
|
|
identity, attachments, authority, budgets, holds and task eligibility. It preserves
|
|
the retry count and does not grant a new work window or replay completed actions.
|
|
The agent must verify the selected attachments before rebuilding files.
|
|
Queued successors recheck their exact reconstruction receipt, sources and current
|
|
execution gates when claimed and immediately before provider handoff. A later
|
|
human block cannot be overwritten by automatic checkout.
|
|
Bounded retries keep the same reconstruction authority. Dispatch verifies the
|
|
system wake and predecessor chain back to the approved first run. It rechecks
|
|
the saved sources, remaining retry limits and current recovery obligations.
|
|
Unrelated executions and uncertain predecessor outcomes still stop dispatch.
|
|
|
|
A later native run-only cancellation may coexist with the older loss decision
|
|
only after its exact Stop acknowledgement, stopped lease receipts, and board
|
|
file-only repair have been recorded. Its external-action outcome must be
|
|
`not_performed`; an unknown or performed action stays held. Reconstruction saves
|
|
the identities and hashes of those repair decisions. Queue claim and dispatch
|
|
recheck the same evidence, so a changed receipt or another later run requires a
|
|
fresh decision. The cancelled run stays cancelled and is never replayed.
|
|
|
|
Stop also preserves a native workspace prepared before `paperclip_finish`.
|
|
The executor pins its files before releasing the environment. Startup and periodic
|
|
recovery process a bounded batch of historical acknowledged cancellations that
|
|
missed this step. The existing stop-only cleanup worker retains their exact
|
|
sandboxes and records provider receipts; this does not reopen a task or start
|
|
another turn. The normal file-only repair control remains required before an
|
|
older lost-work reconstruction can proceed.
|
|
If file retention loses the task lock, teardown keeps the source active for a
|
|
later pass. Historical sources already stopping or stopped remain eligible under
|
|
the existing exact-source checks. A stopped source requires the independent
|
|
stop-and-retain receipts and must not have a competing allocation owner.
|
|
Active warm allocations keep the same retention retry path. Only historical
|
|
stopping or stopped sources require the ephemeral policy, because a reusable
|
|
released allocation may already have a new owner.
|
|
Daytona stop-only cleanup also accepts a freshly verified archived allocation.
|
|
It first confirms that provider auto-delete is disabled. It leaves the archive
|
|
cold, with its files preserved, and records the process as stopped.
|
|
For a cancelled native run, the file-recovery form also offers an optional
|
|
confirmation that no external actions were performed. Leave it unchecked unless
|
|
the run history verifies that fact. The default records files without certifying
|
|
action outcomes. Changing the selected source clears the evidence and confirmations.
|
|
|
|
Committed chat response authorization reads current run identity, completion
|
|
binding, and chat source fields under the existing nonblocking share lock.
|
|
It excludes saved execution input, runner profiles, and log excerpts. Actual
|
|
retry admission still reads the complete run and its recovery evidence. These
|
|
reads preserve stored run data, source revocation checks, and JSON value types.
|
|
External review-wait discovery selects only chat authorization context fields.
|
|
It keeps the current endpoint, membership, resource, and question-receipt checks;
|
|
saved prompts remain stored and do not cross this discovery query.
|
|
|
|
Legacy continuation recovery rejects native runs before it loads their saved
|
|
execution input. The persisted disposition check reads only the task binding
|
|
for a native run. It keeps the same task-binding and native-finalization skip
|
|
reasons, including the original nonblank binding values. Native input and history
|
|
stay stored. Real legacy recovery still reads
|
|
its complete context, profile, excerpts, and required failure evidence.
|
|
|
|
The native Stop continuation recheck reads current run identity, the full Stop
|
|
acknowledgement receipt, and process identity. It does not load saved execution
|
|
input, runner profiles, excerpts, or unrelated result fields. Lease cleanup,
|
|
remote termination receipts, local process checks, task ownership, and admission
|
|
checks keep their existing behavior. Stored runs retain all of their data.
|
|
|
|
One periodic stranded-message pass coalesces successful native release checks
|
|
for the same company, task, source run, and current source row version. The release
|
|
path already drains that
|
|
task's queue through current admission checks. Each saved request still advances
|
|
its recovery cursor. Rejected attempts, changed source runs, and updated source evidence remain eligible,
|
|
and the next pass checks current state again. Legacy per-request handling stays
|
|
unchanged. Pending work is awaited before a source is marked checked.
|