Files
PaperClipAI/server/src/services/remote-workspace-lifecycle.ts
T
DottaandPaperclip 367f29478d fix(recovery): preserve the bound remote writer before continuation (#18803)
## Thinking Path

> - Paperclip manages agent work and preserves task files.
> - A remote workspace records the exact run that owns its unsaved
files.
> - A later setup failure can become the latest task run without owning
those files.
> - Retry only inspected that latest run, so the original writer had no
file-recovery owner.
> - A released reusable source also failed the legacy retention rule.
> - This change registers the bound writer, saves its files, and then
admits the saved user request once.
> - Existing stop, ownership, approval and execution-evidence checks
remain effective.

## Linked Issues or Issue Description

**What happened?**

A native run prepared a remote workspace and failed before finalization.
Its sandbox stopped with a dedicated stop receipt. The workspace stayed
dirty and retained that run as its writer. A later setup failure became
the latest task run. Retry inspected the newer run and left the original
file obligation unregistered. New attempts failed at the writer fence.

Registration alone was insufficient. After file recovery, continuation
still required the Retry source to be one of the older recovery sources.
A newer pre-provider failure did not satisfy that check. Workspace
selection could also choose a fresh directory instead of the recovered
files.

**Expected behavior**

Find the writer through the task's current workspace binding. Register
only its exact prepared source. Preserve the released lease and its stop
receipts. Save the files before continuation. Independently prove that a
newer setup failure never dispatched work. Select the verified saved
workspace and admit the Retry once.

**Steps to reproduce**

1. Fail a native run with a prepared remote workspace and a released
reusable sandbox.
2. Keep matching dedicated stop receipts and the workspace's dirty
writer authority.
3. Record a newer pre-provider setup failure for the same task.
4. Request Retry. Before the fix, no file-maintenance record owns the
actual writer.

**Paperclip version or commit**

Reproduced before production edits on master `f7ad1dc7ca`. The branch
was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`.

**Deployment mode**

Authenticated hosted execution with Paperclip Runner and remote
workspace storage.

Companion: #18798 (now merged upstream) fixes a pre-existing inbox test
synchronization race. Its test-only commit
`110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with
attribution so this PR can run the complete CI matrix.

Related: #17800 registers missing preservation for the selected source
run. #18678 holds unfinished remote exports. #18718 repairs a prior
warm-controller projection failure. #17916 addresses a separate legacy
warm-copyback policy case.

## What Changed

- Discover the failed writer through the company-scoped task/workspace
binding during Retry and authenticated message admission.
- Reuse the exact prepared-source validator for released remote
workspaces. Registration changes neither the lease policy nor its stop
receipts. Physical recovery retains its full allocation checks.
- Check a newer failed preparation separately before allowing an older,
repaired source to release continuation. Provider events, leases,
results and execution ownership remain disqualifying evidence.
- Select the bound saved workspace through the continuation's verified
repair receipt.
- Add PostgreSQL safety cases and an HTTP Retry workflow with real file
capture and saved-copy reads. Document the execution contract.
- Reuse #18798's completion barrier for the existing inbox cancellation
test; preserve its eight-slot limit, rejection and coalescing
assertions.

## Verification

- RED before production edits: the exact stopped writer has no
`workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass
on the baseline.
- GREEN: the expanded Retry/message/preview selection passes 17 cases.
All 627 tests in the five affected recovery suites pass on recovery
commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop,
ownership, saved messages and duplicate Retry.
- The HTTP workflow registers maintenance, keeps execution held before
save, reads back the saved file bytes, and admits exactly one
continuation with `resume_preserved_workspace`. A registration-only
repair first failed after save because it did not reconcile the newer
Retry source. The incident-shaped pair passes with an independent active
stranded-task action present.
- Greptile identified the path where the newer setup failure already
owns a reconciliation hold. Its new regression failed before the
correction (saved files, zero queued continuations); the
contradictory-dispatch control passed. All four HTTP variants pass after
the correction, covering both hold shapes and both dispatch-evidence
controls.
- Repository typecheck and build passed. Server typecheck and build also
passed after the recovery changes. Module-boundary and whitespace checks
pass.
- The full local Vitest command failed in the GitHub fast-response
acknowledgement test (`delivery_unknown`: one response write instead of
zero). That isolated case passes on this branch and unchanged master;
the original full-run cause remains unestablished. A sibling run passed
199 test cases but timed out in workspace-suite database cleanup.
Unchanged master also passed all 126 workspace cases and timed out in
the same cleanup hook; all five acknowledgement outcomes passed in that
baseline comparison. These local failures remain recorded rather than
being reported as suite passes.
- The first hosted matrix failed the pre-existing inbox cancellation
assertion (expected nine computes, saw eight). The same assertion failed
in [master
CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840)
and reproduced locally before applying #18798's existing repair. All six
inbox HTTP/database cases pass after that repair. Final head
`7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed
test change to the verified recovery code.
- [Final hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377)
passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci
/ verify`, `ci / e2e`, typecheck, build, every server/chat/workspace
group, Runner checks and all eight browser shards. There are 54
successful check results, two informational neutral results and three
skipped results, with no pending or failing results.
- Greptile and the Paperclip reviewer both report 5/5 for that final
head; no review threads remain open. The Paperclip review check is
informational (`neutral`) and records its local-testing limitations.
- The file workflow uses real PostgreSQL, subprocess file capture,
encrypted saved objects and production recovery code. Provider
resume/stop calls are simulated at the environment driver boundary. This
does not prove a live Daytona or Codex recovery. No deployment or
original-task retry is claimed.

## Risks

- File-maintenance discovery must not authorize another executor.
Existing writer, coordinator and physical allocation checks still own
that decision.
- A false bootstrap summary cannot prove that a newer run never
executed. The new path also requires its immutable native-adapter claim,
unresolved preparation state, absent
coordinator/process/session/lease/result evidence, and no dispatch
events.
- Additional database reads run only during explicit task admission or
Retry strategy selection. They are company/task scoped and add no
polling loop. The saved-writer query returns one matching snapshot and
omits prompt bodies.
- No schema, credential, provider profile or deployment setting changes.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution and test tools. The runtime does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:25:24 -06:00

83 lines
5.9 KiB
TypeScript

import { and, eq, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, executionWorkspaces, executionWorkspaceSnapshots, type Db } from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLease } from "@paperclipai/shared";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
export type DestroyRemoteWorkspaceLease = (input: {
companyId: string; workspaceId: string; leaseId: string; environmentId: string; providerLeaseId: string;
}) => Promise<EnvironmentLease | null>;
export type WorkspaceSourceLease = Pick<typeof environmentLeases.$inferSelect,
"id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId">;
export async function preparedRemoteWorkspaceSource(db: Db, lease: WorkspaceSourceLease) {
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" || !lease.providerLeaseId) return null;
const [bound] = await db.select({ lease: environmentLeases, run: {
id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode,
controllerBootId: heartbeatRuns.controllerBootId,
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
${heartbeatRuns.contextSnapshot}->>'issueId')`,
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`,
legacyReference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
} }).from(environmentLeases)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
eq(heartbeatRuns.companyId, environmentLeases.companyId)))
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId), eq(environmentLeases.provider, lease.provider),
eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared'`)).limit(1);
if (!bound) return null;
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(bound.run.reference);
if (!parsed.success || parsed.data.state !== "prepared" || bound.run.legacyReference != null) return null;
const reference = parsed.data;
const issueId = bound.run.issueId;
if (typeof issueId !== "string" || bound.lease.issueId !== issueId || reference.leaseId !== bound.lease.id
|| reference.providerLeaseId !== bound.lease.providerLeaseId || reference.remoteCwd !== bound.lease.metadata?.remoteCwd
|| bound.lease.executionWorkspaceId && bound.lease.executionWorkspaceId !== reference.workspaceId) return null;
const [workspace] = await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, lease.companyId), eq(executionWorkspaces.id, reference.workspaceId))).limit(1);
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
if (!workspace || workspace.sourceIssueId !== issueId && workspace.mode !== "shared_workspace"
|| !authority || !authority.dirty || authority.writerRunId !== bound.run.id || authority.epoch !== reference.epoch
|| authority.snapshotId !== reference.snapshotId || !authority.location
|| authority.location.environmentId !== bound.lease.environmentId || authority.location.leaseId !== bound.lease.id
|| authority.location.providerLeaseId !== bound.lease.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd) return null;
return { ...bound, workspace: workspace!, reference, issueId };
}
export async function inspectRemoteWorkspacePersistence(db: Db | Transaction, workspace: Workspace) {
const authority = readRemoteWorkspaceAuthority(workspace.metadata);
if (!authority || authority.dirty || authority.writerRunId || !authority.snapshotId) {
return { ready: false, reason: "The workspace still has unsaved work or an unfinished writer.", authority };
}
const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
eq(executionWorkspaceSnapshots.id, authority.snapshotId), eq(executionWorkspaceSnapshots.status, "committed"),
));
return { ready: Boolean(snapshot), reason: snapshot ? null : "The saved workspace is unavailable. Its remote files were retained.", authority };
}
/** Called within the workspace lifecycle fence; the saved workspace remains durable. */
export async function cleanupRemoteWorkspaceAllocation(input: {
db: Db; workspace: Workspace; destroyLease: DestroyRemoteWorkspaceLease;
}) {
const [workspace] = await input.db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, input.workspace.companyId), eq(executionWorkspaces.id, input.workspace.id),
));
if (!workspace || !["archived", "cleanup_failed"].includes(workspace.status)) throw new Error("remote_workspace_cleanup_binding_changed");
const persistence = await inspectRemoteWorkspacePersistence(input.db, workspace);
if (!persistence.ready || !persistence.authority) throw new Error(persistence.reason ?? "remote_workspace_saved_state_required");
const location = persistence.authority.location;
if (location) {
const released = await input.destroyLease({ companyId: workspace.companyId, workspaceId: workspace.id,
leaseId: location.leaseId, environmentId: location.environmentId, providerLeaseId: location.providerLeaseId });
if (!released || !hasRemoteTerminationReceipt(released)
|| (released.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state !== "destroyed") {
throw new Error("remote_workspace_cleanup_unconfirmed");
}
}
return { cleaned: true, warnings: [] as string[] };
}