Files
PaperClipAI/server/src/services/remote-execution-termination.ts
T
DottaandClaude Opus 5.5 22d3f9a03c fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.

**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.

**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.

**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).

**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).

## What Changed

- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.

## Verification

- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.

## Risks

- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 02:25:02 -06:00

150 lines
8.3 KiB
TypeScript

import { and, desc, eq } from "drizzle-orm";
import { environmentLeases, type Db } from "@paperclipai/db";
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
type Lease = typeof environmentLeases.$inferSelect;
type LeaseIdentity = {
id: string; companyId: string; heartbeatRunId: string | null;
provider: string | null; providerLeaseId: string | null;
};
/** Bind a provider receipt to the exact host-owned lease and run. Old plugins
* return void; that remains supported but grants no continuation authority. */
export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
const receipt = value as { providerLeaseId?: unknown; state?: unknown } | null;
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" ||
!lease.providerLeaseId || receipt?.providerLeaseId !== lease.providerLeaseId ||
!["stopped", "destroyed"].includes(String(receipt?.state))) return undefined;
return {
schema: "paperclip.remote-termination.v1", companyId: lease.companyId,
runId: lease.heartbeatRunId, leaseId: lease.id, provider: lease.provider,
providerLeaseId: lease.providerLeaseId, state: receipt!.state,
confirmedAt: new Date().toISOString(),
};
}
type ReceiptLease = LeaseIdentity & {
releasedAt: unknown; cleanupStatus: string | null; status: string;
metadata: Record<string, unknown> | null;
};
/** The lease carries a provider stop receipt bound to its exact company, run,
* lease, provider and physical task. It proves execution there stopped, not
* that the lease's resources were released. */
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
remoteTerminationReceipt(lease, receipt));
}
/** The lease was released with successful cleanup and its bound stop receipt.
* Resource cleanup gates use this; it never accepts an unreleased lease. */
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
}
/** Release or destroy of this lease has started: plugin release and destroy
* record their cleanup method first, and task settlement, startup cleanup and
* interrupted-creation cleanup mark the task retiring. The provider can then be
* releasing the task's resources while the lease row is still active. */
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
}
/** Execution on this lease has stopped: it was released with its termination
* receipt, or it is still active, its release has not started, and its plugin
* task recorded the provider's stop when it settled. Release can be held long
* after that stop (for example by a pending workspace export); once release
* starts, it owns the receipt until the lease is released. */
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
return hasRemoteTerminationReceipt(lease) ||
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
}
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
* stop receipt it carried forward still proves no provider can run there. */
export function hasRecoveredCleanupStopReceipt(lease: LeaseIdentity & {
releasedAt: unknown; status: string; metadata: Record<string, unknown> | null;
}): boolean {
const cleanup = lease.metadata?.workspaceRecoveryCleanup as Record<string, unknown> | undefined;
if (!cleanup || cleanup.runId !== lease.heartbeatRunId || !lease.releasedAt ||
!["pending_cleanup", "released", "expired", "failed"].includes(lease.status)) return false;
return hasRemoteTerminationReceipt({ ...lease, status: "released", cleanupStatus: "success",
metadata: { remoteExecutionTermination: cleanup.stopReceipt } });
}
export function remoteLeaseCleanupScope(lease: Pick<LeaseIdentity, "provider" | "providerLeaseId">) {
return lease.provider && lease.provider !== "local" && lease.providerLeaseId
? JSON.stringify([lease.provider, lease.providerLeaseId]) : undefined;
}
export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promise<Lease> {
if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" ||
lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease;
const history = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
));
const current = history.find(row => row.id === lease.id);
if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease;
const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor"));
if (tips.length !== 1) return current;
const tip = tips[0]!;
const receipt = tip.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current;
const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id));
if (!predecessors?.some(row => row.id === current.id)) return current;
const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" });
if (!termination) return current;
return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: {
...termination, confirmedAt: receipt.confirmedAt,
} } };
}
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
));
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
}
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
}
/** Every lease of the run has stopped execution. Its resources can still be held. */
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
}
/** Every lease of the run was released with its termination receipt. */
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
const leases = await runRemoteLeases(db, companyId, runId);
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
}
export async function terminatedRemoteCleanupScope(db: Db, companyId: string, remoteCleanupScope: string) {
let parsed: unknown;
try {
parsed = JSON.parse(remoteCleanupScope);
} catch {
return null;
}
if (!Array.isArray(parsed) || parsed.length !== 2 || typeof parsed[0] !== "string" || typeof parsed[1] !== "string"
|| parsed[0] === "local") return null;
const [provider, providerLeaseId] = parsed as [string, string];
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
eq(environmentLeases.provider, provider), eq(environmentLeases.providerLeaseId, providerLeaseId)))
.orderBy(desc(environmentLeases.createdAt), desc(environmentLeases.id));
if (leases.length === 0 || leases.some(lease => !lease.releasedAt)) return null;
const newest = await resolveRemoteTerminationLease(db, leases[0]!);
return hasRemoteTerminationReceipt(newest) ? newest : null;
}