mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.
**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.
**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.
**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).
**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).
## What Changed
- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.
## Verification
- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.
## Risks
- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
150 lines
8.3 KiB
TypeScript
150 lines
8.3 KiB
TypeScript
import { and, desc, eq } from "drizzle-orm";
|
|
import { environmentLeases, type Db } from "@paperclipai/db";
|
|
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
|
|
|
|
type Lease = typeof environmentLeases.$inferSelect;
|
|
|
|
type LeaseIdentity = {
|
|
id: string; companyId: string; heartbeatRunId: string | null;
|
|
provider: string | null; providerLeaseId: string | null;
|
|
};
|
|
|
|
/** Bind a provider receipt to the exact host-owned lease and run. Old plugins
|
|
* return void; that remains supported but grants no continuation authority. */
|
|
export function remoteTerminationReceipt(lease: LeaseIdentity, value: unknown) {
|
|
const receipt = value as { providerLeaseId?: unknown; state?: unknown } | null;
|
|
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" ||
|
|
!lease.providerLeaseId || receipt?.providerLeaseId !== lease.providerLeaseId ||
|
|
!["stopped", "destroyed"].includes(String(receipt?.state))) return undefined;
|
|
return {
|
|
schema: "paperclip.remote-termination.v1", companyId: lease.companyId,
|
|
runId: lease.heartbeatRunId, leaseId: lease.id, provider: lease.provider,
|
|
providerLeaseId: lease.providerLeaseId, state: receipt!.state,
|
|
confirmedAt: new Date().toISOString(),
|
|
};
|
|
}
|
|
|
|
type ReceiptLease = LeaseIdentity & {
|
|
releasedAt: unknown; cleanupStatus: string | null; status: string;
|
|
metadata: Record<string, unknown> | null;
|
|
};
|
|
|
|
/** The lease carries a provider stop receipt bound to its exact company, run,
|
|
* lease, provider and physical task. It proves execution there stopped, not
|
|
* that the lease's resources were released. */
|
|
export function hasBoundRemoteStopReceipt(lease: Pick<ReceiptLease, keyof LeaseIdentity | "metadata">): boolean {
|
|
const receipt = lease.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
|
|
return Boolean(receipt && receipt.schema === "paperclip.remote-termination.v1" &&
|
|
receipt.companyId === lease.companyId && receipt.runId === lease.heartbeatRunId &&
|
|
receipt.leaseId === lease.id && receipt.provider === lease.provider &&
|
|
remoteTerminationReceipt(lease, receipt));
|
|
}
|
|
|
|
/** The lease was released with successful cleanup and its bound stop receipt.
|
|
* Resource cleanup gates use this; it never accepts an unreleased lease. */
|
|
export function hasRemoteTerminationReceipt(lease: ReceiptLease): boolean {
|
|
return Boolean(lease.releasedAt && lease.cleanupStatus === "success" &&
|
|
["released", "expired", "failed"].includes(lease.status) && hasBoundRemoteStopReceipt(lease));
|
|
}
|
|
|
|
/** Release or destroy of this lease has started: plugin release and destroy
|
|
* record their cleanup method first, and task settlement, startup cleanup and
|
|
* interrupted-creation cleanup mark the task retiring. The provider can then be
|
|
* releasing the task's resources while the lease row is still active. */
|
|
function leaseReleaseStarted(lease: Pick<ReceiptLease, "metadata">): boolean {
|
|
return lease.metadata?.pluginLeaseCleanupMethod !== undefined || lease.metadata?.runnerTaskRetiring === true;
|
|
}
|
|
|
|
/** Execution on this lease has stopped: it was released with its termination
|
|
* receipt, or it is still active, its release has not started, and its plugin
|
|
* task recorded the provider's stop when it settled. Release can be held long
|
|
* after that stop (for example by a pending workspace export); once release
|
|
* starts, it owns the receipt until the lease is released. */
|
|
function remoteLeaseExecutionStopped(lease: ReceiptLease): boolean {
|
|
return hasRemoteTerminationReceipt(lease) ||
|
|
lease.status === "active" && !lease.releasedAt && !leaseReleaseStarted(lease) && hasBoundRemoteStopReceipt(lease);
|
|
}
|
|
|
|
/** Workspace recovery saved this run's files, re-stopped its sandbox, and then
|
|
* queued the stopped sandbox for destroy. The destroy is resource cleanup: the
|
|
* stop receipt it carried forward still proves no provider can run there. */
|
|
export function hasRecoveredCleanupStopReceipt(lease: LeaseIdentity & {
|
|
releasedAt: unknown; status: string; metadata: Record<string, unknown> | null;
|
|
}): boolean {
|
|
const cleanup = lease.metadata?.workspaceRecoveryCleanup as Record<string, unknown> | undefined;
|
|
if (!cleanup || cleanup.runId !== lease.heartbeatRunId || !lease.releasedAt ||
|
|
!["pending_cleanup", "released", "expired", "failed"].includes(lease.status)) return false;
|
|
return hasRemoteTerminationReceipt({ ...lease, status: "released", cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: cleanup.stopReceipt } });
|
|
}
|
|
|
|
export function remoteLeaseCleanupScope(lease: Pick<LeaseIdentity, "provider" | "providerLeaseId">) {
|
|
return lease.provider && lease.provider !== "local" && lease.providerLeaseId
|
|
? JSON.stringify([lease.provider, lease.providerLeaseId]) : undefined;
|
|
}
|
|
|
|
export async function resolveRemoteTerminationLease(db: Db, lease: Lease): Promise<Lease> {
|
|
if (hasRemoteTerminationReceipt(lease) || lease.status !== "expired" ||
|
|
lease.leasePolicy !== "reuse_by_environment" || !lease.provider || !lease.providerLeaseId) return lease;
|
|
const history = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.provider, lease.provider), eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
|
|
));
|
|
const current = history.find(row => row.id === lease.id);
|
|
if (!current || current.companyId !== lease.companyId || current.heartbeatRunId !== lease.heartbeatRunId) return lease;
|
|
const tips = history.filter(row => !Object.hasOwn(row.metadata ?? {}, "reusableLeaseSuccessor"));
|
|
if (tips.length !== 1) return current;
|
|
const tip = tips[0]!;
|
|
const receipt = tip.metadata?.remoteExecutionTermination as Record<string, unknown> | undefined;
|
|
if (!hasRemoteTerminationReceipt(tip) || receipt?.state !== "destroyed") return current;
|
|
const predecessors = retiredReusableLeasePredecessors(tip, history.filter(row => row.id !== tip.id));
|
|
if (!predecessors?.some(row => row.id === current.id)) return current;
|
|
const termination = remoteTerminationReceipt(current, { providerLeaseId: current.providerLeaseId, state: "destroyed" });
|
|
if (!termination) return current;
|
|
return { ...current, metadata: { ...current.metadata, remoteExecutionTermination: {
|
|
...termination, confirmedAt: receipt.confirmedAt,
|
|
} } };
|
|
}
|
|
|
|
async function runRemoteLeases(db: Db, companyId: string, runId: string) {
|
|
const leases = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
|
));
|
|
return Promise.all(leases.map(lease => resolveRemoteTerminationLease(db, lease)));
|
|
}
|
|
|
|
/** Cleanup scopes of a run whose every lease has stopped execution, or null. */
|
|
export async function stoppedRemoteCleanupScopes(db: Db, companyId: string, runId: string) {
|
|
const leases = await runRemoteLeases(db, companyId, runId);
|
|
if (leases.length === 0 || !leases.every(remoteLeaseExecutionStopped)) return null;
|
|
return [...new Set(leases.map(lease => remoteLeaseCleanupScope(lease)!))];
|
|
}
|
|
|
|
/** Every lease of the run has stopped execution. Its resources can still be held. */
|
|
export async function remoteExecutionHasStopped(db: Db, companyId: string, runId: string) {
|
|
return await stoppedRemoteCleanupScopes(db, companyId, runId) !== null;
|
|
}
|
|
|
|
/** Every lease of the run was released with its termination receipt. */
|
|
export async function remoteExecutionHasReleased(db: Db, companyId: string, runId: string) {
|
|
const leases = await runRemoteLeases(db, companyId, runId);
|
|
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
|
|
}
|
|
|
|
export async function terminatedRemoteCleanupScope(db: Db, companyId: string, remoteCleanupScope: string) {
|
|
let parsed: unknown;
|
|
try {
|
|
parsed = JSON.parse(remoteCleanupScope);
|
|
} catch {
|
|
return null;
|
|
}
|
|
if (!Array.isArray(parsed) || parsed.length !== 2 || typeof parsed[0] !== "string" || typeof parsed[1] !== "string"
|
|
|| parsed[0] === "local") return null;
|
|
const [provider, providerLeaseId] = parsed as [string, string];
|
|
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
|
|
eq(environmentLeases.provider, provider), eq(environmentLeases.providerLeaseId, providerLeaseId)))
|
|
.orderBy(desc(environmentLeases.createdAt), desc(environmentLeases.id));
|
|
if (leases.length === 0 || leases.some(lease => !lease.releasedAt)) return null;
|
|
const newest = await resolveRemoteTerminationLease(db, leases[0]!);
|
|
return hasRemoteTerminationReceipt(newest) ? newest : null;
|
|
}
|