mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.
**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.
**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.
**Paperclip version or commit**
master at b2b2c1e60b (wave) and 7685b7a4fc (this branch base).
**Deployment mode**
Hosted server with remote Daytona sandboxes (restart-chaos harness).
## What Changed
- `native-session-executor.ts`: the remote checkpoint resolves its
backup owner from the current scope authority, and falls back to the
backend's own authority epoch when the scope has none. The
controller-persistence `authorityRunId` closure uses the same fallback.
A newer run that took over the session still wins.
- `paperclip-runner` `native-session-runtime.ts`: new
`nativeSessionCleanupRemoteScopes` lists the remote resources whose
quarantined cleanup holds a task. New
`completeTerminatedRemoteNativeSessionCleanupForTask` retires settled
quarantines of the task in one terminated remote resource, whichever
run's session they hold. An in-flight close keeps its entry.
- `remote-execution-termination.ts`: new `terminatedRemoteCleanupScope`
confirms that the company's newest lease on the remote resource is
released with its termination receipt and that no lease on it is still
held.
- `native-cleanup-admission-hold.ts`: queue admission now retires a held
cleanup when its sandbox is confirmed terminated. It logs the retirement
and records a `native.session_cleanup_gate_retired` run event with the
termination receipt. The local stopped-process path is unchanged.
- `native-safe-replacement.ts`: a transport failure on a remote or
non-Codex session can never produce the local stop proof. It is now
refused at once with `provider_failure_meaning_unverified`. The existing
disposition pass then continues the task in a fresh isolated run from
saved work. A local Codex run keeps the existing wait for stop proof.
## Verification
- Red/green, each verified red without its fix:
- `server/src/services/native-runtime/native-session-executor.test.ts` —
"saves a closing session's backup under its own owner after a failed
follow-up released the scope authority". Red:
`native_harness_storage_owner_missing` (the incident error).
- `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`
(embedded Postgres) — "starts a queued run once the previous session's
sandbox is confirmed terminated, whichever run released it", plus "keeps
the run queued while a different sandbox of the task is terminated".
Red: the queued run stays queued.
- `server/src/services/native-runtime/native-safe-replacement.test.ts`
(embedded Postgres) — "continues a transport failure whose stop can
never be proven locally without waiting for a person" for a remote
sandbox (refused and continued fresh) and a local Codex run (unchanged).
Red: no denial and no hold.
- `pnpm verify` and server and runner typechecks pass locally.
## Risks
- The admission gate now trusts a provider termination receipt on the
newest lease of the same remote resource, not only one bound to the
session's own run. A reused sandbox with a newer held lease is not
retired.
- A remote transport failure no longer waits for a stop proof that
cannot arrive. The fresh continuation is isolated (write authority
revoked, new root) and stays under the existing per-task and per-minute
continuation brakes. Nothing is added before `turn.start`.
- The run.attach timeout itself is not fixed here. The controller held
the attach frame behind a checkpoint flush to the follow-up's writer.
The run's own diagnostics were lost because they were written to the
predecessor's run log. This PR makes the failure recover by itself; the
attach stall needs a separate fix with better diagnostics.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7685b7a4fc
commit
22d3f9a03c
10 files changed
+341
-16
No files matched your search
@@ -275,6 +275,47 @@ export function completeTerminatedRemoteNativeSessionCleanup(binding: {
|
||||
return true;
|
||||
}
|
||||
|
||||
function quarantinedRemoteCleanupScope(
|
||||
entry: QuarantinedSessionCleanup,
|
||||
input: { companyId: string; issueId: string },
|
||||
): string | null {
|
||||
const [companyId, , , remoteCleanupScope] = JSON.parse(entry.domain.key) as string[];
|
||||
return companyId === input.companyId && remoteCleanupScope &&
|
||||
entry.domain.resources.includes(JSON.stringify(["issue", input.issueId]))
|
||||
? remoteCleanupScope : null;
|
||||
}
|
||||
|
||||
export function nativeSessionCleanupRemoteScopes(input: {
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
}): string[] {
|
||||
const scopes = new Set<string>();
|
||||
for (const entry of quarantinedSessionCleanups) {
|
||||
const scope = quarantinedRemoteCleanupScope(entry, input);
|
||||
if (scope) scopes.add(scope);
|
||||
}
|
||||
return [...scopes];
|
||||
}
|
||||
|
||||
export function completeTerminatedRemoteNativeSessionCleanupForTask(input: {
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
remoteCleanupScope: string;
|
||||
}): number {
|
||||
if (!input.remoteCleanupScope) return 0;
|
||||
let retired = 0;
|
||||
for (const entry of [...quarantinedSessionCleanups]) {
|
||||
if (quarantinedRemoteCleanupScope(entry, input) !== input.remoteCleanupScope) continue;
|
||||
if (entry.attempt || entry.recovery) continue;
|
||||
if (entry.timer) clearTimeout(entry.timer);
|
||||
entry.timer = null;
|
||||
quarantinedSessionCleanups.delete(entry);
|
||||
announceCleanupSettled(entry.domain);
|
||||
retired += 1;
|
||||
}
|
||||
return retired;
|
||||
}
|
||||
|
||||
/** Host-only counterpart of remote resource termination. The caller has verified
|
||||
* both exact local process identities and durably fenced their run. This removes
|
||||
* only cleanup ownership; the retained checkpoint is never made resumable.
|
||||
|
||||
@@ -5,7 +5,7 @@ import { join } from "node:path";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { hostname } from "node:os";
|
||||
import { agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { agents, companies, createDb, environmentLeases, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import {
|
||||
executeNativeSession,
|
||||
NativeSessionCloseUnrecoverableError,
|
||||
@@ -55,7 +55,8 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
|
||||
}
|
||||
|
||||
async function retainPreviousSessionClose(f: Awaited<ReturnType<typeof seed>>, close: () => Promise<void>,
|
||||
owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() }) {
|
||||
owner: { runId: string; runnerInstanceId: string } = { runId: randomUUID(), runnerInstanceId: randomUUID() },
|
||||
remoteCleanupScope?: string) {
|
||||
const runId = owner.runId;
|
||||
const identity = { runId, sessionId: `closing-${runId}`, companyId: f.companyId, issueId: f.issueId, agentId: f.previousId };
|
||||
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
|
||||
@@ -83,6 +84,7 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
|
||||
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
|
||||
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
|
||||
runnerInstanceId: owner.runnerInstanceId, controlPlaneInstanceId: "control",
|
||||
...(remoteCleanupScope ? { remoteCleanupScope } : {}),
|
||||
})).rejects.toThrow("stopped for reassignment");
|
||||
}
|
||||
|
||||
@@ -167,6 +169,63 @@ describeEmbeddedPostgres("queued native run behind the task's previous session c
|
||||
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null, errorCode: null });
|
||||
});
|
||||
|
||||
async function recordSandboxLease(f: Awaited<ReturnType<typeof seed>>, providerLeaseId: string, terminated: boolean) {
|
||||
const [failed] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.nextId,
|
||||
invocationSource: "automation", status: "failed", errorCode: "provider_transport_failed", runtimeMode: "native",
|
||||
nativeIssueId: f.issueId, startedAt: new Date(Date.now() - 60_000), finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const leaseId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: failed!.id, provider: "fake-sandbox", providerLeaseId, leasePolicy: "ephemeral",
|
||||
status: terminated ? "released" : "active", releasedAt: terminated ? new Date() : null,
|
||||
cleanupStatus: terminated ? "success" : null,
|
||||
metadata: terminated ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
|
||||
runId: failed!.id, leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped",
|
||||
confirmedAt: new Date().toISOString() } } : {} });
|
||||
return { leaseId, runId: failed!.id };
|
||||
}
|
||||
|
||||
it("starts a queued run once the previous session's sandbox is confirmed terminated, whichever run released it", async () => {
|
||||
const worker = dispatchWorker();
|
||||
try {
|
||||
const f = await seed();
|
||||
const providerLeaseId = `sandbox-${randomUUID()}`;
|
||||
await retainPreviousSessionClose(f, async () => {
|
||||
throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false });
|
||||
}, undefined, JSON.stringify(["fake-sandbox", providerLeaseId]));
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
|
||||
const held = await recordSandboxLease(f, providerLeaseId, false);
|
||||
const run = await queueNextAssigneeRun(f);
|
||||
await heartbeatService(db).resumeQueuedRuns();
|
||||
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
|
||||
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
|
||||
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
|
||||
runId: held.runId, leaseId: held.leaseId, provider: "fake-sandbox", providerLeaseId, state: "stopped",
|
||||
confirmedAt: new Date().toISOString() } } }).where(eq(environmentLeases.id, held.leaseId));
|
||||
await heartbeatService(db).resumeQueuedRuns();
|
||||
await expect.poll(async () => (await readRun(run.id)).status, { timeout: 15_000 }).not.toBe("queued");
|
||||
expect((await readRun(run.id)).errorCode).not.toBe("native_session_cleanup_quarantined");
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
|
||||
} finally {
|
||||
worker.stop();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the run queued while a different sandbox of the task is terminated", async () => {
|
||||
const f = await seed();
|
||||
const providerLeaseId = `sandbox-${randomUUID()}`;
|
||||
await retainPreviousSessionClose(f, async () => {
|
||||
throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: false });
|
||||
}, undefined, JSON.stringify(["fake-sandbox", providerLeaseId]));
|
||||
await recordSandboxLease(f, `other-${providerLeaseId}`, true);
|
||||
const run = await queueNextAssigneeRun(f);
|
||||
await heartbeatService(db).resumeQueuedRuns();
|
||||
expect(await readRun(run.id)).toMatchObject({ status: "queued", startedAt: null });
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required");
|
||||
});
|
||||
|
||||
it("does not hold a run for a different task", async () => {
|
||||
const f = await seed();
|
||||
await retainPreviousSessionClose(f, () => new Promise<void>(() => {}));
|
||||
|
||||
@@ -593,7 +593,9 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
|
||||
const isolation: FreshContinuationIsolation = fenced || remoteSource ? "remote_epoch"
|
||||
: writer === "ended" && !pastBound() && !refused ? "writer_ended" : "fresh_isolated_workspace";
|
||||
if (isolation === "fresh_isolated_workspace" && delivery === "saved_input") delivery = "recorded";
|
||||
const unverified = await unverifiedActions(tx, run);
|
||||
const unconfirmed = object(coordinator?.failureDetail).unconfirmedActions;
|
||||
const unverified = [...new Set([...await unverifiedActions(tx, run),
|
||||
...(Array.isArray(unconfirmed) ? unconfirmed.filter((name): name is string => typeof name === "string") : [])])].slice(0, 20);
|
||||
const created = await createdObjects(tx, run);
|
||||
const note = freshContinuationNote({ runId: run.id, unverified, created,
|
||||
workspace: isolation === "writer_ended" ? "kept" : "restored_last_save" });
|
||||
|
||||
@@ -1,11 +1,46 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { nativeSessionCleanupBlock } from "../../vendor/paperclip-runner/index.js";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
completeTerminatedRemoteNativeSessionCleanupForTask,
|
||||
nativeSessionCleanupBlock,
|
||||
nativeSessionCleanupRemoteScopes,
|
||||
} from "../../vendor/paperclip-runner/index.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { terminatedRemoteCleanupScope } from "../remote-execution-termination.js";
|
||||
|
||||
async function retireTerminatedRemoteCleanup(db: Db, input: { companyId: string; issueId: string }) {
|
||||
for (const remoteCleanupScope of nativeSessionCleanupRemoteScopes(input)) {
|
||||
const lease = await terminatedRemoteCleanupScope(db, input.companyId, remoteCleanupScope).catch(() => null);
|
||||
if (!lease) continue;
|
||||
const retired = completeTerminatedRemoteNativeSessionCleanupForTask({ ...input, remoteCleanupScope });
|
||||
if (retired === 0) continue;
|
||||
logger.warn({ companyId: input.companyId, issueId: input.issueId, leaseId: lease.id, runId: lease.heartbeatRunId,
|
||||
provider: lease.provider, providerLeaseId: lease.providerLeaseId, retired },
|
||||
"native session cleanup retired because its sandbox is confirmed terminated");
|
||||
if (!lease.heartbeatRunId) continue;
|
||||
const [run] = await db.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, lease.heartbeatRunId)))
|
||||
.limit(1).catch(() => []);
|
||||
if (!run) continue;
|
||||
await appendHeartbeatRunEvent(db, {
|
||||
companyId: input.companyId, runId: lease.heartbeatRunId, agentId: run.agentId,
|
||||
eventType: "native.session_cleanup_gate_retired", stream: "system", level: "info",
|
||||
message: "The task's previous sandbox is confirmed terminated, so its session cleanup gate was retired.",
|
||||
payload: { leaseId: lease.id, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
||||
termination: lease.metadata?.remoteExecutionTermination ?? null, retired },
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
export async function nativeCleanupHoldsAdmission(
|
||||
db: Db,
|
||||
input: { companyId: string; issueId: string },
|
||||
): Promise<"pending" | "operator_required" | null> {
|
||||
let block = nativeSessionCleanupBlock(input);
|
||||
if (block === null) return null;
|
||||
await retireTerminatedRemoteCleanup(db, input);
|
||||
block = nativeSessionCleanupBlock(input);
|
||||
if (block === "operator_required") {
|
||||
const { retireStoppedSessionCleanupGates } = await import("./native-session-executor.js");
|
||||
await retireStoppedSessionCleanupGates(db, input).catch(() => false);
|
||||
|
||||
@@ -226,6 +226,74 @@ const support = externalDatabaseUrl
|
||||
}
|
||||
},
|
||||
);
|
||||
it.each(["remote_sandbox", "remote_unconfirmed_action", "remote_completed_action", "remote_started_only_action",
|
||||
"remote_failed_exit_action", "local_codex", "unknown_provider"] as const)(
|
||||
"continues a transport failure whose stop can never be proven locally without waiting for a person (%s)",
|
||||
async (mode) => {
|
||||
const source = await seed();
|
||||
const workspace = await mkdtemp(join(tmpdir(), "paperclip-remote-transport-"));
|
||||
const remote = mode.startsWith("remote_");
|
||||
try {
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(), errorCode: "provider_transport_failed",
|
||||
runnerProfileJson: mode === "unknown_provider" ? {} : { nativeExecutionInput: {
|
||||
provider: { kind: remote ? "opencode" : "codex" }, workspace: { cwd: workspace } } } })
|
||||
.where(eq(heartbeatRuns.id, source.runId));
|
||||
await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed", failureDetail: {} })
|
||||
.where(eq(nativeRunFinalizations.runId, source.runId));
|
||||
if (remote) await db.insert(environmentLeases).values({ companyId: source.companyId,
|
||||
issueId: source.issueId, heartbeatRunId: source.runId, provider: "fake-sandbox", providerLeaseId: "sandbox-1",
|
||||
status: "released", releasedAt: new Date(), cleanupStatus: "success" });
|
||||
if (mode === "remote_unconfirmed_action") await appendHeartbeatRunEvent(db, {
|
||||
companyId: source.companyId, runId: source.runId, agentId: source.agentId,
|
||||
eventType: "tool.execution.started", stream: "system",
|
||||
payload: { name: "send_email", executionId: "unconfirmed-send", transport: "mcp" },
|
||||
});
|
||||
if (mode === "remote_completed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") {
|
||||
const execution = { schema: "paperclip.tool.execution.v1", executionId: "prt_1e7c7089f001EN3I04Bfmz",
|
||||
transport: "mcp", name: "send_email" };
|
||||
await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId,
|
||||
eventType: "tool.execution.started", stream: "system",
|
||||
payload: { prpEvent: { payload: { ...execution, status: "running" } } } });
|
||||
if (mode !== "remote_started_only_action") await appendHeartbeatRunEvent(db, { companyId: source.companyId,
|
||||
runId: source.runId, agentId: source.agentId, eventType: "tool.execution.completed", stream: "system",
|
||||
payload: { prpEvent: { payload: { ...execution, status: "completed",
|
||||
exitCode: mode === "remote_failed_exit_action" ? 1 : 0 } } } });
|
||||
}
|
||||
const verifyStoppedSession = vi.fn(async () => null);
|
||||
invalidateNativeReplacementDiscovery(db);
|
||||
await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
|
||||
const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.runId));
|
||||
const holds = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
|
||||
if (!remote) {
|
||||
expect(verifyStoppedSession).toHaveBeenCalled();
|
||||
expect(decision!.failureDetail?.replacementDenied).toBeUndefined();
|
||||
expect(holds).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
if (mode === "remote_unconfirmed_action" || mode === "remote_started_only_action" || mode === "remote_failed_exit_action") {
|
||||
expect(decision!.failureDetail?.replacementDenied).toBe("uncertain_external_action");
|
||||
expect(holds).toMatchObject([{ status: "active", cause: "uncertain_external_action",
|
||||
nextAction: expect.stringContaining("send_email") }]);
|
||||
expect(decision!.failureDetail?.unconfirmedActions).toEqual(["send_email"]);
|
||||
await settleUnrecoverableExecutions(db);
|
||||
const [continued] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
|
||||
expect(continued!.evidence).toMatchObject({ freshContinuation: { unverifiedActions: ["send_email"] },
|
||||
executionReconciliation: { outcomeEvidence: expect.stringContaining("unverified outcomes: send_email") } });
|
||||
return;
|
||||
}
|
||||
expect(decision!.failureDetail?.replacementDenied).toBe("provider_failure_meaning_unverified");
|
||||
expect(decision!.failureDetail?.unconfirmedActions).toBeUndefined();
|
||||
expect(holds).toMatchObject([{ status: "active", cause: "provider_failure_meaning_unverified" }]);
|
||||
await settleUnrecoverableExecutions(db);
|
||||
const [settled] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));
|
||||
expect(settled).toMatchObject({ status: "resolved", outcome: "restored",
|
||||
evidence: { freshContinuation: { trigger: "recovery_refused", sourceRunId: source.runId, unverifiedActions: [] },
|
||||
continuationDelivery: "pending" } });
|
||||
} finally {
|
||||
await rm(workspace, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
it("wakes a completed replacement scan after actual local stop proof commits", async () => {
|
||||
const source = await seed(2);
|
||||
const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-"));
|
||||
|
||||
@@ -21,7 +21,7 @@ import {
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
|
||||
import { decideNativeReplacement } from "./native-replacement-evidence.js";
|
||||
import { decideNativeReplacement, type NativeReplacementDecision } from "./native-replacement-evidence.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { buildExecutionContinuation } from "../execution-continuation.js";
|
||||
@@ -435,11 +435,15 @@ export async function reconcileSafeNativeReplacements(
|
||||
if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
|
||||
const preClaim = coordinator.failureCode === NATIVE_PRE_CLAIM_FAILURE_CODE;
|
||||
if (preClaim && !await preClaimFailureEvidence(db, run)) continue;
|
||||
const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim
|
||||
? await options.verifyStoppedSession?.(run) ?? null : null;
|
||||
const needsStopProof = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim;
|
||||
const stoppedSession = needsStopProof ? await options.verifyStoppedSession?.(run) ?? null : null;
|
||||
// A transport label alone is not evidence. Keep inspecting these candidates
|
||||
// as process cleanup and the final transcript become durable.
|
||||
if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !preClaim && !stoppedSession) continue;
|
||||
const providerKind = record(execution.provider).kind;
|
||||
const stopProofUnavailable = needsStopProof && !stoppedSession
|
||||
&& ((typeof providerKind === "string" && providerKind !== "codex")
|
||||
|| leases.some(lease => lease.provider !== null && lease.provider !== "local"));
|
||||
if (needsStopProof && !stoppedSession && !stopProofUnavailable) continue;
|
||||
const invocations = await db
|
||||
.select()
|
||||
.from(toolInvocations)
|
||||
@@ -540,7 +544,40 @@ export async function reconcileSafeNativeReplacements(
|
||||
&& run.runnerProfileJson?.nativeWorkspaceSync == null && nativeRunHasNoWorkspaceAccess(run);
|
||||
if (preClaim && (invocations.length || !noWorkspaceToSave
|
||||
&& (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped))) continue;
|
||||
const decision = decideNativeReplacement({
|
||||
const completedInvocation = (row: typeof invocations[number]) =>
|
||||
row.status === "succeeded" && Boolean(row.completedAt) && Boolean(row.resultHash);
|
||||
const executions = stopProofUnavailable ? (await db
|
||||
.select({ seq: heartbeatRunEvents.seq, eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
|
||||
inArray(heartbeatRunEvents.eventType, ["tool.execution.started", "tool.execution.completed"]))))
|
||||
.map(event => {
|
||||
const envelope = record(event.payload);
|
||||
return { seq: event.seq, eventType: event.eventType,
|
||||
payload: envelope.prpEvent ? record(record(envelope.prpEvent).payload) : envelope };
|
||||
}) : [];
|
||||
const unconfirmedActions = stopProofUnavailable ? [...new Set([
|
||||
...invocations.filter(row => row.riskLevel !== "read" && !completedInvocation(row)).map(row => row.toolName),
|
||||
...uncertainProviderActions.flatMap(action => {
|
||||
const started = executions.find(event => event.eventType === "tool.execution.started"
|
||||
&& String(event.payload.executionId ?? event.payload.name) === action);
|
||||
const executionId = started?.payload.executionId;
|
||||
if (started && executionId != null && executions.some(event => event.eventType === "tool.execution.completed"
|
||||
&& event.seq > started.seq && event.payload.executionId === executionId && event.payload.status === "completed"
|
||||
&& (event.payload.exitCode == null || event.payload.exitCode === 0) && event.payload.error == null))
|
||||
return [];
|
||||
if (started && started.payload.transport !== "process" && invocations.some(row => completedInvocation(row)
|
||||
&& (row.id === executionId || row.correlationId === executionId))) return [];
|
||||
return [typeof started?.payload.name === "string" ? started.payload.name : action];
|
||||
}),
|
||||
])] : [];
|
||||
const unprovenAction = unconfirmedActions[0];
|
||||
const decision: NativeReplacementDecision = stopProofUnavailable ? unprovenAction ? { allowed: false,
|
||||
cause: "uncertain_external_action",
|
||||
nextAction: `Reconcile ${unprovenAction} and preserve its result before continuing. Do not repeat it automatically.` }
|
||||
: { allowed: false, cause: "provider_failure_meaning_unverified",
|
||||
nextAction: "The previous session could not be confirmed stopped and recorded no unconfirmed action, so the task continues in a fresh run from its saved work." }
|
||||
: decideNativeReplacement({
|
||||
failedSession: true,
|
||||
// A facade transport failure can conceal an authoritative protocol
|
||||
// rejection. A stopped process and read receipts do not resolve that.
|
||||
@@ -618,6 +655,7 @@ export async function reconcileSafeNativeReplacements(
|
||||
...current.failureDetail,
|
||||
replacementDenied: decision.cause,
|
||||
nextAction,
|
||||
...(unconfirmedActions.length ? { unconfirmedActions } : {}),
|
||||
},
|
||||
updatedAt: now,
|
||||
})
|
||||
|
||||
@@ -22413,6 +22413,68 @@ describe("runnerd provider runtime wiring", () => {
|
||||
leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 }));
|
||||
});
|
||||
|
||||
it("saves a closing session's backup under its own owner after a failed follow-up released the scope authority", async () => {
|
||||
const current = { ...execution, binding: { ...execution.binding, runId: randomUUID(), issueId: randomUUID(),
|
||||
executionWorkspaceId: randomUUID() }, session: { ...execution.session, normalizedSessionId: randomUUID() } };
|
||||
const coordinator = { phase: "observed", resultId: null as string | null, leaseOwner: "provider-owner" as string | null,
|
||||
controllerGeneration: 1, controllerBootId: "backup-boot" };
|
||||
const db = leaseDb(current), select = db.select.bind(db);
|
||||
db.select = ((fields?: Record<string, unknown>) => ({ from: (table: unknown) => {
|
||||
if (table !== nativeRunFinalizations) return select(fields as never).from(table as never);
|
||||
const promise = Promise.resolve([coordinator]);
|
||||
const query = { then: promise.then.bind(promise), where: () => query, limit: () => query, for: () => query };
|
||||
return query;
|
||||
} })) as unknown as Db["select"];
|
||||
const storage = vi.spyOn(harnessStorage, "durableNativeHarnessStorage").mockReturnValue({ provider: "s3" } as never);
|
||||
const validateOwner = vi.spyOn(harnessStorage, "assertNativeHarnessBackupOwner").mockResolvedValue(undefined);
|
||||
const checkpointRegistration = vi.spyOn(runnerCheckpoint, "nativeRunnerCheckpointRegistration")
|
||||
.mockReturnValue(async () => undefined);
|
||||
onTestFinished(() => { storage.mockRestore(); validateOwner.mockRestore(); checkpointRegistration.mockRestore(); });
|
||||
let savingWorkspace = false;
|
||||
const remoteExecute = vi.fn(async (command: { args?: string[] }) => {
|
||||
if (savingWorkspace && command.args?.[1]?.includes("base64 <")) throw new Error("stopped-provider-inspection-reached");
|
||||
return { exitCode: command.args?.[1]?.includes("base64") ? 1 : 0, timedOut: false, stderr: "", stdout:
|
||||
command.args?.[0] === "--build-metadata" ? JSON.stringify({
|
||||
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", packageName: "@paperclipai/paperclip-runner",
|
||||
binaryContractVersion: 2, durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
|
||||
}) : command.args?.[0] === "--version" ? "codex-cli 0.160.0" : "" };
|
||||
});
|
||||
const controllerArtifact = join(isolatedStateDirectory, "backup-fixture-runnerd");
|
||||
await writeFile(controllerArtifact, "fixture bootstrap artifact");
|
||||
await createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner",
|
||||
runnerRemoteBinaryPath: controllerArtifact, runnerRemoteCodexPath: controllerArtifact,
|
||||
controllerPersistence: { bootId: "backup-boot" }, runnerIngressAuthorized: true, runnerExecutionTarget: {
|
||||
kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "released-scope-backup-lease",
|
||||
providerKey: "daytona", sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "released-scope-backup-sandbox" },
|
||||
effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: remoteExecute, syncIn: vi.fn(async () => undefined) },
|
||||
getRunnerIngressEndpoint: async () => ({ kind: "authenticated_websocket", websocketUrl: "wss://sandbox.invalid/runner",
|
||||
secretHeaders: [{ name: "Authorization", value: "fixture-ingress" }], generation: "1", refresh: vi.fn(), close: vi.fn(async () => undefined) }),
|
||||
} as never });
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & {
|
||||
controlPlaneRegistration: NonNullable<import("../../vendor/paperclip-runner/index.js").RunnerExecutionDriver["controlPlaneRegistration"]>;
|
||||
};
|
||||
const controller = new DurablePrpControlPlane({ stateDirectory: join(options.stateDirectory!, "control-plane"),
|
||||
identity: options.prpIdentity! as ConstructorParameters<typeof DurablePrpControlPlane>[0]["identity"],
|
||||
expectedRunnerVersion: "backup-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
|
||||
controller.queueCommand("run.prepare", {});
|
||||
controller.queueCommand("session.open", { reuse: "same_session" });
|
||||
let registration: Awaited<ReturnType<typeof options.controlPlaneRegistration>> | undefined;
|
||||
onTestFinished(async () => { await registration?.release(); await controller.stop(); });
|
||||
registration = await options.controlPlaneRegistration(controller);
|
||||
coordinator.phase = "workspace_finalizing";
|
||||
coordinator.resultId = "accepted-result";
|
||||
coordinator.leaseOwner = null;
|
||||
savingWorkspace = true;
|
||||
const authorities = sessionToolAuthorityCount();
|
||||
await expect(createRunnerdBackend({ db, execution: current, runnerInstanceId: "released-scope-backup-runner",
|
||||
executionContext: { signal: AbortSignal.abort() } as never })).rejects.toThrow();
|
||||
expect(sessionToolAuthorityCount()).toBe(authorities - 1);
|
||||
await expect(registration.checkpoint!("settled")).rejects.toThrow("stopped-provider-inspection-reached");
|
||||
expect(validateOwner).toHaveBeenCalledWith(db, expect.objectContaining({ runId: current.binding.runId,
|
||||
leaseOwner: "provider-owner", bootId: "backup-boot", generation: 1 }));
|
||||
});
|
||||
|
||||
it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => {
|
||||
const remoteCwd = join(isolatedStateDirectory, "remote");
|
||||
const current: NativeExecutionInputV1 = { ...execution,
|
||||
|
||||
@@ -18130,8 +18130,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
)
|
||||
return false;
|
||||
const durableStorage = durableNativeHarnessStorage();
|
||||
const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
const backupOwner = durableStorage && currentAuthority ? sessionHarnessBackupOwners.get(currentAuthority) : null;
|
||||
const currentAuthority = sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch;
|
||||
const backupOwner = durableStorage
|
||||
? sessionHarnessBackupOwners.get(currentAuthority) ?? sessionHarnessBackupOwners.get(authorityEpoch) ?? null
|
||||
: null;
|
||||
if (durableStorage && !backupOwner) throw new Error("native_harness_storage_owner_missing");
|
||||
if (backupOwner) await assertNativeHarnessBackupOwner(input.db, backupOwner);
|
||||
// Transport release also runs when provider bootstrap failed. In that case
|
||||
@@ -18929,9 +18931,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
target,
|
||||
controllerPersistence: input.controllerPersistence ? nativeRunnerCheckpointPersistence(
|
||||
input.db, input.controllerPersistence.bootId, () => {
|
||||
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
if (!current) throw new Error("native_session_tool_authority_unavailable");
|
||||
return current.runId;
|
||||
return (sessionToolAuthorityEpochs.get(sessionScopeId) ?? authorityEpoch).runId;
|
||||
}, detachNativeSessionsForRestart) : undefined,
|
||||
binding: input.remoteWarmRecovery
|
||||
? { ...input.execution.binding, runId: input.remoteWarmRecovery.identity.runId }
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import { environmentLeases, type Db } from "@paperclipai/db";
|
||||
import { retiredReusableLeasePredecessors } from "./environment-lease-handoff.js";
|
||||
|
||||
@@ -129,3 +129,21 @@ export async function remoteExecutionHasReleased(db: Db, companyId: string, runI
|
||||
const leases = await runRemoteLeases(db, companyId, runId);
|
||||
return leases.length > 0 && leases.every(hasRemoteTerminationReceipt);
|
||||
}
|
||||
|
||||
export async function terminatedRemoteCleanupScope(db: Db, companyId: string, remoteCleanupScope: string) {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(remoteCleanupScope);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!Array.isArray(parsed) || parsed.length !== 2 || typeof parsed[0] !== "string" || typeof parsed[1] !== "string"
|
||||
|| parsed[0] === "local") return null;
|
||||
const [provider, providerLeaseId] = parsed as [string, string];
|
||||
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, companyId),
|
||||
eq(environmentLeases.provider, provider), eq(environmentLeases.providerLeaseId, providerLeaseId)))
|
||||
.orderBy(desc(environmentLeases.createdAt), desc(environmentLeases.id));
|
||||
if (leases.length === 0 || leases.some(lease => !lease.releasedAt)) return null;
|
||||
const newest = await resolveRemoteTerminationLease(db, leases[0]!);
|
||||
return hasRemoteTerminationReceipt(newest) ? newest : null;
|
||||
}
|
||||
+2
@@ -213,6 +213,8 @@ export const nativeRestartContinuationInFlight = runner.nativeRestartContinuatio
|
||||
|
||||
export const completeTerminatedRemoteNativeSessionCleanup = runner.completeTerminatedRemoteNativeSessionCleanup;
|
||||
export const completeTerminatedLocalNativeSessionCleanup = runner.completeTerminatedLocalNativeSessionCleanup;
|
||||
export const completeTerminatedRemoteNativeSessionCleanupForTask = runner.completeTerminatedRemoteNativeSessionCleanupForTask;
|
||||
export const nativeSessionCleanupRemoteScopes = runner.nativeSessionCleanupRemoteScopes;
|
||||
export const nativeSessionCleanupBlock = runner.nativeSessionCleanupBlock;
|
||||
export const onNativeSessionCleanupSettled = runner.onNativeSessionCleanupSettled;
|
||||
export const onNativeSessionCleanupQuarantined = runner.onNativeSessionCleanupQuarantined;
|
||||
|
||||
Reference in new issue
Block a user