mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both
## Linked Issues or Issue Description
Refs #20572, #20578
**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.
**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.
**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.
**Paperclip version or commit**
master at ed3ffc3bc7.
## What Changed
- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.
## Verification
- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER
## Risks
- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed
1 parent
37c50f70c2
commit
805531cc71
9 files changed
+482
-33
No files matched your search
@@ -1754,8 +1754,15 @@ run's own saved checkpoint records a completed provider turn (for example, a
|
||||
same-run disposition recovery that the gate refused), the work is saved. If the
|
||||
task waits on unresolved blocker tasks, the task view shows that wait and no
|
||||
failure banner. Otherwise the missing task disposition stays visible with plain
|
||||
wording. Every quarantined run keeps Retry hidden, because the retry route refuses
|
||||
it; a new message continues once the previous session has stopped. These rules
|
||||
wording. A quarantined run hides Retry only while a live owner settles the gate:
|
||||
an unreleased remote environment, or a scheduled close in this server process.
|
||||
An operator-required gate, or one lost with a restarted process, has no owner, so
|
||||
the task offers Retry. Retry (like a new message) proves the previous runner and
|
||||
every recorded provider process stopped, then continues in a fresh session. When
|
||||
a local run's own close failed but the host has durably recorded that its exact
|
||||
runner process and group stopped and its recorded provider processes are gone,
|
||||
the run's own same-run recovery retires the gate before admission instead of
|
||||
being refused by it. These rules
|
||||
read only `errorCode` and `runnerProfileJson.sessionCheckpoint`, so runs that
|
||||
earlier releases quarantined are shown the same way without a migration.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, chatActions, chatEndpoints, companies, createDb, applyPendingMigrations, heartbeatRuns, issueRecoveryActions,
|
||||
import { agents, agentWakeupRequests, chatActions, chatEndpoints, companies, createDb, applyPendingMigrations, environmentLeases, heartbeatRuns, issueRecoveryActions,
|
||||
issueRelations, issues,
|
||||
toolApplications, toolConnections,
|
||||
nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
||||
@@ -212,20 +212,27 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
|
||||
await db.insert(heartbeatRuns).values(staged(runId, "completed"));
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId: task.id, runId, phase: "terminal_failure",
|
||||
attempt: 2, failureCode: "native_session_cleanup_quarantined" });
|
||||
// Without its own wait, a missing task disposition stays visible, with no Retry.
|
||||
// Nothing settles a gate without an owner (s9107 t04): Retry is offered.
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true });
|
||||
// An unreleased remote environment is that owner: its termination receipt retires the gate.
|
||||
const leaseId = randomUUID();
|
||||
await db.insert(environmentLeases).values({ id: leaseId, companyId, heartbeatRunId: runId, provider: "daytona",
|
||||
providerLeaseId: "sandbox" });
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: false,
|
||||
retryWait: { reason: "cleanup_quarantined" } });
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date() }).where(eq(environmentLeases.id, leaseId));
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true });
|
||||
// The task waits on an active child: it shows that wait, not a failed attempt.
|
||||
const [child] = await db.insert(issues).values({ companyId, title: "Verify review", identifier: "CMP-8",
|
||||
status: "in_progress", assigneeAgentId: agentId }).returning();
|
||||
await db.insert(issueRelations).values({ companyId, issueId: child.id, relatedIssueId: task.id, type: "blocks" });
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "resolved", canRetry: false });
|
||||
// A quarantine whose turn did not complete is still unfinished, but it never
|
||||
// offers the Retry that the retry route refuses.
|
||||
// A quarantine whose turn did not complete is still unfinished; with no
|
||||
// owner settling its gate, Retry is the way forward.
|
||||
await db.update(heartbeatRuns).set({ createdAt: new Date(Date.now() - 120_000) }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.insert(heartbeatRuns).values(staged(otherRunId, "interrupted"));
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: false,
|
||||
observedRunId: otherRunId, retryWait: { reason: "cleanup_quarantined" } });
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true,
|
||||
observedRunId: otherRunId });
|
||||
});
|
||||
|
||||
it("shows a queued-message Interrupt as handing off, not as a failed attempt", async () => {
|
||||
|
||||
@@ -16,8 +16,8 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { canContinueCancelledRun, userStopBoundaryAt } from "./run-cancellation.js";
|
||||
import { readQueuedInteractionResponse } from "./queued-interaction-response.js";
|
||||
import { isCancelledNativeStartup, isFailedNativePreparation, isFailedNativePreparationRetryAncestor } from "./cancelled-native-startup.js";
|
||||
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, nativeRunNeverLaunched, recordedLocalExecutionStopped, recordUnidentifiedLocalRunnerStop } from "./native-local-process-stop.js";
|
||||
import { completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, nativeGatedRunNeverLaunched, nativeRunNeverLaunched, recordedLocalExecutionStopped, recordUnidentifiedLocalRunnerStop } from "./native-local-process-stop.js";
|
||||
import { completeTerminatedLocalNativeSessionCleanup, completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js";
|
||||
import { z } from "zod";
|
||||
import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm";
|
||||
@@ -535,8 +535,30 @@ export async function admitExplicitNativeContinuation(input: {
|
||||
// Never permission to resume the interrupted turn: new input and a
|
||||
// native Retry both start a fresh provider conversation, and only after
|
||||
// the old runner, provider and tools are proven stopped.
|
||||
const { verifyStoppedNativeSessionForContinuation } = await import("./native-runtime/native-session-executor.js");
|
||||
const stopped = await verifyStoppedNativeSessionForContinuation(db, run);
|
||||
const { verifyStoppedNativeSessionForContinuation, provenStoppedOwnNativeSession, retireStoppedSessionCleanupGates } =
|
||||
await import("./native-runtime/native-session-executor.js");
|
||||
let stopped = await verifyStoppedNativeSessionForContinuation(db, run);
|
||||
// The run's own session (s9107 t04): its saved state may be past the
|
||||
// states the inventory proof reads, but its exact runner and every
|
||||
// provider process it recorded are durably proven stopped.
|
||||
const ownStop = stopped || retainedCleanup === "pending" ? null : await provenStoppedOwnNativeSession(db, run);
|
||||
if (ownStop && run.runnerInstanceId) {
|
||||
const runnerInstanceId = run.runnerInstanceId;
|
||||
stopped = { evidence: { schema: "paperclip.stopped_own_native_session.v1", runId: run.id,
|
||||
runnerInstanceId, ...ownStop },
|
||||
retire: () => { try { return completeTerminatedLocalNativeSessionCleanup({ companyId, runId: run.id, runnerInstanceId }); }
|
||||
catch { return false; } } };
|
||||
}
|
||||
// A source the gate refused before it launched (s9107 rerun t05: a
|
||||
// follow-up run after its predecessor's failed close) owns no process.
|
||||
// Once the task's gate is retired from its stopped owner, nothing holds
|
||||
// the old session any more.
|
||||
if (!stopped && retainedCleanup === "operator_required" && await nativeGatedRunNeverLaunched(db, run)
|
||||
// A preview stays read-only; the committing call retires the gate.
|
||||
&& await retireStoppedSessionCleanupGates(db, { companyId, issueId, preview: input.dryRun === true })) {
|
||||
stopped = { evidence: { schema: "paperclip.unlaunched_gated_source.v1", runId: run.id },
|
||||
retire: () => nativeSessionCleanupBlock({ companyId, issueId }) !== "operator_required" };
|
||||
}
|
||||
if (!stopped) return retry
|
||||
? blocked("cleanup_quarantined", "Send a new message after the previous provider has stopped.")
|
||||
: blocked("local_cleanup", "Waiting for the previous provider and its tools to stop. Your message is saved.");
|
||||
|
||||
@@ -2012,6 +2012,15 @@ export function heartbeatService(
|
||||
run = cleared;
|
||||
}
|
||||
}
|
||||
// A failed session close (this run's own, or a finished predecessor's) may
|
||||
// still hold this process's cleanup gate for the task. When its runner and
|
||||
// providers are proven stopped, retire it so admission is not refused.
|
||||
const cleanupGateIssueId = run.nativeIssueId ?? (typeof run.contextSnapshot?.issueId === "string" ? run.contextSnapshot.issueId : null);
|
||||
if (cleanupGateIssueId) {
|
||||
const { retireStoppedSessionCleanupGates } = await import("./native-runtime/native-session-executor.js");
|
||||
await retireStoppedSessionCleanupGates(db, { companyId: run.companyId, issueId: cleanupGateIssueId,
|
||||
...(run.runtimeMode === "native" ? { run } : {}) });
|
||||
}
|
||||
|
||||
if (run.runtimeMode === "legacy" && run.controllerBootId &&
|
||||
run.controllerBootId !== legacyControllerBootId) return;
|
||||
|
||||
@@ -287,6 +287,21 @@ export async function nativeRunNeverLaunched(db: Db, run: Pick<typeof heartbeatR
|
||||
return !launch;
|
||||
}
|
||||
|
||||
/** A native run the cleanup gate refused before it asked for a runner. Its
|
||||
* process fields may name its predecessor's process, copied at admission for
|
||||
* the same session; that process is checked separately. This run launched
|
||||
* nothing of its own. */
|
||||
export async function nativeGatedRunNeverLaunched(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "companyId" | "runtimeMode" | "status" | "errorCode">): Promise<boolean> {
|
||||
if (run.runtimeMode !== "native" || run.status !== "failed" || run.errorCode !== "native_session_cleanup_quarantined") return false;
|
||||
const [launch] = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
|
||||
or(isNotNull(heartbeatRunEvents.sourceEventId),
|
||||
inArray(heartbeatRunEvents.eventType, ["adapter.invoke", PROCESS_START_REQUESTED, PROCESS_IDENTITY_RECORDED])),
|
||||
)).limit(1);
|
||||
return !launch;
|
||||
}
|
||||
|
||||
/** Pre-receipt native runs can retain an exact suspended session after their
|
||||
* mutable process fields were cleared. This is admission evidence for a new
|
||||
* user turn only, never permission to replay the old run or infer its outcomes.
|
||||
|
||||
@@ -187,6 +187,7 @@ import {
|
||||
NativeSessionProtocolIntegrityError,
|
||||
completeRetainedNativeSessionCleanup,
|
||||
completeTerminatedLocalNativeSessionCleanup,
|
||||
nativeSessionCleanupBlock,
|
||||
acpxRuntimeSessionDirectoryName,
|
||||
OPENCODE_ISOLATED_HOMES_DIRECTORY,
|
||||
opencodeRuntimeSessionDirectoryName,
|
||||
@@ -3517,6 +3518,155 @@ export async function verifyStoppedNativeSessionForAutomaticReplacement(
|
||||
return verifyStoppedNativeSessionForContinuation(db, run);
|
||||
}
|
||||
|
||||
const SESSION_OPEN_EVENT_TYPES = ["session.started", "session.resumed", "session.reconciled"] as const;
|
||||
|
||||
/** This local run's exact runner process and group are durably recorded as
|
||||
* stopped (no later launch), and every provider process it recorded is gone.
|
||||
* Remote process IDs belong to another host, so remote runs never qualify. */
|
||||
export async function provenStoppedOwnNativeSession(
|
||||
db: Db,
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runtimeMode" | "runnerInstanceId" | "runnerProfileJson"
|
||||
| "processPid" | "processGroupId" | "processStartedAt">,
|
||||
): Promise<{ processPid: number; processGroupId: number; providerPids: number[] } | null> {
|
||||
if (run.runtimeMode !== "native" || !run.runnerInstanceId || process.platform === "win32") return null;
|
||||
if (activeNativeSessions.has(run.id)) return null;
|
||||
const leases = await db.select({ provider: environmentLeases.provider }).from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id)));
|
||||
if (leases.some(lease => lease.provider !== "local")) return null;
|
||||
// A finished run whose process fields were never cleared proves its stop
|
||||
// from the recorded launch identity instead (the same prover as above).
|
||||
const stopped = await readNativeLocalProcessStop(db, run.companyId, run.id)
|
||||
?? (run.processPid && run.processPid > 1 && run.processGroupId === run.processPid
|
||||
&& await recordedLocalExecutionStopped(db, run) ? { processPid: run.processPid, processGroupId: run.processPid } : null);
|
||||
if (!stopped) return null;
|
||||
const events = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload })
|
||||
.from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
|
||||
inArray(heartbeatRunEvents.eventType, [...SESSION_OPEN_EVENT_TYPES, "harness.diagnostic"])));
|
||||
// Every recorded provider process identity, from the durable projection,
|
||||
// the saved checkpoint and the run's own raw runner journal. Codex session
|
||||
// opens carry the PID only in the raw journal; ACPX reports its sidecar in
|
||||
// a diagnostic. Extra identities can only add a stop requirement.
|
||||
const providerPids: unknown[] = [];
|
||||
const pidKeys = ["processId", "process_id", "pid", "processGroupId", "providerPid", "codexPid", "sidecarPid", "agentPid",
|
||||
"agentProcessId", "previousProcessId"];
|
||||
const collect = (...owners: unknown[]) => {
|
||||
for (const owner of owners) for (const key of pidKeys) {
|
||||
const value = record(owner)[key];
|
||||
if (value != null) providerPids.push(value);
|
||||
}
|
||||
};
|
||||
const collectSessionOpen = (payload: Record<string, unknown>) =>
|
||||
collect(payload, payload.runtimeIdentity, payload.providerDescriptor, payload.providerIdentity, payload.started);
|
||||
let sessionOpens = 0;
|
||||
for (const event of events) {
|
||||
const payload = record(record(record(event.payload).prpEvent).payload);
|
||||
if (event.eventType === "harness.diagnostic") {
|
||||
if (payload.providerMethod === "acpx/process") providerPids.push(payload.pid);
|
||||
if (payload.code === "acpx_process") {
|
||||
const details = record(payload.details);
|
||||
// A sidecar in its own group may leave its agent behind, which no record names.
|
||||
if (details.processGroupId != null) return null;
|
||||
providerPids.push(details.pid);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
sessionOpens += 1;
|
||||
collectSessionOpen(payload);
|
||||
}
|
||||
const checkpoint = record(record(run.runnerProfileJson).sessionCheckpoint);
|
||||
collect(checkpoint.process);
|
||||
// Without a parseable execution input there is no raw journal to locate;
|
||||
// the projection and checkpoint above remain the evidence.
|
||||
let execution: NativeExecutionInput | null = null;
|
||||
try {
|
||||
execution = parseNativeExecutionInput(record(run.runnerProfileJson).nativeExecutionInput);
|
||||
} catch {
|
||||
execution = null;
|
||||
}
|
||||
// An existing journal that cannot be read may name a live provider, so it fails closed.
|
||||
try {
|
||||
const root = execution ? scopedRunnerdStateRoot(execution) : null;
|
||||
if (execution && root && existsSync(root)) {
|
||||
const snapshot = cleanupStateSnapshot(root, runnerProviderStateFilename(execution));
|
||||
if (record(snapshot.control.identity).runId === run.id) {
|
||||
collect(snapshot.provider, record(snapshot.provider).identity, record(snapshot.provider).startupAttempt);
|
||||
for (const entry of Array.isArray(snapshot.control.committedEvents) ? snapshot.control.committedEvents : []) {
|
||||
const event = record(record(record(entry).envelope).payload);
|
||||
if (!(SESSION_OPEN_EVENT_TYPES as readonly string[]).includes(String(event.eventType))) continue;
|
||||
sessionOpens += 1;
|
||||
collectSessionOpen(record(event.payload));
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
// A provider that opened a session but left no process identity cannot be proven gone.
|
||||
if (sessionOpens > 0 && providerPids.length === 0) return null;
|
||||
if (!providerPids.every(cleanupProcessAbsent)) return null;
|
||||
return { processPid: stopped.processPid, processGroupId: stopped.processGroupId, providerPids: providerPids as number[] };
|
||||
}
|
||||
|
||||
type CleanupGateOwner = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId" | "runtimeMode"
|
||||
| "nativeIssueId" | "runnerInstanceId" | "runnerProfileJson" | "processPid" | "processGroupId" | "processStartedAt">;
|
||||
// Runs the gate refused before launch also count toward this bound, so it is
|
||||
// wide enough for a long refused-Retry history behind the owner.
|
||||
const CLEANUP_GATE_OWNER_SCAN_LIMIT = 50;
|
||||
|
||||
/**
|
||||
* A local session close can fail without proof that its runner suspended (for
|
||||
* example while the server was paused), which leaves an operator-required
|
||||
* cleanup gate in this process. It refuses every later session of the task:
|
||||
* the run's own same-run recovery, and a follow-up run for a new message.
|
||||
* The gate guards process ownership only. When the host proves that the
|
||||
* owner's exact runner process and group stopped, and every provider process
|
||||
* the owner recorded is gone, nothing can still own that session, so the gate
|
||||
* is retired before the next session is admitted.
|
||||
*
|
||||
* This never authorizes provider work: the caller's own recovery or admission
|
||||
* decision, made from durable records, still decides how the task continues.
|
||||
* Returns true when no operator-required gate blocks the task afterwards.
|
||||
*/
|
||||
export async function retireStoppedSessionCleanupGates(
|
||||
db: Db,
|
||||
input: { companyId: string; issueId: string; run?: CleanupGateOwner;
|
||||
/** Read-only: true when some finished owner of the task is proven stopped. Retires nothing. */
|
||||
preview?: boolean },
|
||||
): Promise<boolean> {
|
||||
const blocked = () => nativeSessionCleanupBlock({ companyId: input.companyId, issueId: input.issueId }) === "operator_required";
|
||||
if (!blocked()) return true;
|
||||
const finished = await db.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
||||
eq(heartbeatRuns.runtimeMode, "native"), isNotNull(heartbeatRuns.finishedAt), isNotNull(heartbeatRuns.runnerInstanceId),
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
|
||||
)).orderBy(desc(heartbeatRuns.createdAt)).limit(CLEANUP_GATE_OWNER_SCAN_LIMIT);
|
||||
const owners: CleanupGateOwner[] = [...(input.run ? [input.run] : []), ...finished.filter(row => row.id !== input.run?.id)];
|
||||
for (const owner of owners) {
|
||||
if (!blocked()) break;
|
||||
if (!owner.runnerInstanceId || owner.companyId !== input.companyId) continue;
|
||||
const stopped = await provenStoppedOwnNativeSession(db, owner);
|
||||
if (!stopped) continue;
|
||||
if (input.preview) return true;
|
||||
let retired = false;
|
||||
try {
|
||||
retired = completeTerminatedLocalNativeSessionCleanup({
|
||||
companyId: owner.companyId, runId: owner.id, runnerInstanceId: owner.runnerInstanceId,
|
||||
});
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
// True also when this owner held no entry; record only a real release.
|
||||
if (!retired || blocked()) continue;
|
||||
await appendHeartbeatRunEvent(db, {
|
||||
companyId: owner.companyId, runId: owner.id, agentId: owner.agentId,
|
||||
eventType: "native.session_cleanup_gate_retired", stream: "system", level: "info",
|
||||
message: "The run's runner and provider processes are verified stopped, so its session cleanup gate was retired.",
|
||||
payload: { processPid: stopped.processPid, processGroupId: stopped.processGroupId, providerPids: stopped.providerPids },
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
return !blocked();
|
||||
}
|
||||
|
||||
/** Exact local cleanup only: the accepted result and original quarantine are
|
||||
* never rewritten. A failed/interrupted maintenance attempt is retained for
|
||||
* inspection, not retried from an older snapshot with unknown process owners. */
|
||||
|
||||
@@ -9,7 +9,7 @@ import {
|
||||
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { executeNativeSession, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import { executeNativeSession, NativeSessionCloseUnrecoverableError, nativeSessionCleanupBlock, onNativeSessionCleanupSettled, parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import type { NativeSession } from "../vendor/paperclip-runner/index.js";
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
@@ -19,7 +19,7 @@ import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
|
||||
import { currentProcessStartedAtMs } from "./controller-process-identity.js";
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { scopedRunnerdStateRoot, verifyStoppedNativeSessionForAutomaticReplacement, verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
|
||||
import { retireStoppedSessionCleanupGates, scopedRunnerdStateRoot, verifyStoppedNativeSessionForAutomaticReplacement, verifyStoppedNativeSessionForContinuation } from "./native-runtime/native-session-executor.js";
|
||||
import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
|
||||
@@ -405,4 +405,180 @@ describe("task Retry after inherited native cleanup quarantine", () => {
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
// s9107 t04/t05: the server paused (SIGSTOP) after a turn completed, its
|
||||
// runner checkpoint owner lapsed, and the session close could not prove the
|
||||
// runner suspended, leaving an operator-required gate in this process.
|
||||
async function quarantineOwnClose(f: { companyId: string; agentId: string; issueId: string; sourceId: string;
|
||||
workspaceId: string; runnerInstanceId: string }, base: string, runId = f.sourceId) {
|
||||
const identity = { runId, sessionId: "own-close-session", companyId: f.companyId, issueId: f.issueId, agentId: f.agentId };
|
||||
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: false, structuredResult: true };
|
||||
const session: NativeSession = {
|
||||
identity: () => identity, capabilities: async () => capabilities,
|
||||
async *events() { throw new Error("native_runner_checkpoint_owner_lost"); },
|
||||
startTurn: async () => ({ turnId: "own-close-turn" }), result: async () => null,
|
||||
async snapshot() { return { backendKind: "mock", sessionId: "own-close-driver", identity,
|
||||
providerSessionId: "own-close-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }; },
|
||||
close: async () => { throw new NativeSessionCloseUnrecoverableError({ runnerSuspended: false, providerDrained: true }); },
|
||||
};
|
||||
await expect(executeNativeSession({
|
||||
input: parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", provider: { kind: "codex", model: null },
|
||||
binding: { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, runId, executionWorkspaceId: f.workspaceId },
|
||||
task: { identifier: "TEST", title: "Continue saved work", description: null, prompt: "Continue saved work", workMode: "standard" },
|
||||
workspace: { cwd: base, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: identity.sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: "contract", sha256: "sha", schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: { revision: "1", objective: "Continue saved work", criteria: [{ id: "done", requirement: "Save the result" }] } },
|
||||
interactionResponses: [], credentialBindings: [] }),
|
||||
backend: { descriptor: async () => ({ kind: "runner", name: "own-close-backend", version: "1", capabilities }),
|
||||
openSession: async () => session },
|
||||
controlPlane: { openRun: async () => {}, checkpointSession: async () => {},
|
||||
appendEvent: async () => ({ cursor: 0, highestContiguousSourceSeq: 0, disposition: "committed" }),
|
||||
replayEvents: async () => ({ events: [], highestContiguousSourceSeq: 0 }), completeRun: async () => {} },
|
||||
runnerInstanceId: f.runnerInstanceId, controlPlaneInstanceId: "control",
|
||||
})).rejects.toThrow();
|
||||
await vi.waitFor(() => expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBe("operator_required"));
|
||||
}
|
||||
|
||||
// An ACPX session open names no provider PID; its sidecar reports one separately (s9107 t04).
|
||||
async function recordOwnStop(f: { companyId: string; agentId: string; sourceId: string }, providerPid: number,
|
||||
sidecar?: { processGroupId: number | null }, options: { noIdentity?: boolean; startedPid?: boolean } = {}) {
|
||||
const processStartedAt = new Date(Date.now() - 90_000).toISOString();
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: PROCESS_IDENTITY_RECORDED,
|
||||
payload: { processPid: 99_999_996, processGroupId: 99_999_996, processStartedAt, targetKind: "local", hostname: hostname() } });
|
||||
// The real Codex projection names no PID in session opens; the checkpoint does.
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
||||
eventType: "session.started", payload: { prpEvent: { payload: { providerSessionId: "own-provider", driverSessionId: "own-provider",
|
||||
...(options.startedPid ? { started: { pid: providerPid } } : {}) } } } });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
||||
eventType: "session.reconciled", payload: { prpEvent: { payload: { providerSessionId: "own-provider", turnCount: 1,
|
||||
activeTurnId: null, terminalTurnId: "own-turn" } } } });
|
||||
if (!sidecar && !options.noIdentity && !options.startedPid) {
|
||||
const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...row.runnerProfileJson,
|
||||
sessionCheckpoint: { ...(row.runnerProfileJson as { sessionCheckpoint: object }).sessionCheckpoint, process: { providerPid } } } })
|
||||
.where(eq(heartbeatRuns.id, f.sourceId));
|
||||
}
|
||||
if (sidecar) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId,
|
||||
eventType: "harness.diagnostic", payload: { prpEvent: { payload: { code: "acpx_process",
|
||||
details: { pid: providerPid, role: "sidecar", processGroupId: sidecar.processGroupId } } } } });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "native.local_process_stopped",
|
||||
payload: { processPid: 99_999_996, processGroupId: 99_999_996, processStartedAt } });
|
||||
}
|
||||
|
||||
it.each(["stopped", "provider_alive", "no_stop", "no_provider_identity", "started_pid", "acpx_sidecar", "acpx_sidecar_own_group"] as const)(
|
||||
"retires a run's own failed-close gate once its runner and providers are proven stopped (%s)", async mode => {
|
||||
const base = await mkdtemp(join(tmpdir(), "own-close-gate-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
||||
try {
|
||||
const f = await seed(base);
|
||||
await quarantineOwnClose(f, base);
|
||||
if (mode !== "no_stop") await recordOwnStop(f, mode === "provider_alive" ? process.pid : 99_999_995,
|
||||
mode === "acpx_sidecar" ? { processGroupId: null } : mode === "acpx_sidecar_own_group" ? { processGroupId: 99_999_995 } : undefined,
|
||||
{ noIdentity: mode === "no_provider_identity", startedPid: mode === "started_pid" });
|
||||
const retires = mode === "stopped" || mode === "started_pid" || mode === "acpx_sidecar";
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
expect(await retireStoppedSessionCleanupGates(db, { companyId: f.companyId, issueId: f.issueId, run: source })).toBe(retires);
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId }))
|
||||
.toBe(retires ? null : "operator_required");
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["stopped", "provider_alive"] as const)(
|
||||
"starts a task Retry for a run's own quarantine proven stopped by its own process record (%s)", async mode => {
|
||||
const base = await mkdtemp(join(tmpdir(), "own-quarantine-retry-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
||||
try {
|
||||
const f = await seed(base);
|
||||
// The source launched its own runner, so the retained owner's inventory
|
||||
// cannot stand in for it; only its own stop record can.
|
||||
await recordOwnStop(f, mode === "provider_alive" ? process.pid : 99_999_995);
|
||||
await db.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null })
|
||||
.where(eq(heartbeatRuns.id, f.sourceId));
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
expect(await verifyStoppedNativeSessionForContinuation(db, source)).toBeNull();
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
const [state] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
if (mode === "provider_alive") {
|
||||
expect(queued).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
expect(queued, JSON.stringify(state.payload)).toHaveLength(1);
|
||||
expect(queued[0].contextSnapshot).toMatchObject({ previousRunId: f.sourceId, forceFreshSession: true });
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["stopped", "runner_alive"] as const)(
|
||||
"retires a finished predecessor's failed-close gate before a follow-up session (%s)", async mode => {
|
||||
// s9107 rerun t05: the run succeeded, its close kept an operator-required
|
||||
// gate, and the follow-up run for a new message was refused at admission.
|
||||
const base = await mkdtemp(join(tmpdir(), "predecessor-close-gate-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
||||
try {
|
||||
const f = await seed(base);
|
||||
await quarantineOwnClose(f, base);
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
// No stop record: the predecessor's recorded launch identity proves its stop.
|
||||
const pid = mode === "runner_alive" ? process.pid : source.processPid!;
|
||||
const startedAt = mode === "runner_alive" ? new Date((await currentProcessStartedAtMs())!) : source.processStartedAt!;
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null, processPid: pid, processGroupId: pid, processStartedAt: startedAt })
|
||||
.where(eq(heartbeatRuns.id, f.sourceId));
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: PROCESS_IDENTITY_RECORDED,
|
||||
payload: { processPid: pid, processGroupId: pid, processStartedAt: startedAt.toISOString(),
|
||||
targetKind: "local", hostname: hostname() } });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "harness.diagnostic",
|
||||
payload: { prpEvent: { payload: { code: "acpx_process", details: { pid: 99_999_994, role: "sidecar", processGroupId: null } } } } });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.sourceId, agentId: f.agentId, eventType: "session.started",
|
||||
payload: { prpEvent: { payload: { providerSessionId: "own-provider" } } } });
|
||||
expect(await retireStoppedSessionCleanupGates(db, { companyId: f.companyId, issueId: f.issueId })).toBe(mode === "stopped");
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId }))
|
||||
.toBe(mode === "stopped" ? null : "operator_required");
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("starts a task Retry for a gated follow-up that never launched once its predecessor is proven stopped", async () => {
|
||||
// s9107 rerun t05: the gate is the finished owner's; the refused source
|
||||
// launched nothing. The queue previews admission before committing it, so
|
||||
// the preview must leave the gate for the committing call to retire.
|
||||
const base = await mkdtemp(join(tmpdir(), "gated-follow-up-retry-"));
|
||||
const previous = process.env.PAPERCLIP_RUNNER_STATE_DIR;
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = base;
|
||||
try {
|
||||
const f = await seed(base);
|
||||
await quarantineOwnClose(f, base, f.ownerId);
|
||||
// The retained-state inventory proof no longer applies to this saved state.
|
||||
await writeFile(join(f.root, "runner", "codex-provider-state.json"), JSON.stringify({
|
||||
...f.files["runner/codex-provider-state.json"], lifecycle: "ready" }));
|
||||
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
|
||||
expect(await verifyStoppedNativeSessionForContinuation(db, source)).toBeNull();
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running" });
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
actorSource: "local_implicit", observedRunId: f.sourceId, idempotencyKey: randomUUID() });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
const [state] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
expect(queued, JSON.stringify(state.payload)).toHaveLength(1);
|
||||
expect(nativeSessionCleanupBlock({ companyId: f.companyId, issueId: f.issueId })).toBeNull();
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.PAPERCLIP_RUNNER_STATE_DIR; else process.env.PAPERCLIP_RUNNER_STATE_DIR = previous;
|
||||
await rm(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -170,13 +170,22 @@ it("names a budget gate instead of generic execution checks for a refused Retry"
|
||||
message: "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage." } });
|
||||
});
|
||||
|
||||
it.each(["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed"])("does not offer Retry for terminal Retry waits: %s", reason => {
|
||||
it.each(["cancelled_by_operator", "retry_source_changed"])("does not offer Retry for terminal Retry waits: %s", reason => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||||
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: reason } }))
|
||||
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason } });
|
||||
});
|
||||
|
||||
it("holds a Retry on a cleanup gate only while an owner settles it", () => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "cleanup_quarantined" };
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, wait, cleanupSettling: true }))
|
||||
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "cleanup_quarantined" } });
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, wait }))
|
||||
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "cleanup_quarantined" } });
|
||||
});
|
||||
|
||||
it("keeps Retry for a Retry declined for its requester's access, since a new click replaces it", () => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||||
@@ -492,21 +501,42 @@ describe("native cleanup quarantine presentation", () => {
|
||||
waitingOnBlockers: true, paused: false, approval: false })).toMatchObject({ state: "resolved", canRetry: false });
|
||||
});
|
||||
|
||||
it("keeps a missing disposition visible when the task has no wait of its own", () => {
|
||||
expect(projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup: true }, agent,
|
||||
paused: false, approval: false })).toMatchObject({ state: "failed", canRetry: false,
|
||||
retryWait: { reason: "cleanup_quarantined" } });
|
||||
it("waits without Retry while an owner settles the gate", () => {
|
||||
for (const completedTurnCleanup of [true, false]) {
|
||||
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
|
||||
cleanupSettling: true, paused: false, approval: false });
|
||||
expect(presentation).toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "cleanup_quarantined" } });
|
||||
expect(presentation?.retryWait?.message).not.toMatch(/operator|admin/i);
|
||||
}
|
||||
const preparation = { id: "retry", requestedAt: new Date(0), sourceRunId: quarantined.id, preparationStatus: "failed" };
|
||||
expect(projectTaskInterruption(issue, { run: quarantined, agent, retryWait: preparation, cleanupSettling: true,
|
||||
paused: false, approval: false })).toMatchObject({ canRetry: false });
|
||||
});
|
||||
|
||||
it("never offers a Retry that the retry route refuses", () => {
|
||||
const presentation = projectTaskInterruption(issue, { run: quarantined, agent, paused: false, approval: false });
|
||||
expect(presentation).toMatchObject({ state: "failed", canRetry: false,
|
||||
retryWait: { reason: "cleanup_quarantined" } });
|
||||
expect(presentation?.retryWait?.message).not.toMatch(/operator|admin/i);
|
||||
// Every other path that offers Retry is refused for the same run too.
|
||||
const preparation = { id: "retry", requestedAt: new Date(0), sourceRunId: quarantined.id, preparationStatus: "failed" };
|
||||
expect(projectTaskInterruption(issue, { run: quarantined, agent, retryWait: preparation, paused: false, approval: false }))
|
||||
.toMatchObject({ canRetry: false });
|
||||
it("offers Retry when nothing settles the gate by itself (s9107 t04/t05)", () => {
|
||||
// An operator-required gate, or one lost with a restarted process, has no
|
||||
// owner. Waiting would never end, so Retry is the way forward.
|
||||
for (const completedTurnCleanup of [true, false]) {
|
||||
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
|
||||
paused: false, approval: false });
|
||||
expect(presentation).toMatchObject({ state: "failed", canRetry: true });
|
||||
expect(presentation).not.toHaveProperty("retryWait");
|
||||
}
|
||||
});
|
||||
|
||||
it("never leaves a failed task without Retry, a scheduled retry or a settling owner", () => {
|
||||
// Invariant: a failed presentation without Retry must name a wait that an
|
||||
// owner is settling. Enumerate the quarantine-related inputs.
|
||||
const waits = [undefined, ...["cleanup_quarantined", "local_cleanup", "execution_settling"].map(reason =>
|
||||
({ id: "retry", requestedAt: new Date(0), sourceRunId: quarantined.id, executionWaitReason: reason }))];
|
||||
for (const completedTurnCleanup of [true, false]) for (const wait of waits) for (const settling of [true, false]) {
|
||||
const presentation = projectTaskInterruption(issue, { run: { ...quarantined, completedTurnCleanup }, agent,
|
||||
retryWait: wait, cleanupSettling: settling, paused: false, approval: false });
|
||||
if (presentation?.state !== "failed" || presentation.canRetry) continue;
|
||||
expect(presentation.retryWait, JSON.stringify({ completedTurnCleanup, wait: wait?.executionWaitReason, settling }))
|
||||
.toBeDefined();
|
||||
if (presentation.retryWait?.reason === "cleanup_quarantined") expect(settling).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps Retry for ordinary failed native attempts", () => {
|
||||
|
||||
@@ -2,7 +2,8 @@ import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavai
|
||||
import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js";
|
||||
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
|
||||
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
|
||||
import { unionAll } from "drizzle-orm/pg-core";
|
||||
import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
|
||||
@@ -82,6 +83,10 @@ type ProjectionState = {
|
||||
retryDependencies?: RetryDependencies;
|
||||
/** The task waits on unresolved blocker tasks, a wait that does not depend on its last run. */
|
||||
waitingOnBlockers?: boolean;
|
||||
/** A live owner is settling this run's session cleanup gate: an unreleased
|
||||
* remote environment, or a scheduled close in this process. Without one,
|
||||
* nothing settles the gate by itself, so Retry is the way forward. */
|
||||
cleanupSettling?: boolean;
|
||||
};
|
||||
|
||||
// A task read reports the change that made a task fail, not every later read of
|
||||
@@ -183,11 +188,13 @@ function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string |
|
||||
|
||||
/** Presentation never grants authority. Dispatch repeats all gates under its
|
||||
* locks. The same projection is used in detail, lists and attention surfaces.
|
||||
* It never offers a Retry that the retry route refuses: a native run refused
|
||||
* by the cleanup gate cannot be retried (routes/agents.ts, canRetryStoppedRun). */
|
||||
* A native run refused by the cleanup gate offers no Retry only while a live
|
||||
* owner settles that gate. Without one, the task Retry (explicit continuation)
|
||||
* proves the old session stopped and continues in a fresh session. */
|
||||
export function projectTaskInterruption(issue: ProjectionIssue, input: ProjectionState, now = Date.now()): TaskInterruptionPresentation | null {
|
||||
const presentation = projectTaskInterruptionState(issue, input, now);
|
||||
if (!presentation?.canRetry || input.run?.errorCode !== NATIVE_SESSION_CLEANUP_QUARANTINED) return presentation;
|
||||
if (!presentation?.canRetry || input.run?.errorCode !== NATIVE_SESSION_CLEANUP_QUARANTINED || !input.cleanupSettling)
|
||||
return presentation;
|
||||
return { ...presentation, canRetry: false };
|
||||
}
|
||||
|
||||
@@ -317,7 +324,8 @@ function projectTaskInterruptionState(issue: ProjectionIssue, input: ProjectionS
|
||||
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
|
||||
// Keep the latest input receipt so message submission can settle independently.
|
||||
return { ...base, state: "failed",
|
||||
canRetry: !["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required"].includes(reason), retryWait: { reason,
|
||||
canRetry: !["cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required"].includes(reason)
|
||||
&& (reason !== "cleanup_quarantined" || !input.cleanupSettling), retryWait: { reason,
|
||||
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
|
||||
}
|
||||
if (wait && wait.requestedAt.getTime() > now - 10_000 && system?.state !== "exhausted")
|
||||
@@ -328,12 +336,34 @@ function projectTaskInterruptionState(issue: ProjectionIssue, input: ProjectionS
|
||||
// its blockers shows that wait, not a failed attempt. Without such a wait a
|
||||
// missing task disposition stays visible.
|
||||
if (run.completedTurnCleanup && input.waitingOnBlockers) return base;
|
||||
// Waiting is honest only while something settles the gate. Otherwise
|
||||
// Retry proves the old session stopped and continues from saved work.
|
||||
if (!input.cleanupSettling) return { ...base, state: "failed", canRetry: true };
|
||||
return { ...base, state: "failed", canRetry: false,
|
||||
retryWait: { reason: "cleanup_quarantined", message: NATIVE_CLEANUP_SETTLING_MESSAGE } };
|
||||
}
|
||||
return { ...base, state: "failed", canRetry: true, ...(projectAccess ? { projectAccess } : {}) };
|
||||
}
|
||||
|
||||
/** Tasks whose cleanup-gated run still has a live owner settling the gate: a
|
||||
* remote environment not yet released (its termination receipt retires the
|
||||
* gate), or a scheduled close in this process. An operator-required gate or a
|
||||
* gate lost with a restarted process has no owner. */
|
||||
async function quarantineCleanupSettling(db: Db, companyId: string,
|
||||
entries: Array<{ issueId: string; run: { id: string; nativeIssueId?: string | null } }>): Promise<Set<string>> {
|
||||
const settling = new Set<string>();
|
||||
if (!entries.length) return settling;
|
||||
const unreleased = new Set((await db.select({ runId: environmentLeases.heartbeatRunId }).from(environmentLeases).where(and(
|
||||
eq(environmentLeases.companyId, companyId), inArray(environmentLeases.heartbeatRunId, entries.map(entry => entry.run.id)),
|
||||
isNull(environmentLeases.releasedAt), sql`${environmentLeases.provider} is distinct from 'local'`,
|
||||
))).map(row => row.runId));
|
||||
for (const { issueId, run } of entries) {
|
||||
if (unreleased.has(run.id) || nativeSessionCleanupBlock({ companyId, issueId: run.nativeIssueId ?? issueId }) === "pending")
|
||||
settling.add(issueId);
|
||||
}
|
||||
return settling;
|
||||
}
|
||||
|
||||
/** A scheduled retry has no outcome yet, so its provider failure is read from
|
||||
* the predecessor it retries: the countdown keeps its reason.
|
||||
* Batch list reads rather than doing seven queries per row. Keep run logs and
|
||||
@@ -617,6 +647,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
return run && wait.sourceRunId === run.id && wait.executionWaitReason === "process_identity_missing" ? [{ issueId, run }] : [];
|
||||
});
|
||||
const launchRefusals = await unidentifiedLocalLaunchRefusals(db, launchWaits.map(entry => entry.run));
|
||||
const cleanupSettling = await quarantineCleanupSettling(db, companyId,
|
||||
[...runMap].flatMap(([issueId, run]) => run.errorCode === NATIVE_SESSION_CLEANUP_QUARANTINED ? [{ issueId, run }] : []));
|
||||
const confirmableLaunches = new Set(launchWaits.filter(entry => launchRefusals.get(entry.run.id) === null).map(entry => entry.issueId));
|
||||
const now = Date.now();
|
||||
for (const issue of selected) {
|
||||
@@ -636,6 +668,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
admissionWait: admissionWaitMap.get(issue.id),
|
||||
retryDependencies: retryDependencyMap.get(issue.id),
|
||||
waitingOnBlockers: waitingOnBlockers.has(issue.id),
|
||||
cleanupSettling: cleanupSettling.has(issue.id),
|
||||
}, now);
|
||||
result.set(issue.id, presentation);
|
||||
const run = facts.run;
|
||||
|
||||
Reference in new issue
Block a user