fix(native): never leave a cleanup-gated task without Retry or an owner (#20639)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both

## Linked Issues or Issue Description

Refs #20572, #20578

**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.

**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.

**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.

**Paperclip version or commit**
master at ed3ffc3bc7.

## What Changed

- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.

## Verification

- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER

## Risks

- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
DottaandClaude authored and GitHub committed 2026-11-10 15:20:42 -06:00
1 parent 37c50f70c2
commit 805531cc71
9 files changed
+482 -33

No files matched your search

@@ -187,6 +187,7 @@ import {
NativeSessionProtocolIntegrityError,
completeRetainedNativeSessionCleanup,
completeTerminatedLocalNativeSessionCleanup,
nativeSessionCleanupBlock,
acpxRuntimeSessionDirectoryName,
OPENCODE_ISOLATED_HOMES_DIRECTORY,
opencodeRuntimeSessionDirectoryName,
@@ -3517,6 +3518,155 @@ export async function verifyStoppedNativeSessionForAutomaticReplacement(
return verifyStoppedNativeSessionForContinuation(db, run);
}
const SESSION_OPEN_EVENT_TYPES = ["session.started", "session.resumed", "session.reconciled"] as const;
/** This local run's exact runner process and group are durably recorded as
* stopped (no later launch), and every provider process it recorded is gone.
* Remote process IDs belong to another host, so remote runs never qualify. */
export async function provenStoppedOwnNativeSession(
db: Db,
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runtimeMode" | "runnerInstanceId" | "runnerProfileJson"
| "processPid" | "processGroupId" | "processStartedAt">,
): Promise<{ processPid: number; processGroupId: number; providerPids: number[] } | null> {
if (run.runtimeMode !== "native" || !run.runnerInstanceId || process.platform === "win32") return null;
if (activeNativeSessions.has(run.id)) return null;
const leases = await db.select({ provider: environmentLeases.provider }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id)));
if (leases.some(lease => lease.provider !== "local")) return null;
// A finished run whose process fields were never cleared proves its stop
// from the recorded launch identity instead (the same prover as above).
const stopped = await readNativeLocalProcessStop(db, run.companyId, run.id)
?? (run.processPid && run.processPid > 1 && run.processGroupId === run.processPid
&& await recordedLocalExecutionStopped(db, run) ? { processPid: run.processPid, processGroupId: run.processPid } : null);
if (!stopped) return null;
const events = await db.select({ eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload })
.from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
inArray(heartbeatRunEvents.eventType, [...SESSION_OPEN_EVENT_TYPES, "harness.diagnostic"])));
// Every recorded provider process identity, from the durable projection,
// the saved checkpoint and the run's own raw runner journal. Codex session
// opens carry the PID only in the raw journal; ACPX reports its sidecar in
// a diagnostic. Extra identities can only add a stop requirement.
const providerPids: unknown[] = [];
const pidKeys = ["processId", "process_id", "pid", "processGroupId", "providerPid", "codexPid", "sidecarPid", "agentPid",
"agentProcessId", "previousProcessId"];
const collect = (...owners: unknown[]) => {
for (const owner of owners) for (const key of pidKeys) {
const value = record(owner)[key];
if (value != null) providerPids.push(value);
}
};
const collectSessionOpen = (payload: Record<string, unknown>) =>
collect(payload, payload.runtimeIdentity, payload.providerDescriptor, payload.providerIdentity, payload.started);
let sessionOpens = 0;
for (const event of events) {
const payload = record(record(record(event.payload).prpEvent).payload);
if (event.eventType === "harness.diagnostic") {
if (payload.providerMethod === "acpx/process") providerPids.push(payload.pid);
if (payload.code === "acpx_process") {
const details = record(payload.details);
// A sidecar in its own group may leave its agent behind, which no record names.
if (details.processGroupId != null) return null;
providerPids.push(details.pid);
}
continue;
}
sessionOpens += 1;
collectSessionOpen(payload);
}
const checkpoint = record(record(run.runnerProfileJson).sessionCheckpoint);
collect(checkpoint.process);
// Without a parseable execution input there is no raw journal to locate;
// the projection and checkpoint above remain the evidence.
let execution: NativeExecutionInput | null = null;
try {
execution = parseNativeExecutionInput(record(run.runnerProfileJson).nativeExecutionInput);
} catch {
execution = null;
}
// An existing journal that cannot be read may name a live provider, so it fails closed.
try {
const root = execution ? scopedRunnerdStateRoot(execution) : null;
if (execution && root && existsSync(root)) {
const snapshot = cleanupStateSnapshot(root, runnerProviderStateFilename(execution));
if (record(snapshot.control.identity).runId === run.id) {
collect(snapshot.provider, record(snapshot.provider).identity, record(snapshot.provider).startupAttempt);
for (const entry of Array.isArray(snapshot.control.committedEvents) ? snapshot.control.committedEvents : []) {
const event = record(record(record(entry).envelope).payload);
if (!(SESSION_OPEN_EVENT_TYPES as readonly string[]).includes(String(event.eventType))) continue;
sessionOpens += 1;
collectSessionOpen(record(event.payload));
}
}
}
} catch {
return null;
}
// A provider that opened a session but left no process identity cannot be proven gone.
if (sessionOpens > 0 && providerPids.length === 0) return null;
if (!providerPids.every(cleanupProcessAbsent)) return null;
return { processPid: stopped.processPid, processGroupId: stopped.processGroupId, providerPids: providerPids as number[] };
}
type CleanupGateOwner = Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId" | "runtimeMode"
| "nativeIssueId" | "runnerInstanceId" | "runnerProfileJson" | "processPid" | "processGroupId" | "processStartedAt">;
// Runs the gate refused before launch also count toward this bound, so it is
// wide enough for a long refused-Retry history behind the owner.
const CLEANUP_GATE_OWNER_SCAN_LIMIT = 50;
/**
* A local session close can fail without proof that its runner suspended (for
* example while the server was paused), which leaves an operator-required
* cleanup gate in this process. It refuses every later session of the task:
* the run's own same-run recovery, and a follow-up run for a new message.
* The gate guards process ownership only. When the host proves that the
* owner's exact runner process and group stopped, and every provider process
* the owner recorded is gone, nothing can still own that session, so the gate
* is retired before the next session is admitted.
*
* This never authorizes provider work: the caller's own recovery or admission
* decision, made from durable records, still decides how the task continues.
* Returns true when no operator-required gate blocks the task afterwards.
*/
export async function retireStoppedSessionCleanupGates(
db: Db,
input: { companyId: string; issueId: string; run?: CleanupGateOwner;
/** Read-only: true when some finished owner of the task is proven stopped. Retires nothing. */
preview?: boolean },
): Promise<boolean> {
const blocked = () => nativeSessionCleanupBlock({ companyId: input.companyId, issueId: input.issueId }) === "operator_required";
if (!blocked()) return true;
const finished = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.runtimeMode, "native"), isNotNull(heartbeatRuns.finishedAt), isNotNull(heartbeatRuns.runnerInstanceId),
inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]),
)).orderBy(desc(heartbeatRuns.createdAt)).limit(CLEANUP_GATE_OWNER_SCAN_LIMIT);
const owners: CleanupGateOwner[] = [...(input.run ? [input.run] : []), ...finished.filter(row => row.id !== input.run?.id)];
for (const owner of owners) {
if (!blocked()) break;
if (!owner.runnerInstanceId || owner.companyId !== input.companyId) continue;
const stopped = await provenStoppedOwnNativeSession(db, owner);
if (!stopped) continue;
if (input.preview) return true;
let retired = false;
try {
retired = completeTerminatedLocalNativeSessionCleanup({
companyId: owner.companyId, runId: owner.id, runnerInstanceId: owner.runnerInstanceId,
});
} catch {
continue;
}
// True also when this owner held no entry; record only a real release.
if (!retired || blocked()) continue;
await appendHeartbeatRunEvent(db, {
companyId: owner.companyId, runId: owner.id, agentId: owner.agentId,
eventType: "native.session_cleanup_gate_retired", stream: "system", level: "info",
message: "The run's runner and provider processes are verified stopped, so its session cleanup gate was retired.",
payload: { processPid: stopped.processPid, processGroupId: stopped.processGroupId, providerPids: stopped.providerPids },
}).catch(() => undefined);
}
return !blocked();
}
/** Exact local cleanup only: the accepted result and original quarantine are
* never rewritten. A failed/interrupted maintenance attempt is retained for
* inspection, not retried from an older snapshot with unknown process owners. */