fix(restart-chaos): grade double executors on local runner identities only (#22307)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness grades release swaps. One rule,
`MS_double_executor`, flags a run that records a second runner identity
before the first one is shown stopped.
> - The server records stop evidence (`native.local_process_stopped`)
only for local processes. It never records it for a sandbox
(`targetKind: "remote"`) process.
> - So when a sandbox run relaunches its runner inside the same run (a
session retry after a reconnect), the grader counts each new sandbox PID
as a second executor. Sandbox PIDs are also in a different PID space
from the server.
> - In a 197-task wave, run `c5b1f8ae` (a Daytona run that retried its
session after a reconnect) was reported as 2 double executors. The
runner journal shows the first turn was stopped with
`providerExitConfirmed: true` before the relaunch, so nothing ran twice.
> - This pull request makes the rule grade only local identities, as
`SCENARIOS.md` already describes it ("a second local runner identity").
> - The benefit is that the duplicate count in release-swap results
shows only real duplicates.

## Linked Issues or Issue Description

**What happened?**
The release-swap grade reported "2 double executor(s)" for one Daytona
run whose session retried in the same run. The previous provider process
had confirmed exit. No two processes ran together.

**Expected behavior**
`MS_double_executor` charges only local runner identities. A sandbox
relaunch is not a double executor.

**Steps to reproduce**
1. Run a restart-chaos wave on Daytona.
2. Have one run retry its native session in the same run (a reconnect).
3. Read `release-swap-summary.md`. The run appears under double
executors once per relaunch.

**Paperclip version or commit**
`master` at 1c67687998.

## What Changed

- `tests/restart-chaos/multi-server.ts`: reads `payload.targetKind` for
runner identity events and skips `remote` identities in
`MS_double_executor`.
- `tests/restart-chaos/multi-server.test.ts`: adds a red/green test with
three sandbox relaunches of one run.
- `tests/restart-chaos/SCENARIOS.md`: states that sandbox identities are
not graded, and why.

## Verification

- Red: without the filter, the new test fails (2 violations for 3 remote
identities).
- Green: `npx vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/multi-server.test.ts` passes (16 tests).
- `tsc -p tests/restart-chaos/tsconfig.json` shows no errors in the
changed files.

## Risks

- Low risk. Harness only; no product code.
- A real double executor inside a sandbox is not caught by this rule. It
was never reliably caught before: no remote stop evidence exists, so
every remote relaunch was flagged. Sandbox ownership stays graded by the
Runner checkpoint owner checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-19 03:50:33 -06:00
1 parent 7eaf38130b
commit 70a4351343
4 files changed
+18 -3

No files matched your search

+2
View File
@@ -1686,6 +1686,8 @@ server's pid, start, exit and freezes). A violation fails the wave.
`controller_lease_lost` counts only when no freeze past the lease came before it.
- `MS_double_executor`: a run recorded a second local runner identity
(`native.process_identity_recorded`) before the first was shown stopped (`native.local_process_stopped`).
Sandbox identities (`targetKind: "remote"`) are not graded: the server records stop evidence only for
local processes, so every in-run sandbox relaunch would look like a second executor.
- Ownership failures whose owner is not a wave server (or is unknown) are listed as `unattributed`.
The usual R1–R6, U1–U4 and T checks grade every task as before. In the speed grade, MS1 and MS2 take no
@@ -33,6 +33,9 @@ describe.skipIf(!support.supported)("reading cross-server evidence from a wave d
await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a2', $1, null, 'succeeded', null, $2, $2, null)`, [COMPANY, iso(NOON)]);
await db.$client.unsafe(`insert into heartbeat_run_events values ('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 11}', $1),
('00000000-0000-4000-8000-0000000000a2', 'native.process_identity_recorded', '{"processPid": 12}', $2)`, [iso(NOON - 2_000), iso(NOON - 1_000)]);
await db.$client.unsafe(`insert into heartbeat_runs values ('00000000-0000-4000-8000-0000000000a3', $1, null, 'succeeded', null, $2, $2, null)`, [COMPANY, iso(NOON)]);
await db.$client.unsafe(`insert into heartbeat_run_events values ('00000000-0000-4000-8000-0000000000a3', 'native.process_identity_recorded', '{"processPid": 21, "targetKind": "remote"}', $1),
('00000000-0000-4000-8000-0000000000a3', 'native.process_identity_recorded', '{"processPid": 22, "targetKind": "remote"}', $2)`, [iso(NOON - 2_000), iso(NOON - 1_000)]);
}, 120_000);
afterAll(async () => {
@@ -45,6 +48,7 @@ describe.skipIf(!support.supported)("reading cross-server evidence from a wave d
expect(evidence.runs).toEqual([expect.objectContaining({ runId: "00000000-0000-4000-8000-0000000000a1", status: "running",
errorCode: "native_execution_ownership_unverified", endedAt: NOON, ownerBootId: "boot-a", ownerPid: 100 })]);
expect(evidence.events.map(event => event.pid)).toEqual(expect.arrayContaining([11, 12]));
expect(evidence.events.filter(event => event.runId.endsWith("a3")).map(event => event.targetKind)).toEqual(["remote", "remote"]);
const windows: ServerProcessWindow[] = [
{ pid: 100, role: "primary", spawnedAt: NOON - 60_000, exitedAt: null, frozen: [{ from: NOON - 50_000, to: NOON - 5_000 }] },
{ pid: 200, role: "secondary", spawnedAt: NOON - 55_000, exitedAt: null, frozen: [] },
+7
View File
@@ -145,4 +145,11 @@ describe("checkCrossServer", () => {
// The same runner recorded again (a re-attach) is one executor.
expect(checkCrossServer({ runs, events: [recorded(1, 1), recorded(1, 5)], windows: [A] }).violations).toEqual([]);
});
it("does not charge sandbox runner relaunches, which never record local stop evidence", () => {
const remote = (pid: number, at: number) => ({ runId: "r1", kind: "recorded" as const, pid, at, targetKind: "remote" as const });
const runs = [run({ errorCode: null, status: "succeeded" })];
expect(checkCrossServer({ runs, events: [remote(65122, 1), remote(57393, 5), remote(49664, 9)], windows: [A] }).violations)
.toEqual([]);
});
});
+5 -3
View File
@@ -173,7 +173,7 @@ export interface CrossServerRun {
ownerPid: number | null;
}
export interface RunIdentityEvent { runId: string; kind: "recorded" | "stopped"; pid: number; at: number }
export interface RunIdentityEvent { runId: string; kind: "recorded" | "stopped"; pid: number; at: number; targetKind?: "local" | "remote" }
export interface CrossServerViolation {
rule: "MS_foreign_failure" | "MS_double_executor";
@@ -228,6 +228,7 @@ export function checkCrossServer(input: { runs: readonly CrossServerRun[]; event
}
const byRun = new Map<string, RunIdentityEvent[]>();
for (const event of input.events) {
if (event.targetKind === "remote") continue;
const list = byRun.get(event.runId) ?? [];
list.push(event);
byRun.set(event.runId, list);
@@ -268,7 +269,7 @@ export async function readCrossServerEvidence(db: Db, companyId: string): Promis
[companyId]) as Row[];
const events = await client.unsafe(
`select e.run_id::text as "runId", e.event_type as "eventType", e.payload->>'processPid' as pid,
floor(extract(epoch from e.created_at) * 1000) as at
e.payload->>'targetKind' as "targetKind", floor(extract(epoch from e.created_at) * 1000) as at
from heartbeat_run_events e join heartbeat_runs r on r.id = e.run_id
where r.company_id::text = $1 and e.event_type in ('native.process_identity_recorded', 'native.local_process_stopped')`, [companyId]) as Row[];
return {
@@ -279,7 +280,8 @@ export async function readCrossServerEvidence(db: Db, companyId: string): Promis
const pid = num(row.pid);
const at = num(row.at);
if (pid === null || at === null || !Number.isSafeInteger(pid)) return [];
return [{ runId: String(row.runId), kind: row.eventType === "native.local_process_stopped" ? "stopped" as const : "recorded" as const, pid, at }];
return [{ runId: String(row.runId), kind: row.eventType === "native.local_process_stopped" ? "stopped" as const : "recorded" as const, pid, at,
...(row.targetKind === "remote" ? { targetKind: "remote" as const } : {}) }];
}),
};
}