Files
PaperClipAI/server/src/services/native-preclaim-continuation.test.ts
T
0e38c80323 fix(recovery): do not auto-replace a chat Retry under its consumed exact-request authority (#22414)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs that fail before they do provider work can continue
automatically. The native safe-replacement sweep schedules a system
successor run for them.
> - The successor copies the failed run's context. The sweep removes
consumed wake fields, but it keeps the `chatFailedRunRetry` selector and
`taskRetryRequestId`.
> - That selector is the authority of an exact chat Retry. It is bound
to the failed run's own chat action and to the user's Retry intent. The
successor gets a new system wakeup request, so admission can never
authorize it.
> - Thus every automatic successor of a chat Retry fails at setup with
`FailedChatRunRetryAuthorizationError`. The sweep has already retired
the task's recovery hold at that time, so the task can be left with no
owner of the next step.
> - This pull request stops the sweep from replacing a run that carries
`chatFailedRunRetry`.
> - The benefit is that the task keeps its hold. The user's next Retry
or message continues it through the normal authorized path, and no run
fails for a known reason.

## Linked Issues or Issue Description

**What happened?**
On staging, a user pressed Retry on a failed GitHub chat run. The Retry
run failed before its first claim, after a workspace restore failure.
Its own exact-request authority refused it
(`chat_failed_run_retry_not_authorized`). The pre-claim continuation
path then scheduled a `native_safe_replacement` successor. The successor
kept the copied `chatFailedRunRetry` selector under a new wakeup request
ID, and `authorizeFailedChatRunRetryWake` refused it at setup.

**Expected behavior**
An automatic system successor must not be scheduled under chat Retry
authority that it cannot hold. The task must keep its recovery hold for
the user's next Retry or message.

**Steps to reproduce**
1. Make a native run whose context has `chatFailedRunRetry` (an exact
chat Retry).
2. Fail it before its first coordinator claim, or with a provider
terminal failure, so it qualifies for safe replacement.
3. Run `reconcileSafeNativeReplacements`.
4. See that a successor run is scheduled with the copied selector. At
admission it always fails with `FailedChatRunRetryAuthorizationError`.

Related: #22413 changes the same file for a different defect (the
reviewed chat message batch on fresh-session retries). It does not touch
the eligibility predicate or `chatFailedRunRetry`. The two changes are
independent.

## What Changed

- `nativeReplacementPending()` now excludes a failed run whose context
carries `chatFailedRunRetry`. Because the shared eligibility predicate
has the guard, the pre-claim, historical listener, and provider
terminal-failure replacement paths all get it.
- New regression in `native-preclaim-continuation.test.ts`: this is the
staging shape (a pre-claim failure of an exact chat Retry after its
saved work is recovered). It schedules no successor, the hold stays
effective, and the task stays blocked.
- New paired regression in `native-safe-replacement.test.ts` for the
provider terminal-failure path. An ordinary run still gets exactly one
successor. An exact chat Retry gets none.

## Verification

- Red before the fix: the pre-claim chat Retry case failed with
`expected length 0, got 1`. The terminal-failure chat Retry case also
failed. Its ordinary-run control passed.
- Green after the fix, rebased on `fe0c4aa95d`:
- `npx vitest run src/services/native-preclaim-continuation.test.ts
src/services/native-runtime/native-safe-replacement.test.ts`: 89 passed.
- `npx vitest run
src/services/native-runtime/native-safe-replacement.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/execution-control-work.test.ts
src/__tests__/recovery-settled-records-census.test.ts
src/__tests__/durable-chat-wakeup.test.ts`: 206 passed.
  - `pnpm --filter @paperclipai/server typecheck`: passed.
- Second-order check: `nativeInterruptionHold()` uses
`not(nativeReplacementPending())`. Thus an excluded run can now reach
the existing fresh-continuation fallback, the same as other pre-claim
runs that do not qualify. That fallback records a continuation decision
and lets normal admission build a fresh context. It does not copy the
predecessor's selector.
- `pnpm verify` selects 2,950 of 3,817 files for this widely imported
module. I ran the focused suites above and rely on required CI for the
broad coverage.

## Risks

- Low risk. The change only removes candidates from automatic
replacement. Before this change, a removed candidate always produced a
successor that failed at setup.
- Behavior shift: after a chat Retry run fails, the user (or a new chat
message) continues the task. A system successor that cannot be
authorized does not start. Ordinary chat-message runs and non-chat runs
are not changed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 07:53:08 -08:00

198 lines
13 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { recoverRetainedWorkspace } from "./workspace-recovery.js";
import { adoptSystemRecovery } from "./system-recovery.js";
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
import { getExecutionBlocker } from "./execution-blocker.js";
import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
import * as storageBackend from "../storage/index.js";
import { acceptedRemoteExport, action, connectionString, db, run, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js";
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
vi.mock("./environment-execution-target.js", async importOriginal => {
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
});
/**
* The staging shape of a native run that failed before its first coordinator
* claim (2026-11-07: a resumed run aborted at the harness-backup restore
* deadline). No provider process, no claim, no provider or tool events. The
* failure path moved the coordinator to `terminal_failure` with attempt 0 and
* an earlier release recorded a resolved `preserve_without_replay_v1` hold
* that blocks replay and the task.
*/
async function preClaimNativeFailure() {
const f = await acceptedRemoteExport("published");
const source = await run(f.runId);
await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, f.runId));
await db.delete(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", attempt: 0, leaseOwner: null, leaseExpiresAt: null,
failureCode: "native_continuation_requires_reconciliation", failureDetail: null })
.where(eq(nativeRunFinalizations.runId, f.runId));
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] };
const remoteWorkspace = source.runnerProfileJson!.remoteWorkspace as { remoteCwd: string };
await db.update(heartbeatRuns).set({
status: "failed", finishedAt: new Date(), error: "The operation was aborted",
errorCode: "native_continuation_requires_reconciliation", completionContractId: null,
sessionIdBefore: "ses_previous_provider_session", nativeSessionId: randomUUID(), nativePhase: "terminal_failure",
contextSnapshot: { issueId: f.issueId, wakeReason: "External chat message received" },
resultJson: { stopReason: "adapter_failed", timeoutFired: false, timeoutSource: "config",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false, effectiveTimeoutSec: 0,
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery },
runnerProfileJson: { ...source.runnerProfileJson,
nativeExecutionInput: { workspace: { cwd: remoteWorkspace.remoteCwd } } },
}).where(eq(heartbeatRuns.id, f.runId));
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
eventType: "lifecycle", stream: "system", level: "info", message: "run started" });
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
eventType: "error", stream: "system", level: "error", message: "The operation was aborted" });
await db.update(issues).set({ status: "blocked", statusVersion: 18, executionRunId: null, checkoutRunId: null })
.where(eq(issues.id, f.issueId));
// The hold exactly as e2d490c61 recorded it (resolved, replay blocked, board-owned).
await db.update(issueRecoveryActions).set({
cause: "native_continuation_requires_reconciliation", fingerprint: `native-workspace:${f.runId}`,
status: "resolved", outcome: "blocked", ownerType: "board", resolvedAt: new Date(),
nextAction: "The original sandbox is retained for workspace repair. Verify its exact stop receipt, recover the missing files, and record workspaceRepairEvidence without changing the current task. Repair does not replay the stopped run. Explicitly remove the retained allocation after recovery.",
evidence: { runId: f.runId, originalFailureCode: null, continuationDelivery: "invalidated",
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery,
automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked",
actionOutcome: "unknown", recordedAt: new Date().toISOString() },
nativeFailureBlock: { runId: f.runId, statusVersion: 18 } },
}).where(eq(issueRecoveryActions.id, f.actionId));
await adoptSystemRecovery(db, f);
return { ...f, agentId: source.agentId };
}
async function successorsOf(runId: string) {
return db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
}
async function sweep() {
invalidateNativeReplacementDiscovery(db);
return reconcileSafeNativeReplacements(db);
}
describe("native run that failed before its first coordinator claim", () => {
useWorkspaceRecoveryFixture(targets);
// A published remote workspace: the shape staging persists and restores.
const priorEnv = { databaseUrl: process.env.DATABASE_URL, masterKey: process.env.PAPERCLIP_SECRETS_MASTER_KEY };
beforeEach(() => {
const local = storageBackend.getStorageService();
vi.spyOn(storageBackend, "getStorageService").mockReturnValue({ ...local, provider: "s3" });
process.env.DATABASE_URL = connectionString; process.env.PAPERCLIP_SECRETS_MASTER_KEY = "63".repeat(32);
});
afterEach(() => {
vi.restoreAllMocks();
if (priorEnv.databaseUrl === undefined) delete process.env.DATABASE_URL; else process.env.DATABASE_URL = priorEnv.databaseUrl;
if (priorEnv.masterKey === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY;
else process.env.PAPERCLIP_SECRETS_MASTER_KEY = priorEnv.masterKey;
});
it("continues automatically exactly once after its saved work is recovered", async () => {
const f = await preClaimNativeFailure();
await recoverRetainedWorkspace(f.input);
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
await Promise.all([sweep(), sweep()]);
await sweep();
const successors = await successorsOf(f.runId);
expect(successors).toHaveLength(1);
expect(successors[0]).toMatchObject({ status: "scheduled_retry", agentId: f.agentId,
scheduledRetryReason: "native_safe_replacement",
contextSnapshot: { issueId: f.issueId, retryOfRunId: f.runId, nativePreClaimContinuationOf: f.runId } });
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, successors[0].id));
expect(wake).toMatchObject({ status: "queued", requestedByActorType: "system" });
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
expect(coordinator.failureDetail).toMatchObject({ successorRunId: successors[0].id, preClaimContinuation: true });
const hold = await action(f.actionId);
expect(hold).toMatchObject({ status: "resolved", outcome: "handed_back",
evidence: { automaticRecovery: { replay: "verified_safe_replacement", successorRunId: successors[0].id } } });
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task).toMatchObject({ status: "in_progress", executionRunId: null, checkoutRunId: null });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
}, 60_000);
it("waits for saved work before continuing", async () => {
const f = await preClaimNativeFailure();
await sweep();
expect(await successorsOf(f.runId)).toHaveLength(0);
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task.status).toBe("blocked");
}, 60_000);
it("does not chain a second automatic continuation from its own successor", async () => {
const f = await preClaimNativeFailure();
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, nativePreClaimContinuationOf: randomUUID() } })
.where(eq(heartbeatRuns.id, f.runId));
await recoverRetainedWorkspace(f.input);
await sweep();
expect(await successorsOf(f.runId)).toHaveLength(0);
}, 60_000);
// Staging 2026-11-20: a user's exact chat Retry failed before its first
// claim. Its authority is bound to its own chat action and Retry intent, so
// a system successor carrying the copied selector is always refused at
// setup. The saved hold keeps the task for the user's next Retry or message.
it("does not continue a chat Retry automatically under its consumed exact-request authority", async () => {
const f = await preClaimNativeFailure();
const current = await run(f.runId);
// The failed run the user retried. It precedes the Retry, so it does not
// supersede it.
const [failedSource] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
status: "failed", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(current.createdAt.getTime() - 60_000) })
.returning();
const retryOfRunId = failedSource!.id;
await db.update(heartbeatRuns).set({
errorCode: "chat_failed_run_retry_not_authorized", retryOfRunId,
contextSnapshot: { ...current.contextSnapshot, retryOfRunId, forceFreshSession: true, taskRetryRequestId: randomUUID(),
chatFailedRunRetry: { version: 1, actionId: current.wakeupRequestId ?? randomUUID(), failedRunId: retryOfRunId } },
}).where(eq(heartbeatRuns.id, f.runId));
await recoverRetainedWorkspace(f.input);
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
await sweep();
expect(await successorsOf(f.runId)).toHaveLength(0);
const hold = await action(f.actionId);
expect(hold).toMatchObject({ outcome: "blocked", evidence: { automaticRecovery: { replay: "blocked" } } });
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(task.status).toBe("blocked");
}, 60_000);
it.each(["claimed", "provider_event", "tool_call", "uncertain_side_effect", "process", "superseded"] as const)(
"keeps reconciliation when the run may have acted: %s", async gate => {
const f = await preClaimNativeFailure();
await recoverRetainedWorkspace(f.input);
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
if (gate === "claimed") await db.update(nativeRunFinalizations).set({ attempt: 1, controllerGeneration: 1 })
.where(eq(nativeRunFinalizations.runId, f.runId));
if (gate === "provider_event") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
agentId: f.agentId, eventType: "turn.started", stream: "system", level: "info", payload: {} });
if (gate === "tool_call") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
agentId: f.agentId, eventType: "tool.execution.started", stream: "system", level: "info",
payload: { name: "paperclip_create_issue", executionId: randomUUID() } });
if (gate === "uncertain_side_effect") {
const current = await run(f.runId);
await db.update(heartbeatRuns).set({ resultJson: { ...current.resultJson,
executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } }).where(eq(heartbeatRuns.id, f.runId));
}
if (gate === "process") await db.update(heartbeatRuns).set({ processPid: 2_000_000_000, processStartedAt: new Date() })
.where(eq(heartbeatRuns.id, f.runId));
if (gate === "superseded") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
status: "succeeded", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(Date.now() + 1000) });
await sweep();
expect(await successorsOf(f.runId)).toHaveLength(0);
const hold = await action(f.actionId);
expect(hold).toMatchObject({ outcome: "blocked", evidence: { automaticRecovery: { replay: "blocked" } } });
const [task] = await db.select().from(issues).where(and(eq(issues.id, f.issueId)));
expect(task.status).toBe("blocked");
}, 60_000);
});