mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs that fail before they do provider work can continue automatically. The native safe-replacement sweep schedules a system successor run for them. > - The successor copies the failed run's context. The sweep removes consumed wake fields, but it keeps the `chatFailedRunRetry` selector and `taskRetryRequestId`. > - That selector is the authority of an exact chat Retry. It is bound to the failed run's own chat action and to the user's Retry intent. The successor gets a new system wakeup request, so admission can never authorize it. > - Thus every automatic successor of a chat Retry fails at setup with `FailedChatRunRetryAuthorizationError`. The sweep has already retired the task's recovery hold at that time, so the task can be left with no owner of the next step. > - This pull request stops the sweep from replacing a run that carries `chatFailedRunRetry`. > - The benefit is that the task keeps its hold. The user's next Retry or message continues it through the normal authorized path, and no run fails for a known reason. ## Linked Issues or Issue Description **What happened?** On staging, a user pressed Retry on a failed GitHub chat run. The Retry run failed before its first claim, after a workspace restore failure. Its own exact-request authority refused it (`chat_failed_run_retry_not_authorized`). The pre-claim continuation path then scheduled a `native_safe_replacement` successor. The successor kept the copied `chatFailedRunRetry` selector under a new wakeup request ID, and `authorizeFailedChatRunRetryWake` refused it at setup. **Expected behavior** An automatic system successor must not be scheduled under chat Retry authority that it cannot hold. The task must keep its recovery hold for the user's next Retry or message. **Steps to reproduce** 1. Make a native run whose context has `chatFailedRunRetry` (an exact chat Retry). 2. Fail it before its first coordinator claim, or with a provider terminal failure, so it qualifies for safe replacement. 3. Run `reconcileSafeNativeReplacements`. 4. See that a successor run is scheduled with the copied selector. At admission it always fails with `FailedChatRunRetryAuthorizationError`. Related: #22413 changes the same file for a different defect (the reviewed chat message batch on fresh-session retries). It does not touch the eligibility predicate or `chatFailedRunRetry`. The two changes are independent. ## What Changed - `nativeReplacementPending()` now excludes a failed run whose context carries `chatFailedRunRetry`. Because the shared eligibility predicate has the guard, the pre-claim, historical listener, and provider terminal-failure replacement paths all get it. - New regression in `native-preclaim-continuation.test.ts`: this is the staging shape (a pre-claim failure of an exact chat Retry after its saved work is recovered). It schedules no successor, the hold stays effective, and the task stays blocked. - New paired regression in `native-safe-replacement.test.ts` for the provider terminal-failure path. An ordinary run still gets exactly one successor. An exact chat Retry gets none. ## Verification - Red before the fix: the pre-claim chat Retry case failed with `expected length 0, got 1`. The terminal-failure chat Retry case also failed. Its ordinary-run control passed. - Green after the fix, rebased on `fe0c4aa95d`: - `npx vitest run src/services/native-preclaim-continuation.test.ts src/services/native-runtime/native-safe-replacement.test.ts`: 89 passed. - `npx vitest run src/services/native-runtime/native-safe-replacement.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/execution-control-work.test.ts src/__tests__/recovery-settled-records-census.test.ts src/__tests__/durable-chat-wakeup.test.ts`: 206 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Second-order check: `nativeInterruptionHold()` uses `not(nativeReplacementPending())`. Thus an excluded run can now reach the existing fresh-continuation fallback, the same as other pre-claim runs that do not qualify. That fallback records a continuation decision and lets normal admission build a fresh context. It does not copy the predecessor's selector. - `pnpm verify` selects 2,950 of 3,817 files for this widely imported module. I ran the focused suites above and rely on required CI for the broad coverage. ## Risks - Low risk. The change only removes candidates from automatic replacement. Before this change, a removed candidate always produced a successor that failed at setup. - Behavior shift: after a chat Retry run fails, the user (or a new chat message) continues the task. A system successor that cannot be authorized does not start. Ordinary chat-message runs and non-chat runs are not changed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in an agent harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
198 lines
13 KiB
TypeScript
198 lines
13 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults } from "@paperclipai/db";
|
|
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
|
import { recoverRetainedWorkspace } from "./workspace-recovery.js";
|
|
import { adoptSystemRecovery } from "./system-recovery.js";
|
|
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
|
import { getExecutionBlocker } from "./execution-blocker.js";
|
|
import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
|
|
import * as storageBackend from "../storage/index.js";
|
|
import { acceptedRemoteExport, action, connectionString, db, run, useWorkspaceRecoveryFixture } from "./__tests__/workspace-recovery-fixture.js";
|
|
|
|
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
|
vi.mock("./environment-execution-target.js", async importOriginal => {
|
|
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
|
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
|
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
|
});
|
|
|
|
/**
|
|
* The staging shape of a native run that failed before its first coordinator
|
|
* claim (2026-11-07: a resumed run aborted at the harness-backup restore
|
|
* deadline). No provider process, no claim, no provider or tool events. The
|
|
* failure path moved the coordinator to `terminal_failure` with attempt 0 and
|
|
* an earlier release recorded a resolved `preserve_without_replay_v1` hold
|
|
* that blocks replay and the task.
|
|
*/
|
|
async function preClaimNativeFailure() {
|
|
const f = await acceptedRemoteExport("published");
|
|
const source = await run(f.runId);
|
|
await db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await db.delete(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
|
|
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", attempt: 0, leaseOwner: null, leaseExpiresAt: null,
|
|
failureCode: "native_continuation_requires_reconciliation", failureDetail: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] };
|
|
const remoteWorkspace = source.runnerProfileJson!.remoteWorkspace as { remoteCwd: string };
|
|
await db.update(heartbeatRuns).set({
|
|
status: "failed", finishedAt: new Date(), error: "The operation was aborted",
|
|
errorCode: "native_continuation_requires_reconciliation", completionContractId: null,
|
|
sessionIdBefore: "ses_previous_provider_session", nativeSessionId: randomUUID(), nativePhase: "terminal_failure",
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "External chat message received" },
|
|
resultJson: { stopReason: "adapter_failed", timeoutFired: false, timeoutSource: "config",
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false, effectiveTimeoutSec: 0,
|
|
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery },
|
|
runnerProfileJson: { ...source.runnerProfileJson,
|
|
nativeExecutionInput: { workspace: { cwd: remoteWorkspace.remoteCwd } } },
|
|
}).where(eq(heartbeatRuns.id, f.runId));
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
|
eventType: "lifecycle", stream: "system", level: "info", message: "run started" });
|
|
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
|
eventType: "error", stream: "system", level: "error", message: "The operation was aborted" });
|
|
await db.update(issues).set({ status: "blocked", statusVersion: 18, executionRunId: null, checkoutRunId: null })
|
|
.where(eq(issues.id, f.issueId));
|
|
// The hold exactly as e2d490c61 recorded it (resolved, replay blocked, board-owned).
|
|
await db.update(issueRecoveryActions).set({
|
|
cause: "native_continuation_requires_reconciliation", fingerprint: `native-workspace:${f.runId}`,
|
|
status: "resolved", outcome: "blocked", ownerType: "board", resolvedAt: new Date(),
|
|
nextAction: "The original sandbox is retained for workspace repair. Verify its exact stop receipt, recover the missing files, and record workspaceRepairEvidence without changing the current task. Repair does not replay the stopped run. Explicitly remove the retained allocation after recovery.",
|
|
evidence: { runId: f.runId, originalFailureCode: null, continuationDelivery: "invalidated",
|
|
workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery,
|
|
automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked",
|
|
actionOutcome: "unknown", recordedAt: new Date().toISOString() },
|
|
nativeFailureBlock: { runId: f.runId, statusVersion: 18 } },
|
|
}).where(eq(issueRecoveryActions.id, f.actionId));
|
|
await adoptSystemRecovery(db, f);
|
|
return { ...f, agentId: source.agentId };
|
|
}
|
|
|
|
async function successorsOf(runId: string) {
|
|
return db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId));
|
|
}
|
|
|
|
async function sweep() {
|
|
invalidateNativeReplacementDiscovery(db);
|
|
return reconcileSafeNativeReplacements(db);
|
|
}
|
|
|
|
describe("native run that failed before its first coordinator claim", () => {
|
|
useWorkspaceRecoveryFixture(targets);
|
|
// A published remote workspace: the shape staging persists and restores.
|
|
const priorEnv = { databaseUrl: process.env.DATABASE_URL, masterKey: process.env.PAPERCLIP_SECRETS_MASTER_KEY };
|
|
beforeEach(() => {
|
|
const local = storageBackend.getStorageService();
|
|
vi.spyOn(storageBackend, "getStorageService").mockReturnValue({ ...local, provider: "s3" });
|
|
process.env.DATABASE_URL = connectionString; process.env.PAPERCLIP_SECRETS_MASTER_KEY = "63".repeat(32);
|
|
});
|
|
afterEach(() => {
|
|
vi.restoreAllMocks();
|
|
if (priorEnv.databaseUrl === undefined) delete process.env.DATABASE_URL; else process.env.DATABASE_URL = priorEnv.databaseUrl;
|
|
if (priorEnv.masterKey === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
|
else process.env.PAPERCLIP_SECRETS_MASTER_KEY = priorEnv.masterKey;
|
|
});
|
|
|
|
it("continues automatically exactly once after its saved work is recovered", async () => {
|
|
const f = await preClaimNativeFailure();
|
|
await recoverRetainedWorkspace(f.input);
|
|
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
|
|
|
|
await Promise.all([sweep(), sweep()]);
|
|
await sweep();
|
|
|
|
const successors = await successorsOf(f.runId);
|
|
expect(successors).toHaveLength(1);
|
|
expect(successors[0]).toMatchObject({ status: "scheduled_retry", agentId: f.agentId,
|
|
scheduledRetryReason: "native_safe_replacement",
|
|
contextSnapshot: { issueId: f.issueId, retryOfRunId: f.runId, nativePreClaimContinuationOf: f.runId } });
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, successors[0].id));
|
|
expect(wake).toMatchObject({ status: "queued", requestedByActorType: "system" });
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(coordinator.failureDetail).toMatchObject({ successorRunId: successors[0].id, preClaimContinuation: true });
|
|
const hold = await action(f.actionId);
|
|
expect(hold).toMatchObject({ status: "resolved", outcome: "handed_back",
|
|
evidence: { automaticRecovery: { replay: "verified_safe_replacement", successorRunId: successors[0].id } } });
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task).toMatchObject({ status: "in_progress", executionRunId: null, checkoutRunId: null });
|
|
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
|
}, 60_000);
|
|
|
|
it("waits for saved work before continuing", async () => {
|
|
const f = await preClaimNativeFailure();
|
|
await sweep();
|
|
expect(await successorsOf(f.runId)).toHaveLength(0);
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task.status).toBe("blocked");
|
|
}, 60_000);
|
|
|
|
it("does not chain a second automatic continuation from its own successor", async () => {
|
|
const f = await preClaimNativeFailure();
|
|
await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, nativePreClaimContinuationOf: randomUUID() } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await recoverRetainedWorkspace(f.input);
|
|
await sweep();
|
|
expect(await successorsOf(f.runId)).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
// Staging 2026-11-20: a user's exact chat Retry failed before its first
|
|
// claim. Its authority is bound to its own chat action and Retry intent, so
|
|
// a system successor carrying the copied selector is always refused at
|
|
// setup. The saved hold keeps the task for the user's next Retry or message.
|
|
it("does not continue a chat Retry automatically under its consumed exact-request authority", async () => {
|
|
const f = await preClaimNativeFailure();
|
|
const current = await run(f.runId);
|
|
// The failed run the user retried. It precedes the Retry, so it does not
|
|
// supersede it.
|
|
const [failedSource] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "failed", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(current.createdAt.getTime() - 60_000) })
|
|
.returning();
|
|
const retryOfRunId = failedSource!.id;
|
|
await db.update(heartbeatRuns).set({
|
|
errorCode: "chat_failed_run_retry_not_authorized", retryOfRunId,
|
|
contextSnapshot: { ...current.contextSnapshot, retryOfRunId, forceFreshSession: true, taskRetryRequestId: randomUUID(),
|
|
chatFailedRunRetry: { version: 1, actionId: current.wakeupRequestId ?? randomUUID(), failedRunId: retryOfRunId } },
|
|
}).where(eq(heartbeatRuns.id, f.runId));
|
|
await recoverRetainedWorkspace(f.input);
|
|
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
|
|
|
|
await sweep();
|
|
|
|
expect(await successorsOf(f.runId)).toHaveLength(0);
|
|
const hold = await action(f.actionId);
|
|
expect(hold).toMatchObject({ outcome: "blocked", evidence: { automaticRecovery: { replay: "blocked" } } });
|
|
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task.status).toBe("blocked");
|
|
}, 60_000);
|
|
|
|
it.each(["claimed", "provider_event", "tool_call", "uncertain_side_effect", "process", "superseded"] as const)(
|
|
"keeps reconciliation when the run may have acted: %s", async gate => {
|
|
const f = await preClaimNativeFailure();
|
|
await recoverRetainedWorkspace(f.input);
|
|
expect(readWorkspaceRecovery((await action(f.actionId)).evidence)).toMatchObject({ status: "recovered" });
|
|
if (gate === "claimed") await db.update(nativeRunFinalizations).set({ attempt: 1, controllerGeneration: 1 })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (gate === "provider_event") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
|
|
agentId: f.agentId, eventType: "turn.started", stream: "system", level: "info", payload: {} });
|
|
if (gate === "tool_call") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
|
|
agentId: f.agentId, eventType: "tool.execution.started", stream: "system", level: "info",
|
|
payload: { name: "paperclip_create_issue", executionId: randomUUID() } });
|
|
if (gate === "uncertain_side_effect") {
|
|
const current = await run(f.runId);
|
|
await db.update(heartbeatRuns).set({ resultJson: { ...current.resultJson,
|
|
executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } }).where(eq(heartbeatRuns.id, f.runId));
|
|
}
|
|
if (gate === "process") await db.update(heartbeatRuns).set({ processPid: 2_000_000_000, processStartedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
if (gate === "superseded") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
|
status: "succeeded", contextSnapshot: { issueId: f.issueId }, createdAt: new Date(Date.now() + 1000) });
|
|
await sweep();
|
|
expect(await successorsOf(f.runId)).toHaveLength(0);
|
|
const hold = await action(f.actionId);
|
|
expect(hold).toMatchObject({ outcome: "blocked", evidence: { automaticRecovery: { replay: "blocked" } } });
|
|
const [task] = await db.select().from(issues).where(and(eq(issues.id, f.issueId)));
|
|
expect(task.status).toBe("blocked");
|
|
}, 60_000);
|
|
});
|