mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
384 lines
26 KiB
TypeScript
384 lines
26 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
||
import { projectTaskInterruption } from "./task-interruption-presentation.js";
|
||
|
||
const issue = { id: "task", companyId: "11111111-1111-4111-8111-111111111111", status: "blocked", assigneeAgentId: "agent", projectId: "22222222-2222-4222-8222-222222222222" };
|
||
const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: null,
|
||
errorCode: "project_access_denied", error: null, livenessState: null,
|
||
resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } };
|
||
const agent = { status: "idle", pauseReason: null };
|
||
|
||
describe("saved governed wait shutdown", () => {
|
||
const now = Date.parse("2026-11-09T12:00:00Z");
|
||
const pending = { ...run, runtimeMode: "native", errorCode: "native_governed_wait_settlement_pending", resultJson: null };
|
||
it("names automatic shutdown settlement without offering a duplicate Retry", () => {
|
||
expect(projectTaskInterruption(issue, { run: { ...pending, nativeResumeAt: new Date(now + 30_000) }, agent,
|
||
paused: false, approval: false }, now)).toMatchObject({ state: "working", canRetry: false,
|
||
retryWait: { reason: "governed_wait_settlement", message: "Your work is saved. Paperclip is finishing shutdown." } });
|
||
});
|
||
it("explains exhausted shutdown recovery while preserving the saved work", () => {
|
||
expect(projectTaskInterruption(issue, { run: pending, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason: "governed_wait_settlement",
|
||
message: "Paperclip couldn’t confirm shutdown. Your work is saved; continuation is paused." } });
|
||
});
|
||
it("keeps genuine approvals authoritative", () => {
|
||
expect(projectTaskInterruption(issue, { run: pending, agent, paused: false, approval: true }, now))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
||
});
|
||
});
|
||
|
||
describe("queued dependency wait presentation", () => {
|
||
const waitingRun = { ...run, status: "cancelled", runtimeMode: "legacy" as const, startedAt: null,
|
||
errorCode: "issue_dependencies_blocked", resultJson: {
|
||
stopReason: "issue_dependencies_blocked", timeoutSource: "dependency_gate",
|
||
cancellation: { source: "unknown", expected: false },
|
||
} };
|
||
|
||
it("keeps a dependency-gated wake out of the failed-attempt UI, including retained input", () => {
|
||
for (const wait of [undefined, { id: "saved-input", requestedAt: new Date(0) }]) {
|
||
expect(projectTaskInterruption(issue, { run: waitingRun, agent, wait, paused: false, approval: false }))
|
||
.toMatchObject({ state: "resolved", canRetry: false });
|
||
}
|
||
});
|
||
|
||
it.each(["legacy", "native"])("keeps an unstarted %s dispatcher dependency wait neutral", runtimeMode => {
|
||
const dispatchedWait = { ...waitingRun, runtimeMode,
|
||
resultJson: { ...waitingRun.resultJson, timeoutSource: "stale_queued_run_gate" } };
|
||
expect(projectTaskInterruption(issue, { run: dispatchedWait, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "resolved", canRetry: false });
|
||
expect(projectTaskInterruption(issue, { run: { ...dispatchedWait, startedAt: new Date() }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
});
|
||
|
||
it("keeps genuine failures and current user gates visible", () => {
|
||
for (const changed of [{ status: "failed" }, { runtimeMode: "native" }, { startedAt: new Date() },
|
||
{ errorCode: "setup_failed" }, { resultJson: {} }, { resultJson: { ...waitingRun.resultJson, timeoutFired: true } }]) {
|
||
expect(projectTaskInterruption(issue, { run: { ...waitingRun, ...changed }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
}
|
||
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: true, approval: false }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: true }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
||
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
|
||
action: { evidence: {}, status: "active" } as never })).toMatchObject({ state: "failed", canRetry: true });
|
||
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
|
||
retryWait: { id: "retry", sourceRunId: waitingRun.id, requestedAt: new Date(), preparationStatus: "failed" } }))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
});
|
||
});
|
||
|
||
describe("never-started task refused by admission", () => {
|
||
const todo = { ...issue, status: "todo" };
|
||
const admissionWait = { id: "wake", status: "skipped", reason: "budget.blocked", agentId: "agent" };
|
||
it("names an accounting hold as an automatic wait without a Retry", () => {
|
||
expect(projectTaskInterruption(todo, { agent, admissionWait, paused: false, approval: false })).toMatchObject({
|
||
responsibility: "system", state: "working", canRetry: false,
|
||
retryWait: { reason: "budget.blocked", message: expect.stringContaining("starts automatically") } });
|
||
});
|
||
it("keeps pauses and budget stops as user actions instead of promising an automatic start", () => {
|
||
for (const [state, action] of [
|
||
[{ project: { pausedAt: new Date(), pauseReason: null } }, "resume"],
|
||
[{ company: { status: "paused", pauseReason: "manual" } }, "resume"],
|
||
[{ paused: true }, "resume"],
|
||
[{ agent: { status: "paused", pauseReason: "manual" } }, "resume"],
|
||
[{ company: { status: "paused", pauseReason: "budget" } }, "budget"],
|
||
[{ approval: true }, "approval"],
|
||
] as const) {
|
||
expect(projectTaskInterruption(todo, { agent, admissionWait, paused: false, approval: false, ...state }))
|
||
.toMatchObject({ responsibility: "user", state: "waiting_for_user", userAction: action });
|
||
}
|
||
});
|
||
it("shows nothing for other refusals, started tasks or another assignee", () => {
|
||
for (const changed of [{ reason: "agent.not_invokable" }, { status: "queued" }, { agentId: "other" }]) {
|
||
expect(projectTaskInterruption(todo, { agent, admissionWait: { ...admissionWait, ...changed }, paused: false, approval: false }))
|
||
.toBeNull();
|
||
}
|
||
expect(projectTaskInterruption({ ...todo, status: "cancelled" }, { agent, admissionWait, paused: false, approval: false })).toBeNull();
|
||
});
|
||
});
|
||
|
||
describe("unadmitted chat Retry presentation", () => {
|
||
const now = Date.parse("2026-01-01T12:00:00Z");
|
||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: run.id,
|
||
preparationStatus: "issued", preparationRetryAt: now + 250 };
|
||
const input = { run: failed, agent, wait, paused: false, approval: false };
|
||
|
||
it.each(["issued", "processing"])("uses the persisted %s outbox deadline without implying provider work", status => {
|
||
for (const offset of [0, 20_000]) expect(projectTaskInterruption(issue, {
|
||
...input, wait: { ...wait, preparationStatus: status },
|
||
}, now + offset)).toMatchObject({ state: "retry_scheduled", canRetry: false,
|
||
nextRetryAt: new Date(wait.preparationRetryAt).toISOString() });
|
||
});
|
||
|
||
it.each(["failed", "cancelled"])("keeps terminal %s preparation out of the working state", status => {
|
||
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, preparationStatus: status,
|
||
preparationRetryAt: null } }, now)).toMatchObject({ state: "failed", nextRetryAt: null });
|
||
});
|
||
|
||
it("keeps current gates and admitted execution authoritative", () => {
|
||
expect(projectTaskInterruption(issue, { ...input, paused: true }, now))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||
expect(projectTaskInterruption(issue, { ...input, approval: true }, now))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
||
for (const status of ["queued", "running"]) expect(projectTaskInterruption(issue, {
|
||
...input, run: { ...failed, id: "successor", status },
|
||
}, now)).toMatchObject({ state: "working", nextRetryAt: null });
|
||
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, sourceRunId: "old-run" } }, now))
|
||
.toMatchObject({ state: "working", nextRetryAt: null });
|
||
});
|
||
});
|
||
|
||
// The native coordinator resumes the same run itself after an attach failure.
|
||
// A manual Retry offered meanwhile would race that automatic resume.
|
||
it("shows a pending native same-run resume instead of a competing Retry", () => {
|
||
const now = Date.parse("2026-11-06T15:25:50.000Z");
|
||
const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null };
|
||
const resumeAt = new Date("2026-11-06T15:26:18.687Z");
|
||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
|
||
// A due resume is being dispatched; it is still not the user's turn.
|
||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "working", canRetry: false });
|
||
// A resume that never ran, or no resume at all, leaves exactly one Retry.
|
||
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
});
|
||
|
||
it("explains a persisted Retry wait without exposing the saved payload", () => {
|
||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
|
||
executionWaitReason: "workspace_repair_required" };
|
||
const presentation = projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false });
|
||
expect(presentation).toMatchObject({ state: "failed", canRetry: true,
|
||
retryWait: { reason: "workspace_repair_required", message: "Retry is waiting for Paperclip to recover the saved files. Your message is saved." } });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, sourceRunId: "another-run" }, paused: false, approval: false }))
|
||
.not.toHaveProperty("retryWait");
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, executionWaitReason: "secret-payload" }, paused: false, approval: false }))
|
||
.toMatchObject({ retryWait: { reason: "execution_recovery", message: "Retry is waiting for execution checks. Your message is saved." } });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: true, approval: false }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||
});
|
||
|
||
it("names a budget gate instead of generic execution checks for a refused Retry", () => {
|
||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "budget.blocked" } }))
|
||
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "budget.blocked",
|
||
message: "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage." } });
|
||
});
|
||
|
||
it.each(["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed"])("does not offer Retry for terminal Retry waits: %s", reason => {
|
||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: reason } }))
|
||
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason } });
|
||
});
|
||
|
||
it("keeps Retry for a Retry declined for its requester's access, since a new click replaces it", () => {
|
||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "retry_authority_denied" } }))
|
||
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "retry_authority_denied" } });
|
||
});
|
||
|
||
it("preserves the actual workspace preparation error without changing retry authority", () => {
|
||
const failedRun = { ...run, errorCode: "workspace_validation_failed",
|
||
error: "The selected project has no repository source for a remote workspace.", resultJson: null };
|
||
expect(projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true, workspaceFailure: { message: failedRun.error } });
|
||
for (const status of ["done", "cancelled"]) expect(projectTaskInterruption({ ...issue, status },
|
||
{ run: failedRun, agent, paused: false, approval: false })).not.toHaveProperty("workspaceFailure");
|
||
expect(projectTaskInterruption(issue, { run: { ...failedRun, errorCode: "unrelated_failure" }, agent, paused: false, approval: false }))
|
||
.not.toHaveProperty("workspaceFailure");
|
||
});
|
||
|
||
function operatorStoppedRun() {
|
||
return { ...run, status: "cancelled", runtimeMode: "native" as const, companyId: issue.companyId,
|
||
nativeIssueId: issue.id, errorCode: "cancelled", resultJson: {
|
||
cancelledByActorType: "user", cancelledByUserId: "board",
|
||
executionCancellation: { state: "requested" },
|
||
nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: run.id,
|
||
companyId: issue.companyId, issueId: issue.id, scope: "run",
|
||
reasonCode: "cancellation_run_only", dispatched: true, dispatchState: "acknowledged",
|
||
intentAuditId: "intent", acknowledgementAuditId: "ack" },
|
||
} };
|
||
}
|
||
|
||
describe("intentional native Stop presentation", () => {
|
||
const workingIssue = { ...issue, status: "in_progress" };
|
||
it("does not offer a failed-attempt Retry for an audited operator Stop that keeps its task in progress", () => {
|
||
expect(projectTaskInterruption(workingIssue, { run: operatorStoppedRun(), agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "resolved", canRetry: false, userAction: null });
|
||
expect(workingIssue.status).toBe("in_progress");
|
||
});
|
||
|
||
it.each([{ runId: "foreign" }, { companyId: "foreign" }, { issueId: "foreign" },
|
||
{ dispatchState: "pending" }, { dispatched: false }, { acknowledgementAuditId: undefined },
|
||
{ intentAuditId: 123 }, { acknowledgementAuditId: {} },
|
||
{ scope: "subtree" }])("retains failure presentation without the matching operator receipt %j", change => {
|
||
const stopped = operatorStoppedRun();
|
||
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: { ...stopped.resultJson,
|
||
nativeCancellation: { ...stopped.resultJson.nativeCancellation, ...change } } }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
});
|
||
|
||
it("keeps pause, approval, non-user cancellation and new input authoritative", () => {
|
||
const stopped = operatorStoppedRun();
|
||
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: true, approval: false }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: true }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
||
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: {
|
||
...stopped.resultJson, cancelledByActorType: "system" } }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: false,
|
||
wait: { id: "new-input", requestedAt: new Date() } }))
|
||
.toMatchObject({ state: "working", canRetry: false });
|
||
});
|
||
});
|
||
|
||
it("preserves a provider explanation on a failed task without changing retry authority", () => {
|
||
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
|
||
category: "rate_limit", message: "API error: 429 This request would exceed your account's rate limit.",
|
||
httpStatus: 429, retryability: "unknown" };
|
||
const failedRun = { ...run, errorCode: "native_session_interrupted", resultJson: { providerFailure } };
|
||
const presentation = projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false });
|
||
expect(presentation).toMatchObject({ state: "failed", canRetry: true, providerFailure });
|
||
expect(projectTaskInterruption({ ...issue, status: "done" }, { run: failedRun, agent, paused: false, approval: false }))
|
||
.not.toHaveProperty("providerFailure");
|
||
});
|
||
|
||
describe("task project-access interruption", () => {
|
||
it("explains a project denial without offering a futile retry", () => {
|
||
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: false })).toMatchObject({
|
||
state: "waiting_for_user", responsibility: "user", userAction: "project_access", canRetry: false,
|
||
projectAccess: { agentId: "agent", projectId: "22222222-2222-4222-8222-222222222222", ready: false },
|
||
});
|
||
});
|
||
|
||
it("projects the historical setup denial without requiring another failed run", () => {
|
||
expect(projectTaskInterruption(issue, { run: { ...run, errorCode: "setup_failed",
|
||
error: "Project is outside this low-trust boundary.", resultJson: null }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ userAction: "project_access", canRetry: false });
|
||
});
|
||
|
||
it("offers an explicit retry after the project is added, without changing the failed run", () => {
|
||
expect(projectTaskInterruption(issue, { run, agent, projectAccessAllowed: true, paused: false, approval: false }))
|
||
.toMatchObject({ state: "failed", canRetry: true, projectAccess: { ready: true } });
|
||
expect(run.status).toBe("failed");
|
||
});
|
||
|
||
it("shows an accepted retry as working after access is repaired", () => {
|
||
const input = { run, agent, paused: false, approval: false, wait: { id: "request", requestedAt: new Date() } };
|
||
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true }))
|
||
.toMatchObject({ state: "working", canRetry: false });
|
||
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true })).not.toHaveProperty("projectAccess");
|
||
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: false }))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "project_access", canRetry: false });
|
||
});
|
||
|
||
it("does not reinterpret unrelated failures or an old assignee's denial", () => {
|
||
for (const otherRun of [{ ...run, errorCode: "setup_failed", error: "Disk full", resultJson: null },
|
||
{ ...run, agentId: "previous-agent" }]) {
|
||
expect(projectTaskInterruption(issue, { run: otherRun, agent, paused: false, approval: false }))
|
||
.not.toHaveProperty("projectAccess");
|
||
}
|
||
});
|
||
|
||
it("keeps approval, pause, running and completed states authoritative", () => {
|
||
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: true })).toMatchObject({ userAction: "approval" });
|
||
expect(projectTaskInterruption(issue, { run, agent, paused: true, approval: false })).toMatchObject({ userAction: "resume" });
|
||
expect(projectTaskInterruption(issue, { run: { ...run, status: "running" }, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "working" });
|
||
expect(projectTaskInterruption({ ...issue, status: "done" }, { run, agent, paused: false, approval: false }))
|
||
.toMatchObject({ state: "resolved" });
|
||
});
|
||
});
|
||
|
||
describe("saved-work recovery step", () => {
|
||
const now = Date.parse("2026-01-01T12:00:00Z");
|
||
const failed = { ...run, errorCode: "setup_failed", resultJson: null };
|
||
const system = (phase: string, state: string, nextAttemptAt: number | null) => ({
|
||
schema: "paperclip.system-recovery.v1", policyVersion: 1, sourceRunId: "33333333-3333-4333-8333-333333333333", revision: 3,
|
||
generationId: "44444444-4444-4444-8444-444444444444", intentId: null, phase, state, attemptId: null, attempts: 2,
|
||
retriesUsed: 1, firstFailureAt: now - 60_000, nextAttemptAt, deadlineAt: now + 600_000, providerCooldownAt: null,
|
||
failure: null, preservationOperationIds: [], freshAttemptRunId: null });
|
||
const action = (evidence: Record<string, unknown>) => ({ evidence }) as never;
|
||
|
||
it("names the step a scheduled recovery will retry", () => {
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||
action: action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) }) }, now))
|
||
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run" });
|
||
});
|
||
|
||
it("names the step a running recovery is on", () => {
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||
action: action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 }) }, now))
|
||
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
|
||
});
|
||
|
||
it("names recovery over a pending chat preparation and keeps failed preparation authoritative", () => {
|
||
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: failed.id, preparationStatus: "issued", preparationRetryAt: now + 250 };
|
||
const running = action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: running }, now))
|
||
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
|
||
const scheduled = action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: scheduled }, now))
|
||
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run", canRetry: false });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, action: scheduled }, now))
|
||
.toMatchObject({ state: "retry_scheduled", canRetry: true });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, preparationStatus: "failed" },
|
||
paused: false, approval: false, action: running }, now)).toMatchObject({ state: "failed" });
|
||
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "retry_scheduled", nextRetryAt: new Date(now + 250).toISOString() });
|
||
});
|
||
|
||
it("does not name a step for an ordinary scheduled retry", () => {
|
||
expect(projectTaskInterruption(issue, { run: { ...failed, scheduledRetryAt: new Date(now + 30_000) }, agent,
|
||
paused: false, approval: false }, now)).not.toHaveProperty("recoveryStep");
|
||
});
|
||
});
|
||
|
||
describe("scheduled resource wait presentation", () => {
|
||
const now = Date.parse("2026-01-01T12:00:00Z");
|
||
const waiting = { ...run, status: "scheduled_retry", errorCode: null, resultJson: null,
|
||
scheduledRetryReason: "workspace_busy", scheduledRetryAt: new Date(now - 60_000) };
|
||
|
||
it("keeps an overdue workspace wait out of the failed-attempt UI", () => {
|
||
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false }, now))
|
||
.toMatchObject({ state: "working", canRetry: false });
|
||
});
|
||
|
||
it("keeps failure retries, user gates and failed Retry preparation authoritative", () => {
|
||
expect(projectTaskInterruption(issue, { run: { ...waiting, scheduledRetryReason: "transient_failure" },
|
||
agent, paused: false, approval: false }, now)).toMatchObject({ state: "failed", canRetry: true });
|
||
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: true, approval: false }, now))
|
||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false,
|
||
retryWait: { id: "retry", sourceRunId: run.id, requestedAt: new Date(now), preparationStatus: "failed" } }, now))
|
||
.toMatchObject({ state: "failed", canRetry: true });
|
||
});
|
||
});
|
||
|
||
describe("saved Retry held by dependencies", () => {
|
||
const failedRun = { ...run, status: "failed", errorCode: "native_session_interrupted", resultJson: {} };
|
||
const retryWait = { id: "retry", requestedAt: new Date(0), sourceRunId: "run" };
|
||
const project = (retryDependencies: { count: number; blocked: boolean; cancelled: boolean; savingFilesOnly: boolean }) =>
|
||
projectTaskInterruption(issue, { run: failedRun, agent, retryWait, paused: false, approval: false, retryDependencies });
|
||
|
||
it.each([
|
||
[{ count: 1, blocked: true, cancelled: false, savingFilesOnly: false }, "Retry or unblock it"],
|
||
[{ count: 1, blocked: false, cancelled: true, savingFilesOnly: false }, "Remove or replace that dependency"],
|
||
[{ count: 1, blocked: false, cancelled: false, savingFilesOnly: true }, "to save its files"],
|
||
[{ count: 2, blocked: false, cancelled: false, savingFilesOnly: false }, "to finish"],
|
||
])("names what the saved Retry waits on (%o)", (dependencies, text) => {
|
||
const presentation = project(dependencies);
|
||
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: "retry",
|
||
retryWait: { reason: "dependency_blocked" } });
|
||
expect(presentation?.retryWait?.message).toContain(text);
|
||
});
|
||
});
|