mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - A control-plane restart that kills a provider turn mid-flight can lose that turn's final usage receipt. ACPX Claude reports usage only at turn end, so the receipt never becomes complete > - Accounting kept such a run "awaiting receipt" forever. Every later admission in the company then refused with "completed runs await accounting", and the budget evaluator opened a hard-stop incident that paused the company. No product action could clear it: raising the budget refuses while runs await accounting, accounting retry finds no new data, and allowing unpriced usage does not apply to pending runs > - Restart chaos waves showed this as a dead task Retry, a new task blocked as "over budget" at 22 of 500 cents, and a hard stop that cancelled queued Retries and an unrelated running task > - This pull request closes the debt from evidence: once native recovery reaches terminal failure and the provider stop is verified, the known part settles as an unpriced charge. Until then the run holds new work as recovering accounting, without a hard stop > - The benefit is that a restart no longer bricks a company with a hard budget. The operator's unpriced-usage policy decides admission, and unknown spend still never shows as zero cost ## Linked Issues or Issue Description Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the budget layer under the same journey). **What happened?** After a control-plane SIGKILL during an ACPX Claude turn, the killed run ended `native_provider_terminal_failed` with `usage_json.accountingReceiptReady = false` and `cost_accounting_pending = true`. It stayed that way for good. The task Retry was accepted but parked with `budget.blocked` ("Company cannot start work while completed runs await accounting"), which the UI showed as "Retry is waiting for execution checks". A new Codex task was escalated to `blocked` with "the original recovery target is over budget" while the company had spent 22 of 500 cents. After the next restart the budget evaluator opened a hard-stop incident (`amount_observed` 22.08 < 500), paused the company and cancelled both queued Retries. **What did you expect?** A restart should not stop the company. Unknown usage from a stopped provider should become an honest, operator-governed state, and Retry should start a run. **Steps to reproduce** 1. Give a company a hard-stop budget. 2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so the resumed turn fails. 3. Observe the run stay `cost_accounting_pending` with an incomplete receipt, and every new admission refuse. ## What Changed - `budgets.ts`: completed runs that await accounting now hold new work only. They no longer open a hard-stop incident, pause the company, place an agent budget hold, or deliver Stop to running work. Only spend at the limit, or unpriced usage under a blocking policy, does that. Chaos wave A and a pre-restart lifecycle cell showed the wedge: a run that had just succeeded stayed terminal for about 100 ms before its charge committed, the evaluator paused the company in that gap, and the Stop it delivered left a cancelled run whose receipt never closed. - `run-cost-accounting.ts`: a failed native run is a closed receipt source only with positive stop evidence: its finalization is `terminal_failure` with no result (not a finalization-deadline failure), it no longer records a process identity, and its latest server-authored process event is `native.local_process_stopped`. One SQL predicate serves the reconciler scan and settlement. Accounting now settles its known part as an `unpriced` charge instead of waiting forever. The background reconciler selects these runs. The reservation settles. Other incomplete receipts stay pending as before. A cancelled run keeps its existing accounting drain, so Stop recovery can still deliver its final receipt. - `native-local-process-stop.ts` and its transactional callers: releasing a verified local or remote process stop wakes the accounting worker, so the closed debt settles promptly. - `usage-receipts.ts`: a charge settled without its receipt is marked `accountingReceiptUnavailable`. A later complete receipt that carries its own price (including a rate-card price) corrects it. A receipt with no own price and unchanged usage does not reopen the hold. - `budget-exposure.ts`: the blocking unpriced-usage policy no longer counts a charge settled from a lost receipt with no observed usage. That is unknown, not "usage with no reliable price". Complete receipts that report no tokens (for example Cursor Cloud) still count. A live wave under the default policy hard-stopped its company at 0 of 500 cents on one such charge. - `budget-exposure.ts`: a terminal-failure native run with an incomplete, not capture-failed receipt counts as recovering accounting. Admission holds new work ("New work must wait while a native run recovers its unfinished accounting") without a hard-stop incident, a company pause or cancellation of running work. - `recovery/service.ts`: the stranded-work comment names the actual budget gate instead of always saying "over budget". - `task-interruption-presentation.ts`: a Retry parked by `budget.blocked` says it waits for the budget, not for "execution checks". - `doc/SPEC-implementation.md`: documents the closed-receipt contract next to the existing incomplete-receipt rule. ## Verification - New embedded-Postgres suite `server/src/__tests__/closed-native-receipt-accounting.test.ts`, built from the persisted shape of the killed run: - Red before the fix: the run stayed pending, `reconcileRunCosts` accounted 0, and admission refused with "completed runs await accounting". - Debt stays pending with no verified stop event, with a finalization-deadline failure, with a recorded process identity, and while the coordinator can still produce a result. - Under a blocking policy, a lost receipt with no usage does not hard-stop; one with tokens still blocks. A run with a still-recorded process identity paused the company with a hard-stop incident. - Green after: the stopped run settles as one `unpriced` charge with the reservation settled; admission is clear under an `allow` policy and names "no reliable price" under `block`; a run with a recorded process identity or a non-terminal coordinator stays pending; the closing window holds admission without an incident or pause; a final receipt that arrives later still enters late-receipt reconciliation. - Pending accounting from a run that just finished holds admission with no incident, no pause, no enforcement version bump, and an unrelated running run stays running. Red before the `budgets.ts` change: the company was paused. - A later priced receipt corrects the unavailable charge to its price; an unpriced one leaves it settled. - `task-interruption-presentation.test.ts`: the `budget.blocked` Retry wait message. - `budget-enforcement-scoping.test.ts`: the accounting-hold release cases now assert that pending accounting places no budget hold and that admission clears after settlement. - Existing suites pass locally: all budget suites (control publication, enforcement scoping, enforcement worker, evaluation work, live attempts, history, service, live usage, budget cancellation), native-stop-accounting, explicit-native-continuation, native-case-handoff, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, native-usage-delivery. Server typecheck passes. - Runner e2e cell `lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`, which failed on master with a false agent budget hold before any restart, passes on this branch. - Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy: allow`) with the accounting fix: every killed run settled (no `cost_accounting_pending` left), the company stayed active, the OpenCode Retry completed the task on the first click and the Codex Retry finished its task. That wave also exposed a 1-second false hard stop in the closing window (observed 0 of 500 cents) that cancelled the running OpenCode turn; the `budget-exposure.ts` change fixes it and has its own red/green test. - Confirming live wave with both commits, same plan (2× ACPX Claude, OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry click started a run that completed the task), OpenCode passed R1 (continued by itself), and the Codex Retry completed its task. Zero budget incidents, zero cancelled runs, zero runs left pending accounting, company active; three killed runs recorded as `unpriced` charges. The only open signature is an unrelated workspace-durability check (`R2:pre_fault_files_lost` on the Codex long-bench task). ## Risks - Behavior change: a stopped native run's lost receipt now becomes an `unpriced` ledger charge with its known usage. If that usage has tokens, the default `block` policy still stops new work with "recorded usage has no reliable price". If it has none, the charge is visible in Costs but does not stop the company. Other unpriced charges keep their existing effect. Unknown spend is never recorded as a priced receipt. Budget enforcement stays best effort, as the spec says; killed OpenCode and Codex requests already lose their in-flight usage. - Remote runs and runs without a verified local stop keep their debt pending (an admission hold only, no hard stop). - Contract change: pending accounting no longer pauses a scope or stops running work. New work still waits with "completed runs await accounting". A scope's real hard stop (spend at the limit, or blocked unpriced usage) is unchanged. - If a provider stop can never be verified, the run stays "recovering accounting": new work waits without a hard-stop incident. Before, the same state paused the company. The hold is visible in admission errors and Costs. - A receipt that arrives after settlement posts a late correction. The original unpriced charge stays in the ledger. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests, live restart-chaos harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
316 lines
21 KiB
TypeScript
316 lines
21 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { projectTaskInterruption } from "./task-interruption-presentation.js";
|
|
|
|
const issue = { id: "task", companyId: "11111111-1111-4111-8111-111111111111", status: "blocked", assigneeAgentId: "agent", projectId: "22222222-2222-4222-8222-222222222222" };
|
|
const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: null,
|
|
errorCode: "project_access_denied", error: null, livenessState: null,
|
|
resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } };
|
|
const agent = { status: "idle", pauseReason: null };
|
|
|
|
describe("queued dependency wait presentation", () => {
|
|
const waitingRun = { ...run, status: "cancelled", runtimeMode: "legacy" as const, startedAt: null,
|
|
errorCode: "issue_dependencies_blocked", resultJson: {
|
|
stopReason: "issue_dependencies_blocked", timeoutSource: "dependency_gate",
|
|
cancellation: { source: "unknown", expected: false },
|
|
} };
|
|
|
|
it("keeps a dependency-gated wake out of the failed-attempt UI, including retained input", () => {
|
|
for (const wait of [undefined, { id: "saved-input", requestedAt: new Date(0) }]) {
|
|
expect(projectTaskInterruption(issue, { run: waitingRun, agent, wait, paused: false, approval: false }))
|
|
.toMatchObject({ state: "resolved", canRetry: false });
|
|
}
|
|
});
|
|
|
|
it.each(["legacy", "native"])("keeps an unstarted %s dispatcher dependency wait neutral", runtimeMode => {
|
|
const dispatchedWait = { ...waitingRun, runtimeMode,
|
|
resultJson: { ...waitingRun.resultJson, timeoutSource: "stale_queued_run_gate" } };
|
|
expect(projectTaskInterruption(issue, { run: dispatchedWait, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "resolved", canRetry: false });
|
|
expect(projectTaskInterruption(issue, { run: { ...dispatchedWait, startedAt: new Date() }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("keeps genuine failures and current user gates visible", () => {
|
|
for (const changed of [{ status: "failed" }, { runtimeMode: "native" }, { startedAt: new Date() },
|
|
{ errorCode: "setup_failed" }, { resultJson: {} }, { resultJson: { ...waitingRun.resultJson, timeoutFired: true } }]) {
|
|
expect(projectTaskInterruption(issue, { run: { ...waitingRun, ...changed }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
}
|
|
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: true, approval: false }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
|
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: true }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
|
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
|
|
action: { evidence: {}, status: "active" } as never })).toMatchObject({ state: "failed", canRetry: true });
|
|
expect(projectTaskInterruption(issue, { run: waitingRun, agent, paused: false, approval: false,
|
|
retryWait: { id: "retry", sourceRunId: waitingRun.id, requestedAt: new Date(), preparationStatus: "failed" } }))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
});
|
|
|
|
describe("unadmitted chat Retry presentation", () => {
|
|
const now = Date.parse("2026-01-01T12:00:00Z");
|
|
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
|
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: run.id,
|
|
preparationStatus: "issued", preparationRetryAt: now + 250 };
|
|
const input = { run: failed, agent, wait, paused: false, approval: false };
|
|
|
|
it.each(["issued", "processing"])("uses the persisted %s outbox deadline without implying provider work", status => {
|
|
for (const offset of [0, 20_000]) expect(projectTaskInterruption(issue, {
|
|
...input, wait: { ...wait, preparationStatus: status },
|
|
}, now + offset)).toMatchObject({ state: "retry_scheduled", canRetry: false,
|
|
nextRetryAt: new Date(wait.preparationRetryAt).toISOString() });
|
|
});
|
|
|
|
it.each(["failed", "cancelled"])("keeps terminal %s preparation out of the working state", status => {
|
|
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, preparationStatus: status,
|
|
preparationRetryAt: null } }, now)).toMatchObject({ state: "failed", nextRetryAt: null });
|
|
});
|
|
|
|
it("keeps current gates and admitted execution authoritative", () => {
|
|
expect(projectTaskInterruption(issue, { ...input, paused: true }, now))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
|
expect(projectTaskInterruption(issue, { ...input, approval: true }, now))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
|
for (const status of ["queued", "running"]) expect(projectTaskInterruption(issue, {
|
|
...input, run: { ...failed, id: "successor", status },
|
|
}, now)).toMatchObject({ state: "working", nextRetryAt: null });
|
|
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, sourceRunId: "old-run" } }, now))
|
|
.toMatchObject({ state: "working", nextRetryAt: null });
|
|
});
|
|
});
|
|
|
|
// The native coordinator resumes the same run itself after an attach failure.
|
|
// A manual Retry offered meanwhile would race that automatic resume.
|
|
it("shows a pending native same-run resume instead of a competing Retry", () => {
|
|
const now = Date.parse("2026-11-06T15:25:50.000Z");
|
|
const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null };
|
|
const resumeAt = new Date("2026-11-06T15:26:18.687Z");
|
|
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
|
|
// A due resume is being dispatched; it is still not the user's turn.
|
|
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "working", canRetry: false });
|
|
// A resume that never ran, or no resume at all, leaves exactly one Retry.
|
|
expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("explains a persisted Retry wait without exposing the saved payload", () => {
|
|
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
|
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
|
|
executionWaitReason: "workspace_repair_required" };
|
|
const presentation = projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false });
|
|
expect(presentation).toMatchObject({ state: "failed", canRetry: true,
|
|
retryWait: { reason: "workspace_repair_required", message: "Retry is waiting for Paperclip to recover the saved files. Your message is saved." } });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, sourceRunId: "another-run" }, paused: false, approval: false }))
|
|
.not.toHaveProperty("retryWait");
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, executionWaitReason: "secret-payload" }, paused: false, approval: false }))
|
|
.toMatchObject({ retryWait: { reason: "execution_recovery", message: "Retry is waiting for execution checks. Your message is saved." } });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: true, approval: false }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
|
});
|
|
|
|
it("names a budget gate instead of generic execution checks for a refused Retry", () => {
|
|
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
|
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "budget.blocked" } }))
|
|
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "budget.blocked",
|
|
message: "Retry is waiting for the budget to allow new work. Check Costs for unrecorded or unpriced usage." } });
|
|
});
|
|
|
|
it.each(["cleanup_quarantined", "cancelled_by_operator", "retry_source_changed"])("does not offer Retry for terminal Retry waits: %s", reason => {
|
|
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
|
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: reason } }))
|
|
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason } });
|
|
});
|
|
|
|
it("keeps Retry for a Retry declined for its requester's access, since a new click replaces it", () => {
|
|
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
|
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: "retry_authority_denied" } }))
|
|
.toMatchObject({ state: "failed", canRetry: true, retryWait: { reason: "retry_authority_denied" } });
|
|
});
|
|
|
|
it("preserves the actual workspace preparation error without changing retry authority", () => {
|
|
const failedRun = { ...run, errorCode: "workspace_validation_failed",
|
|
error: "The selected project has no repository source for a remote workspace.", resultJson: null };
|
|
expect(projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true, workspaceFailure: { message: failedRun.error } });
|
|
for (const status of ["done", "cancelled"]) expect(projectTaskInterruption({ ...issue, status },
|
|
{ run: failedRun, agent, paused: false, approval: false })).not.toHaveProperty("workspaceFailure");
|
|
expect(projectTaskInterruption(issue, { run: { ...failedRun, errorCode: "unrelated_failure" }, agent, paused: false, approval: false }))
|
|
.not.toHaveProperty("workspaceFailure");
|
|
});
|
|
|
|
function operatorStoppedRun() {
|
|
return { ...run, status: "cancelled", runtimeMode: "native" as const, companyId: issue.companyId,
|
|
nativeIssueId: issue.id, errorCode: "cancelled", resultJson: {
|
|
cancelledByActorType: "user", cancelledByUserId: "board",
|
|
executionCancellation: { state: "requested" },
|
|
nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: run.id,
|
|
companyId: issue.companyId, issueId: issue.id, scope: "run",
|
|
reasonCode: "cancellation_run_only", dispatched: true, dispatchState: "acknowledged",
|
|
intentAuditId: "intent", acknowledgementAuditId: "ack" },
|
|
} };
|
|
}
|
|
|
|
describe("intentional native Stop presentation", () => {
|
|
const workingIssue = { ...issue, status: "in_progress" };
|
|
it("does not offer a failed-attempt Retry for an audited operator Stop that keeps its task in progress", () => {
|
|
expect(projectTaskInterruption(workingIssue, { run: operatorStoppedRun(), agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "resolved", canRetry: false, userAction: null });
|
|
expect(workingIssue.status).toBe("in_progress");
|
|
});
|
|
|
|
it.each([{ runId: "foreign" }, { companyId: "foreign" }, { issueId: "foreign" },
|
|
{ dispatchState: "pending" }, { dispatched: false }, { acknowledgementAuditId: undefined },
|
|
{ intentAuditId: 123 }, { acknowledgementAuditId: {} },
|
|
{ scope: "subtree" }])("retains failure presentation without the matching operator receipt %j", change => {
|
|
const stopped = operatorStoppedRun();
|
|
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: { ...stopped.resultJson,
|
|
nativeCancellation: { ...stopped.resultJson.nativeCancellation, ...change } } }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("keeps pause, approval, non-user cancellation and new input authoritative", () => {
|
|
const stopped = operatorStoppedRun();
|
|
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: true, approval: false }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
|
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: true }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
|
|
expect(projectTaskInterruption(workingIssue, { run: { ...stopped, resultJson: {
|
|
...stopped.resultJson, cancelledByActorType: "system" } }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
expect(projectTaskInterruption(workingIssue, { run: stopped, agent, paused: false, approval: false,
|
|
wait: { id: "new-input", requestedAt: new Date() } }))
|
|
.toMatchObject({ state: "working", canRetry: false });
|
|
});
|
|
});
|
|
|
|
it("preserves a provider explanation on a failed task without changing retry authority", () => {
|
|
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
|
|
category: "rate_limit", message: "API error: 429 This request would exceed your account's rate limit.",
|
|
httpStatus: 429, retryability: "unknown" };
|
|
const failedRun = { ...run, errorCode: "native_session_interrupted", resultJson: { providerFailure } };
|
|
const presentation = projectTaskInterruption(issue, { run: failedRun, agent, paused: false, approval: false });
|
|
expect(presentation).toMatchObject({ state: "failed", canRetry: true, providerFailure });
|
|
expect(projectTaskInterruption({ ...issue, status: "done" }, { run: failedRun, agent, paused: false, approval: false }))
|
|
.not.toHaveProperty("providerFailure");
|
|
});
|
|
|
|
describe("task project-access interruption", () => {
|
|
it("explains a project denial without offering a futile retry", () => {
|
|
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: false })).toMatchObject({
|
|
state: "waiting_for_user", responsibility: "user", userAction: "project_access", canRetry: false,
|
|
projectAccess: { agentId: "agent", projectId: "22222222-2222-4222-8222-222222222222", ready: false },
|
|
});
|
|
});
|
|
|
|
it("projects the historical setup denial without requiring another failed run", () => {
|
|
expect(projectTaskInterruption(issue, { run: { ...run, errorCode: "setup_failed",
|
|
error: "Project is outside this low-trust boundary.", resultJson: null }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ userAction: "project_access", canRetry: false });
|
|
});
|
|
|
|
it("offers an explicit retry after the project is added, without changing the failed run", () => {
|
|
expect(projectTaskInterruption(issue, { run, agent, projectAccessAllowed: true, paused: false, approval: false }))
|
|
.toMatchObject({ state: "failed", canRetry: true, projectAccess: { ready: true } });
|
|
expect(run.status).toBe("failed");
|
|
});
|
|
|
|
it("shows an accepted retry as working after access is repaired", () => {
|
|
const input = { run, agent, paused: false, approval: false, wait: { id: "request", requestedAt: new Date() } };
|
|
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true }))
|
|
.toMatchObject({ state: "working", canRetry: false });
|
|
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: true })).not.toHaveProperty("projectAccess");
|
|
expect(projectTaskInterruption(issue, { ...input, projectAccessAllowed: false }))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "project_access", canRetry: false });
|
|
});
|
|
|
|
it("does not reinterpret unrelated failures or an old assignee's denial", () => {
|
|
for (const otherRun of [{ ...run, errorCode: "setup_failed", error: "Disk full", resultJson: null },
|
|
{ ...run, agentId: "previous-agent" }]) {
|
|
expect(projectTaskInterruption(issue, { run: otherRun, agent, paused: false, approval: false }))
|
|
.not.toHaveProperty("projectAccess");
|
|
}
|
|
});
|
|
|
|
it("keeps approval, pause, running and completed states authoritative", () => {
|
|
expect(projectTaskInterruption(issue, { run, agent, paused: false, approval: true })).toMatchObject({ userAction: "approval" });
|
|
expect(projectTaskInterruption(issue, { run, agent, paused: true, approval: false })).toMatchObject({ userAction: "resume" });
|
|
expect(projectTaskInterruption(issue, { run: { ...run, status: "running" }, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "working" });
|
|
expect(projectTaskInterruption({ ...issue, status: "done" }, { run, agent, paused: false, approval: false }))
|
|
.toMatchObject({ state: "resolved" });
|
|
});
|
|
});
|
|
|
|
describe("saved-work recovery step", () => {
|
|
const now = Date.parse("2026-01-01T12:00:00Z");
|
|
const failed = { ...run, errorCode: "setup_failed", resultJson: null };
|
|
const system = (phase: string, state: string, nextAttemptAt: number | null) => ({
|
|
schema: "paperclip.system-recovery.v1", policyVersion: 1, sourceRunId: "33333333-3333-4333-8333-333333333333", revision: 3,
|
|
generationId: "44444444-4444-4444-8444-444444444444", intentId: null, phase, state, attemptId: null, attempts: 2,
|
|
retriesUsed: 1, firstFailureAt: now - 60_000, nextAttemptAt, deadlineAt: now + 600_000, providerCooldownAt: null,
|
|
failure: null, preservationOperationIds: [], freshAttemptRunId: null });
|
|
const action = (evidence: Record<string, unknown>) => ({ evidence }) as never;
|
|
|
|
it("names the step a scheduled recovery will retry", () => {
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
|
action: action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) }) }, now))
|
|
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run" });
|
|
});
|
|
|
|
it("names the step a running recovery is on", () => {
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
|
action: action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 }) }, now))
|
|
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
|
|
});
|
|
|
|
it("names recovery over a pending chat preparation and keeps failed preparation authoritative", () => {
|
|
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: failed.id, preparationStatus: "issued", preparationRetryAt: now + 250 };
|
|
const running = action({ systemRecovery: system("preserve_files", "running", null), systemRecoveryActiveAt: now - 1_000 });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: running }, now))
|
|
.toMatchObject({ state: "working", recoveryStep: "saving_previous_files" });
|
|
const scheduled = action({ systemRecovery: system("verify_execution", "scheduled", now + 30_000) });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false, action: scheduled }, now))
|
|
.toMatchObject({ state: "retry_scheduled", recoveryStep: "checking_previous_run", canRetry: false });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false, action: scheduled }, now))
|
|
.toMatchObject({ state: "retry_scheduled", canRetry: true });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, preparationStatus: "failed" },
|
|
paused: false, approval: false, action: running }, now)).toMatchObject({ state: "failed" });
|
|
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "retry_scheduled", nextRetryAt: new Date(now + 250).toISOString() });
|
|
});
|
|
|
|
it("does not name a step for an ordinary scheduled retry", () => {
|
|
expect(projectTaskInterruption(issue, { run: { ...failed, scheduledRetryAt: new Date(now + 30_000) }, agent,
|
|
paused: false, approval: false }, now)).not.toHaveProperty("recoveryStep");
|
|
});
|
|
});
|
|
|
|
describe("scheduled resource wait presentation", () => {
|
|
const now = Date.parse("2026-01-01T12:00:00Z");
|
|
const waiting = { ...run, status: "scheduled_retry", errorCode: null, resultJson: null,
|
|
scheduledRetryReason: "workspace_busy", scheduledRetryAt: new Date(now - 60_000) };
|
|
|
|
it("keeps an overdue workspace wait out of the failed-attempt UI", () => {
|
|
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false }, now))
|
|
.toMatchObject({ state: "working", canRetry: false });
|
|
});
|
|
|
|
it("keeps failure retries, user gates and failed Retry preparation authoritative", () => {
|
|
expect(projectTaskInterruption(issue, { run: { ...waiting, scheduledRetryReason: "transient_failure" },
|
|
agent, paused: false, approval: false }, now)).toMatchObject({ state: "failed", canRetry: true });
|
|
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: true, approval: false }, now))
|
|
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
|
expect(projectTaskInterruption(issue, { run: waiting, agent, paused: false, approval: false,
|
|
retryWait: { id: "retry", sourceRunId: run.id, requestedAt: new Date(now), preparationStatus: "failed" } }, now))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
});
|