mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip manages agents and the tasks they work on. > - Recovery should continue unfinished work when it is eligible. > - An active unanswered question is a persisted reason to wait for a person. > - Treating a later Retry as an answer makes the recovery sweep repeatedly request continuations that admission refuses. > - Provider-limit holds also produce one cancellation toast per task, even though provider work never started. > - This PR aligns recovery with persisted interaction status and presents these waits accurately. ## Linked Issues or Issue Description Refs: #20466. This addresses the question blocker presentation and automatic recovery loop; it preserves the existing interaction-resolution policy. Related work: #19420 schedules provider quota retries, and #21570 handles notices and reply ownership in agent chats. This change concerns assigned-task reconciliation, task/inbox status, and per-run cancellation toasts. **What happened?** The assigned-work sweep ignored pending interactions whenever a newer run carried `explicitUserContinuation`. Continuation admission still refused the pending question, so repeated sweeps recorded unsuccessful continuation attempts. The task could meanwhile show an older provider error as its current blocker. Separately, each pre-dispatch provider-limit hold generated a cancellation toast, and the one-hour fallback was described as a known reset and promised start time. **Expected behavior** Wait quietly while the question remains active under the shared interaction eligibility rules, then react to the existing resolution notification. Dismissed or historical ordinary questions remain answerable without blocking work; governed requests keep their gates. Show the current question on task and inbox surfaces. Keep provider failure history, suppress cancellation toasts for expected provider holds, and describe scheduled availability checks without promising the provider will accept work. **Steps to reproduce** 1. Leave an `ask_user_questions` interaction pending with `supersedeOnUserComment: false`. 2. Create a later failed/interrupted explicit Retry and run assigned-work reconciliation repeatedly. 3. Observe that the question remains pending while the sweep attempts fresh continuations. 4. Separately, let an agent with recent provider rate-limit evidence receive automatic task wakes without a reported reset time. ## What Changed - Remove inferred question supersession from both assigned-work fact loading and candidate selection. Use the existing active-interaction predicate consistently in those reads, fresh-continuation admission, and the task projection. Cover dismissed/historical ordinary questions, retained governed gates, repeated checks, idle workers, and the existing resolution notification. - Add the `question` interruption action, load active pending question IDs alongside approvals in one company-scoped query, and show that blocker on tasks and inbox rows. Preserve a supported manual Retry after a failed attempt, suppress duplicate actions for saved input, and keep other current user gates authoritative. This preserves the existing issue-detail SQL statement budget. Historical run failures remain intact. Refresh the shared source SDK input pin. - Suppress cancellation toasts specifically for cancelled `provider_limit_wait` runs. Label provider retry deadlines as availability rechecks, retain relative-time formatting, and record successful scheduling as an informational lifecycle event. - Document the wait contract in the continuation architecture guide. ## Verification - Demonstrated failing regressions before the implementation: continuation loop, task projection, task notice, cancellation toast, provider deadline wording, and deadline formatting. - Initial focused Vitest qualification: 642 tests passed across 14 files, using: ```sh pnpm exec vitest run \ server/src/__tests__/{assigned-work-invariant,assigned-work-recheck-idle,heartbeat-provider-limit-wait,task-interruption-query}.test.ts \ server/src/services/{task-interruption-presentation,execution-projection,fresh-continuation-fallback,issue-thread-interactions}.test.ts \ server/src/services/recovery/assigned-work-invariant.test.ts \ ui/src/{components/ExecutionBlockerNotice.test.tsx,context/LiveUpdatesProvider.test.ts,context/LiveUpdatesProvider.hook.test.tsx,lib/provider-limit-message.test.ts,pages/Inbox.test.tsx} ``` - `pnpm --filter @paperclipai/shared typecheck` — passed. - Review follow-up: 157 tests passed in `assigned-work-invariant`, `task-interruption-query`, and `fresh-continuation-fallback`; 575 passed in `task-interruption-query`, `assigned-work-recheck-idle`, `task-interruption-presentation`, `issue-question-context`, `issue-thread-interactions`, and `explicit-native-continuation` (the query file overlaps). The strengthened final dismissed/historical/governed projection regression also passed separately. - Final query-consolidation verification: 157 tests passed across five files, including the unchanged 64-statement issue-detail census and mixed approval/question batches: ```sh pnpm exec vitest run \ server/src/__tests__/{issue-detail-statements,task-interruption-query,assigned-work-invariant,assigned-work-recheck-idle}.test.ts \ server/src/services/task-interruption-presentation.test.ts ``` - `pnpm --filter @paperclipai/server typecheck` — passed, including test layers. - Manual-recovery follow-up: 204 tests passed across seven projection/recovery/notice suites, and all 104 Inbox tests passed. The lost-runner integration still starts exactly one run from a manual Retry and leaves the question unanswered. The saved-decision census verifies one decision batch query for a 41-task list, with at most three statements referencing interactions; the overall 64-statement issue-detail limit remains unchanged. - `pnpm --filter @paperclipai/ui typecheck` — passed. - `pnpm --filter @paperclipai/ui build` — passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` — passed. - `pnpm verify` — exit 2: broader CI is required for changed global inputs. Focused tests and package checks supply local evidence; broad CI remains required. - `git diff --check` and a local secret/instance-data scan — passed. - Final CI on `1c16fe7c0d8`: [trusted CI workflow](https://github.com/paperclipai/paperclip/actions/runs/44650155842) passed, including all server shards, native-server verification, build, canary, e2e, release registry, and serialized suites. All PR checks are complete, with no failures; the branch is conflict-free. - Greptile reviewed the latest revision, `1c16fe7c0d8`, at 5/5 with no actionable findings or unresolved review threads. The focused tests cover genuine provider failures, live work and terminal tasks, unchanged task locks/retry delivery, preservation of historical failures, and question resolution waking the affected task while other active pending questions remain quiet. Review regressions also cover dismissed/historical ordinary questions retaining Retry and governed requests retaining their gates. No live task was retried or answered for validation. ## Risks - An active pending question now consistently blocks automatic assigned-work recovery. Shared interaction eligibility determines when ordinary conversation input becomes deferred or historical; governed requests retain their authority. Automatic recovery no longer infers that a newer Retry resolved an active question. - Manual Retry keeps its existing admission policy and recovery checks. The question notice preserves that supported action after a failed attempt without changing approval or credential authority; pending saved input prevents duplicate actions. - The shared interruption action gains `question`; the task and inbox consumers are updated together. The pending-question query returns only task IDs and uses the existing company/task/status index. - Provider cooldown policy, quotas, credentials, routing, cancellation history, and durable retry timing are unchanged. The actual provider can still return a rate limit at the recheck. - No migration is required. Existing large owner modules receive small localized changes; no code file crosses 1,000 lines. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, xhigh reasoning effort, with code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
392 lines
20 KiB
TypeScript
392 lines
20 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { projectExecution } from "./execution-projection.js";
|
|
import { startTaskDrain, stopTaskDrain } from "./task-admission.js";
|
|
import { resetStartupRecoveryStateForTests, setStartupRecoveryPhase } from "../startup-recovery-state.js";
|
|
|
|
type Run = Parameters<typeof projectExecution>[0];
|
|
type Coordinator = NonNullable<Parameters<typeof projectExecution>[1]>;
|
|
const now = new Date("2026-09-08T12:00:00Z");
|
|
const run = (values: Partial<Run> = {}) =>
|
|
({
|
|
status: "running",
|
|
runtimeMode: "native",
|
|
errorCode: null,
|
|
nextAction: null,
|
|
lastUsefulActionAt: now,
|
|
lastOutputAt: null,
|
|
startedAt: now,
|
|
scheduledRetryAt: null,
|
|
scheduledRetryAttempt: 1,
|
|
retryOfRunId: null,
|
|
executionControlDeadlineAt: null,
|
|
...values,
|
|
}) as Run;
|
|
const coordinator = (values: Partial<Coordinator> = {}) =>
|
|
({
|
|
phase: "observed",
|
|
attempt: 1,
|
|
failureCode: null,
|
|
failureDetail: {},
|
|
nextAttemptAt: null,
|
|
leaseExpiresAt: new Date(now.getTime() + 30_000),
|
|
controlDeadlineAt: null,
|
|
...values,
|
|
}) as Coordinator;
|
|
const project = (
|
|
r = run(),
|
|
c: Coordinator | undefined = coordinator(),
|
|
pending: { kind: string }[] = [],
|
|
) => projectExecution(r, c, pending, undefined, now);
|
|
|
|
describe("execution truth projection", () => {
|
|
it.each(["requested", "acknowledged"])("shows a %s operator Stop without calling it a failed attempt", state => {
|
|
const stopped = run({ status: "cancelled", runtimeMode: "legacy", finishedAt: now, errorCode: "cancelled",
|
|
resultJson: { cancellation: { source: "operator", expected: true, initiator: { type: "user", id: "board" },
|
|
reason: "Cancelled by a board operator", recordedAt: now.toISOString() }, executionCancellation: { state, proof: "preparation_settled" } } });
|
|
if (state === "acknowledged") {
|
|
expect(project(stopped, undefined)).toMatchObject({ phase: "completed", label: "Stopped", cause: null,
|
|
nextAction: "Send a new message to continue from saved work." });
|
|
} else {
|
|
expect(project(stopped, undefined).phase).not.toBe("completed");
|
|
}
|
|
// Unrelated provider failures and recovery holds remain visible.
|
|
expect(project(run({ ...stopped, resultJson: {} }), undefined).phase).toBe("failed");
|
|
expect(projectExecution(stopped, undefined, [], { cause: "workspace_restore_failed", nextAction: "Save files" }, now).phase)
|
|
.toBe("recovery_needed");
|
|
});
|
|
it.each([undefined, "unknown", "provider_api_stop_receipt"])("retains failure presentation for an unqualified legacy Stop proof: %s", proof => {
|
|
const stopped = run({ status: "cancelled", runtimeMode: "legacy", finishedAt: now,
|
|
resultJson: { cancellation: { source: "operator", expected: true, initiator: { type: "user" },
|
|
reason: "Stop", recordedAt: now.toISOString() }, executionCancellation: { state: "acknowledged", proof } } });
|
|
expect(project(stopped, undefined).phase).toBe("failed");
|
|
});
|
|
it.each(["contained", "ordinary", "wrong-source", "continued"] as const)("projects controller containment without inventing queued work: %s", state => {
|
|
const stopped = run({ id: "old", status: "failed" });
|
|
const action = { status: "resolved" as const, cause: "native_restart_recovery_blocked", nextAction: "Inspect the original owner",
|
|
evidence: { reason: "controller_process_space_unverified", continuationDelivery: "not_requested",
|
|
executionReconciliation: { runId: state === "wrong-source" ? "another-run" : "old", providerStopped: true,
|
|
...(state === "ordinary" ? {} : { controllerStopped: true, controllerAuthorityDigest: "a".repeat(64) }) },
|
|
...(state === "continued" ? { explicitUserContinuation: { previousRunId: "old", runId: "new" } } : {}),
|
|
} };
|
|
const projected = projectExecution(stopped, coordinator({ phase: "terminal_failure" }), [], action, now);
|
|
expect(projected.phase).toBe(state === "contained" ? "failed" : state === "continued" ? "completed" : "queued");
|
|
if (state === "contained") {
|
|
expect(projected.label).toBe("Execution contained");
|
|
expect(projected.nextAction).toContain("Use Retry");
|
|
expect(projected.successorRunId).toBeNull();
|
|
}
|
|
if (state === "continued") expect(projected.successorRunId).toBe("new");
|
|
});
|
|
|
|
it("offers Stop for every live run, including a long-silent native turn", () => {
|
|
const silentSince = new Date(now.getTime() - 35 * 60_000);
|
|
const silent = run({ lastUsefulActionAt: silentSince, startedAt: silentSince });
|
|
expect(project(silent)).toMatchObject({ phase: "working", canStop: true });
|
|
expect(project(run({ runtimeMode: "legacy" }), undefined).canStop).toBe(true);
|
|
expect(projectExecution(run({ status: "queued", startedAt: null }), undefined, [], undefined, now).canStop).toBe(true);
|
|
for (const status of ["succeeded", "cancelled", "timed_out", "failed"] as const) {
|
|
expect(project(run({ status }), undefined).canStop).toBe(false);
|
|
}
|
|
});
|
|
it("shows setup progress without overriding real holds or started work", () => {
|
|
const queued = run({ status: "queued", startedAt: null });
|
|
const setup = { lifecycleState: "verifying", lifecycleHolds: [], lifecycleError: null };
|
|
const projectSetup = (r = queued, agent = setup) => projectExecution(r, undefined, [], undefined, now, undefined, false, agent);
|
|
expect(projectSetup()).toMatchObject({ label: "Waiting for agent setup", cause: "agent_setup_pending" });
|
|
expect(projectSetup(queued, { ...setup, lifecycleState: "paused" })).toMatchObject({ label: "Queued", cause: null });
|
|
expect(projectExecution(queued, undefined, [], undefined, now, undefined, false,
|
|
{ ...setup, lifecycleHolds: ["budget"] })).toMatchObject({ label: "Queued", cause: null });
|
|
expect(projectSetup(run({ status: "queued", startedAt: now }))).toMatchObject({ label: "Queued", cause: null });
|
|
expect(projectSetup(run({ status: "running" })).label).toBe("Confirming execution");
|
|
});
|
|
|
|
it("shows maintenance only for admission that has not started", () => {
|
|
startTaskDrain();
|
|
try {
|
|
expect(projectExecution(run({ status: "queued", startedAt: null }), undefined, [], undefined, now))
|
|
.toMatchObject({ phase: "queued", label: "Waiting for maintenance", cause: "task_drain" });
|
|
expect(project().phase).toBe("working");
|
|
} finally { stopTaskDrain(); }
|
|
expect(projectExecution(run({ status: "queued", startedAt: null }), undefined, [], undefined, now))
|
|
.toMatchObject({ phase: "queued", label: "Queued", cause: null });
|
|
});
|
|
it.each(["starting", "recovering", "draining"] as const)("names a server restart while it holds queued work: %s", phase => {
|
|
setStartupRecoveryPhase(phase);
|
|
try {
|
|
for (const queued of [run({ status: "queued", startedAt: null }), run({ status: "queued", startedAt: now, retryOfRunId: "prior" })]) {
|
|
const projection = projectExecution(queued, undefined, [], undefined, now);
|
|
expect(projection).toMatchObject({ phase: "queued", label: "Waiting for server restart", cause: "server_restarting" });
|
|
expect(projection.nextAction).toMatch(/starts automatically/);
|
|
}
|
|
// A live run is not relabelled; restart recovery owns its own state.
|
|
expect(project().phase).toBe("working");
|
|
} finally { resetStartupRecoveryStateForTests(); }
|
|
expect(projectExecution(run({ status: "queued", startedAt: null }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Queued", cause: null });
|
|
});
|
|
it("keeps the maintenance hold reason while a restarted server is still recovering", () => {
|
|
setStartupRecoveryPhase("recovering");
|
|
startTaskDrain();
|
|
try {
|
|
expect(projectExecution(run({ status: "queued", startedAt: null }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Waiting for maintenance", cause: "task_drain" });
|
|
} finally { stopTaskDrain(); resetStartupRecoveryStateForTests(); }
|
|
});
|
|
it.each(["queued", "recovering", "blocked", "recovered"] as const)("shows the exact saved workspace recovery state instead of stale repair instructions: %s", status => {
|
|
const sourceRunId = "ec036e06-90a3-4b69-81db-a8eab7dbfbac";
|
|
const source = run({ id: sourceRunId, status: "failed", errorCode: "native_continuation_requires_reconciliation" });
|
|
const action = {
|
|
status: "resolved" as const,
|
|
cause: "native_continuation_requires_reconciliation",
|
|
nextAction: "The original sandbox is retained for workspace repair.",
|
|
evidence: {
|
|
runId: sourceRunId,
|
|
automaticRecovery: { policy: "preserve_without_replay_v1", actionOutcome: "unknown", replay: "blocked" },
|
|
continuationDelivery: "invalidated",
|
|
workspaceRecovery: {
|
|
schema: "paperclip.workspace-recovery.v1", runId: sourceRunId, status,
|
|
stage: "saving_receipts", attempt: 1, requestIds: [],
|
|
...(status === "blocked" ? { message: "The saved source cannot be read safely." } : {}),
|
|
},
|
|
},
|
|
};
|
|
const projected = projectExecution(source, coordinator({ phase: "terminal_failure" }), [], action, now);
|
|
expect(projected).toMatchObject({
|
|
phase: "recovery_needed", label: "Stopped", cause: action.cause,
|
|
nextAction: status === "recovered" ? "Saved work recovered."
|
|
: status === "blocked" ? "The saved source cannot be read safely." : "Recovering saved work…",
|
|
permittedActions: ["inspect_run"], successorRunId: null,
|
|
});
|
|
expect(source.status).toBe("failed");
|
|
expect(action.evidence.automaticRecovery.replay).toBe("blocked");
|
|
if (status === "recovered") {
|
|
expect(projectExecution({ ...source, status: "timed_out" }, coordinator({ phase: "terminal_failure" }), [], action, now)
|
|
.nextAction).toBe("Saved work recovered.");
|
|
expect(projectExecution({ ...source, errorCode: "native_session_cleanup_quarantined" },
|
|
coordinator({ phase: "terminal_failure" }), [], action, now).nextAction).toBe("Saved work recovered.");
|
|
}
|
|
expect(projectExecution(source, undefined, [], { ...action,
|
|
evidence: { ...action.evidence, workspaceRecovery: { ...action.evidence.workspaceRecovery,
|
|
runId: "5f227d75-b314-4511-bf23-9c615277e69b" } },
|
|
}, now).nextAction).toBe(action.nextAction);
|
|
});
|
|
|
|
it("keeps cleanup quarantine visible without a coordinator or recovery row", () => {
|
|
const stopped = run({ id: "stopped", status: "failed", errorCode: "native_session_cleanup_quarantined" });
|
|
expect(projectExecution(stopped, undefined, [], undefined, now)).toMatchObject({
|
|
phase: "recovery_needed", cause: "native_session_cleanup_quarantined", recoveryOwner: "board",
|
|
permittedActions: ["inspect_run", "inspect_recovery"],
|
|
});
|
|
expect(projectExecution(stopped, coordinator({ phase: "retryable_failure" }), [], undefined, now).phase).toBe("recovery_needed");
|
|
expect(projectExecution({ ...stopped, finishedAt: now }, coordinator({
|
|
phase: "terminal_failure", failureCode: "native_provider_terminal_failed",
|
|
}), [], undefined, now)).toMatchObject({ phase: "recovery_needed", recoveryOwner: "board" });
|
|
expect(projectExecution({ ...stopped, errorCode: "provider_frame_too_large" }, undefined, [], undefined, now).phase).toBe("failed");
|
|
expect(projectExecution(stopped, undefined, [], { status: "resolved", cause: "native_session_cleanup_quarantined", nextAction: "Verify prior execution",
|
|
evidence: { executionReconciliation: { providerStopped: true }, continuationRunId: "successor" } }, now))
|
|
.toMatchObject({ phase: "completed", successorRunId: "successor", cause: null });
|
|
});
|
|
it("shows subscription contention as a resource wait without failed provider attempts", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy",
|
|
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeAiConnectionWait: 0 } }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Waiting for AI subscription", phase: "retry_scheduled", attempt: 1, recoveryOwner: null });
|
|
});
|
|
it("shows a provider usage limit recheck without promising work will start then", () => {
|
|
const scheduledRetryAt = new Date("2026-11-05T13:47:50.375Z");
|
|
expect(projectExecution(run({ runtimeMode: "native", status: "scheduled_retry", scheduledRetryReason: "ai_connection_pool_wait",
|
|
scheduledRetryAt, scheduledRetryAttempt: 3, contextSnapshot: { wakeReason: "provider_limit_retry", failureRetriesBeforeAiConnectionWait: 0 } }),
|
|
undefined, [], undefined, now)).toMatchObject({
|
|
label: "Waiting for provider usage", phase: "retry_scheduled", attempt: 1, recoveryOwner: null,
|
|
nextAction: expect.stringContaining(scheduledRetryAt.toISOString()),
|
|
});
|
|
});
|
|
it("shows a workspace wait without presenting its deferral count as failed attempts", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "workspace_busy",
|
|
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } }), undefined, [], undefined, now))
|
|
.toMatchObject({ label: "Waiting for workspace", phase: "retry_scheduled", attempt: 2, recoveryOwner: null });
|
|
});
|
|
it.each([
|
|
{ status: "resolved", previousRunId: "old", nextRunId: "next", continued: true },
|
|
{ status: "active", previousRunId: "old", nextRunId: "next", continued: false },
|
|
{ status: "resolved", previousRunId: "other", nextRunId: "next", continued: false },
|
|
{ status: "resolved", previousRunId: "old", nextRunId: "old", continued: false },
|
|
{ status: "resolved", previousRunId: "old", nextRunId: null, continued: false },
|
|
])("projects the recorded explicit successor without hiding unresolved recovery: $status/$previousRunId/$nextRunId", ({ status, previousRunId, nextRunId, continued }) => {
|
|
const source = run({ id: "old", status: "failed", errorCode: "adapter_failed" });
|
|
expect(projectExecution(source, coordinator({ phase: "terminal_failure" }), [], {
|
|
status,
|
|
cause: "native_continuation_requires_reconciliation",
|
|
nextAction: "Inspect the stopped execution.",
|
|
evidence: {
|
|
automaticRecovery: { policy: "preserve_without_replay_v1" },
|
|
explicitUserContinuation: { previousRunId, runId: nextRunId },
|
|
},
|
|
}, now)).toMatchObject(continued ? {
|
|
phase: "completed", label: "Continued in another run", successorRunId: "next",
|
|
cause: "native_continuation_requires_reconciliation", nextAction: null,
|
|
} : { phase: "recovery_needed", successorRunId: null });
|
|
expect(source.status).toBe("failed");
|
|
});
|
|
|
|
it("shows a reconciled continuation as queued until its durable delivery is recorded", () => {
|
|
const action = {
|
|
cause: "native_session_retry_exhausted",
|
|
nextAction: "Old recovery action",
|
|
status: "resolved" as const,
|
|
evidence: {
|
|
executionReconciliation: { runId: "old" },
|
|
continuationDelivery: "pending",
|
|
},
|
|
};
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
action,
|
|
now,
|
|
),
|
|
).toMatchObject({
|
|
phase: "queued",
|
|
label: "Continuation queued",
|
|
cause: null,
|
|
});
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
{
|
|
...action,
|
|
evidence: { ...action.evidence, continuationRunId: "next" },
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({ phase: "completed", successorRunId: "next" });
|
|
expect(
|
|
projectExecution(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "terminal_failure" }),
|
|
[],
|
|
{
|
|
...action,
|
|
evidence: { ...action.evidence, continuationDelivery: "invalidated" },
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({ phase: "failed", label: "Continuation cancelled" });
|
|
});
|
|
it("requires current execution evidence, without treating output silence as failure", () => {
|
|
expect(project(run({ lastOutputAt: new Date(0) })).phase).toBe("working");
|
|
expect(
|
|
project(run(), coordinator({ leaseExpiresAt: new Date(0) })),
|
|
).toMatchObject({ phase: "reconnecting", label: "Confirming execution" });
|
|
expect(
|
|
project(
|
|
run({ runtimeMode: "legacy", processPid: process.pid }),
|
|
undefined,
|
|
).phase,
|
|
).toBe("working");
|
|
});
|
|
it("distinguishes provider work, finalization and timed retry", () => {
|
|
expect(
|
|
project(
|
|
run({ executionControlDeadlineAt: new Date(now.getTime() + 60_000) }),
|
|
).phase,
|
|
).toBe("finishing");
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({
|
|
phase: "retryable_failure",
|
|
nextAttemptAt: new Date(now.getTime() + 30_000),
|
|
attempt: 2,
|
|
}),
|
|
),
|
|
).toMatchObject({ phase: "retry_scheduled", attempt: 2, maxAttempts: 3 });
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({ phase: "retryable_failure", nextAttemptAt: new Date(0) }),
|
|
).phase,
|
|
).toBe("reconnecting");
|
|
});
|
|
it("includes the original legacy attempt in the displayed incident budget", () => {
|
|
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryAttempt: 1 }), undefined, [], undefined, now)).toMatchObject({ attempt: 2, maxAttempts: 3 });
|
|
});
|
|
it("keeps saved governed work in shutdown reconciliation without offering a new attempt", () => {
|
|
expect(project(
|
|
run({ status: "failed", errorCode: "native_governed_wait_settlement_pending" }),
|
|
coordinator({ phase: "retryable_failure", failureCode: "native_governed_wait_settlement_pending",
|
|
nextAttemptAt: new Date(now.getTime() + 30_000) }),
|
|
)).toMatchObject({ phase: "finishing", label: "Finishing shutdown", permittedActions: ["inspect_run"],
|
|
nextAction: "The work and waiting state are saved. Paperclip is confirming shutdown before continuing." });
|
|
});
|
|
it("shows pending interaction only after productive work has stopped", () => {
|
|
expect(
|
|
project(run(), coordinator(), [{ kind: "connection_intent" }]).phase,
|
|
).toBe("working");
|
|
expect(
|
|
project(run({ status: "succeeded" }), undefined, [
|
|
{ kind: "connection_intent" },
|
|
]).phase,
|
|
).toBe("waiting_for_access");
|
|
expect(
|
|
project(run({ status: "succeeded" }), undefined, [
|
|
{ kind: "ask_user_questions" },
|
|
]).phase,
|
|
).toBe("waiting_for_answer");
|
|
});
|
|
it("retains replacement lineage and exposes operator-owned failures for legacy runs", () => {
|
|
expect(
|
|
project(
|
|
run({ status: "failed" }),
|
|
coordinator({
|
|
phase: "terminal_failure",
|
|
failureDetail: { successorRunId: "replacement" },
|
|
}),
|
|
),
|
|
).toMatchObject({ phase: "completed", successorRunId: "replacement" });
|
|
expect(
|
|
projectExecution(
|
|
run({ runtimeMode: "legacy", status: "failed" }),
|
|
undefined,
|
|
[],
|
|
{
|
|
cause: "uncertain_external_action",
|
|
nextAction: "Reconcile the email delivery.",
|
|
},
|
|
now,
|
|
),
|
|
).toMatchObject({
|
|
phase: "recovery_needed",
|
|
recoveryOwner: "board",
|
|
nextAction: "Reconcile the email delivery.",
|
|
});
|
|
});
|
|
it("bounds recovery checking and surfaces its next action if verification never finishes", () => {
|
|
const c = coordinator({
|
|
phase: "terminal_failure",
|
|
failureCode: "native_provider_terminal_failed",
|
|
});
|
|
expect(project(run({ status: "failed", finishedAt: now }), c).label).toBe(
|
|
"Checking recovery",
|
|
);
|
|
expect(
|
|
project(
|
|
run({ status: "failed", finishedAt: new Date(now.getTime() - 60_000) }),
|
|
c,
|
|
).phase,
|
|
).toBe("recovery_needed");
|
|
expect(
|
|
project(
|
|
run({ status: "failed", finishedAt: now }),
|
|
coordinator({
|
|
...c,
|
|
failureDetail: { replacementDenied: "uncertain_provider_action" },
|
|
}),
|
|
).phase,
|
|
).toBe("recovery_needed");
|
|
});
|
|
});
|