mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip lets people start and stop work by AI agents. > - Stop must end the current turn and preserve the task for a later message. > - A person can press Stop while the server still prepares the turn. > - The preparation owner can prove that no model work started. > - That path saved the proof but left the Stop receipt at `requested`. > - The page then reported an unverified Stop and a failed attempt. > - This PR completes the receipt after proof and shows the confirmed Stop without a failure notice. ## Linked Issues or Issue Description **What happened?** A person submits a task by mistake and presses Stop during preparation. The run ends before the adapter starts. The server records that no provider work started, but the Stop receipt stays at `requested`. The page waits for a receipt that will never change. It then reports that it cannot verify Stop. The task also shows a failed-attempt notice. **Expected behavior** Stop ends the current turn without a failure notice. A later message starts one new turn through normal admission. Stop does not restart the task by itself. **Steps to reproduce** Hold session preparation before adapter dispatch. Press Stop through the run API. Release preparation and wait for the owner to finish. Read the run and the task's interruption state. Send a new task comment. Related: #20644 covers saved Retry after a committed native cancellation. #21382 covers a provider failure after native Stop. This PR covers Stop before adapter dispatch and its page presentation. ## What Changed - Complete the Stop receipt in `settleCancelledLegacyPreparation`. Keep its existing checks for process, session, events, leases and saved files. - Require no saved `budgetDispatch.handedOffAt` marker under the run lock. This marker can exist before an adapter records its start. A stale controller must keep the recovery hold after this handoff. - Write and require the `preparation_settled` proof before clearing the failed-attempt notice. Both page queries use one bounded SQL projection. Unknown and foreign proof labels keep the notice. - Clear an obsolete automatic-recovery marker only for a user's operator Stop. Keep system cancellation delivery metadata so saved comments still reach admission. Cover both cases with persisted recovery records and the stale-comment journey. - Give the bounded SQL result an explicit column name. Keep its real-query tests for typed and bounded Stop evidence. - Give the Retry test an active company membership and remove it during cleanup. The test uses normal authorization. - Record an expected cancellation as information in the run log. - Add a regression through the real Stop, run-read and comment routes with a real test database. Only the adapter boundary is simulated. Check that preparation cannot confirm Stop early, Stop does not restart work, and one later message starts one turn. - Document preparation Stop settlement in the heartbeat module notes. ## Verification - Head: `372ad381f4b60258e17b6b05eda133ed875e7de3`. This revision includes master `20b0d4e5a5`. [Full CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43910771322/attempts/2) completed successfully: all 52 executed jobs passed, including Build, Typecheck, all Runner lanes, all nine serialized suites, all twelve server shards and all eight browser shards. Greptile and Paperclip Review are both 5/5, and Superagent passed on this exact head. All three inline threads are resolved. The final commit inventory has 57 successful checks, one neutral review and three expected skips, with no pending or failing checks; Snyk also passed. - All 339 focused tests pass at this head across 11 suites. The Stop, saved-comment and fresh-continuation suites pass 132 tests. Eight other Stop contract suites pass 207 tests. - Red: the preparation regression receives `executionCancellation.state = requested` after the owner finishes. The presentation regressions report a failed attempt after a confirmed Stop. - Red: automatic recovery resolves the hold first, then preparation confirms Stop, but the run still shows a recovery notice. Green: the confirmed operator Stop clears the obsolete marker and projects `completed` / `Stopped`. - Red: the previous repair also changed system cancellations. Both stale-comment and Retry cases pass on the base and fail on the previous PR head. A direct regression also catches a preparation Stop receipt written for a system cancellation. Green: only a user's operator Stop gets this receipt; system cancellations keep their recovery delivery metadata. - Red: the real-query tests read PostgreSQL's implicit expression-column name after the expression changed. Green: the query has the explicit `result_json` alias, and all typed and bounded evidence assertions pass. - Red: a saved dispatch handoff marker was present before any adapter event. Both new cases returned an acknowledged Stop instead of keeping the hold. Green: the current locked row must have no marker, including a marker with JSON null. The receipt stays requested and the hold stays active. The tests use a stale owner snapshot. - The HTTP Stop/read/comment test uses real routes and PostgreSQL. It proves that pending preparation cannot confirm Stop, the owner confirms it after cleanup, Stop does not restart work, and a later message starts exactly one turn. Only the adapter boundary is simulated. - All source typecheck layers and the changed `test-tree-0` layer pass in low-memory mode. The first full typecheck attempt lost the compiler to the workspace memory limit while tests ran in parallel. The affected layers then passed alone. Other test layers passed on the previous head. The full current-head CI Typecheck + Release Registry job passed. `git diff --check` passes. - `pnpm verify` returned exit 2 and requested broader CI. No full local suite was started. - Environment: Linux x64, Node 25.9.0, fresh embedded PostgreSQL. This does not execute a live model or change staging. CI supplies Rust and full-suite coverage. - Previous-head CI found the SQL assertion and continuation regressions above. Its cancelled workspace lane lost its runner before a test result and was retried once. Previous-head results do not count as final-head passes. ## Risks - Only a requested, expected operator Stop with an acknowledged receipt gets neutral presentation. Unknown termination and file recovery remain visible. - The preparation owner writes acknowledgement only after all existing evidence checks pass. Terminal status alone is insufficient. A retained local lease test checks both refusal before release and confirmation after release. - The receipt does not grant replay authority. Budget, approval, dependency and cleanup gates still control later messages. - No schema or migration changes are required. ## Model Used OpenAI Codex, GPT-6, with reasoning and code tools. The exact served model build and context size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
808 lines
63 KiB
TypeScript
808 lines
63 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { agents, agentWakeupRequests, chatActions, chatEndpoints, companies, createDb, applyPendingMigrations, environmentLeases, heartbeatRuns, issueRecoveryActions,
|
|
issueRelations, issues,
|
|
toolApplications, toolConnections,
|
|
nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
|
|
import { createSystemRecovery } from "../services/system-recovery-state.js";
|
|
import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
|
|
import { readNativeSessionResumptionAgent } from "../services/native-runtime/native-session-resumption-admission.js";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
|
import { captureInterruptionQuery, seedInterruptionFixture } from "./helpers/task-interruption-fixture.js";
|
|
|
|
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TASK_INTERRUPTION_TEST_DATABASE_URL");
|
|
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
|
const captureTaskInterruptionFailure = vi.hoisted(() => vi.fn());
|
|
vi.mock("../sentry.js", () => ({ captureTaskInterruptionFailure }));
|
|
|
|
describe.skipIf(!support.supported)("task interruption latest run query", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
beforeAll(async () => {
|
|
if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl);
|
|
database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} }
|
|
: await startEmbeddedPostgresTestDatabase("interruption-query-");
|
|
db = createDb(database.connectionString);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
|
|
// The shared fixture's failures are historical; reporting tests need a current failure.
|
|
const markFailuresFresh = (companyId: string) => db.update(heartbeatRuns).set({ finishedAt: new Date() })
|
|
.where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "failed")));
|
|
|
|
it("shows a pending native same-run resume instead of a competing Retry", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Resuming task", issuePrefix: `R${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Resume after attach failure",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() });
|
|
const resumeAt = new Date(Date.now() + 30_000);
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
|
|
attempt: 1, nextAttemptAt: resumeAt });
|
|
for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
|
|
await taskInterruptionPresentation(db, issue)]) {
|
|
expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
|
|
}
|
|
// The coordinator also pushes nextAttemptAt while it defers an ineligible
|
|
// candidate. Those deferrals, or one far past the failure, never hide Retry.
|
|
const otherAgentId = randomUUID();
|
|
await db.insert(agents).values({ id: otherAgentId, companyId, name: "Other agent", adapterType: "paperclip_runner" });
|
|
const admission = async () => {
|
|
const [currentIssue] = await db.select().from(issues).where(eq(issues.id, issue.id));
|
|
const [currentRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
|
return readNativeSessionResumptionAgent(db, { issue: currentIssue!, run: currentRun!, coordinator: coordinator! });
|
|
};
|
|
expect(await admission()).not.toBeNull();
|
|
const otherRunId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: otherRunId, companyId, agentId: otherAgentId, status: "cancelled",
|
|
finishedAt: new Date(Date.now() - 60_000), createdAt: new Date(Date.now() - 60 * 60_000) });
|
|
const terminatedManagerId = randomUUID();
|
|
await db.insert(agents).values({ id: terminatedManagerId, companyId, name: "Former manager",
|
|
adapterType: "paperclip_runner", status: "terminated" });
|
|
for (const [table, change, restore] of [
|
|
["issue", { assigneeAgentId: otherAgentId }, { assigneeAgentId: agentId }],
|
|
["issue", { status: "blocked" }, { status: "in_progress" }],
|
|
["issue", { executionRunId: otherRunId }, { executionRunId: null }],
|
|
["issue", { checkoutRunId: otherRunId }, { checkoutRunId: null }],
|
|
["run", { contextSnapshot: { nativeQuestionCancellation: { requestId: "q" } } }, { contextSnapshot: {} }],
|
|
// Invokability is more than status: an idle agent under a terminated manager is refused.
|
|
["agent", { reportsTo: terminatedManagerId }, { reportsTo: null }],
|
|
] as const) {
|
|
if (table === "issue") await db.update(issues).set(change).where(eq(issues.id, issue.id));
|
|
else if (table === "run") await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId));
|
|
else await db.update(agents).set(change).where(eq(agents.id, agentId));
|
|
// The scheduler would defer this candidate, so the task offers Retry.
|
|
if (table !== "run") expect(await admission()).toBeNull();
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
if (table === "issue") await db.update(issues).set(restore).where(eq(issues.id, issue.id));
|
|
else if (table === "run") await db.update(heartbeatRuns).set(restore).where(eq(heartbeatRuns.id, runId));
|
|
else await db.update(agents).set(restore).where(eq(agents.id, agentId));
|
|
}
|
|
expect(await admission()).not.toBeNull();
|
|
// A provider cooldown can schedule the resume an hour out; it stays visible.
|
|
const cooldown = new Date(Date.now() + 60 * 60_000);
|
|
await db.update(nativeRunFinalizations).set({ nextAttemptAt: cooldown }).where(eq(nativeRunFinalizations.runId, runId));
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
|
|
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: cooldown.toISOString() });
|
|
// Exhausted recovery is terminal: exactly one Retry returns.
|
|
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", nextAttemptAt: null })
|
|
.where(eq(nativeRunFinalizations.runId, runId));
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("a blocked parent whose run awaits its exact same-run resume shows retry_scheduled with no Retry", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Blocked parent", issuePrefix: `B${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Parent agent", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Wait for subtasks",
|
|
status: "blocked", assigneeAgentId: agentId }).returning();
|
|
await db.insert(issues).values({ companyId, title: "Open subtask", status: "todo", parentId: issue.id, assigneeAgentId: agentId });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date(),
|
|
processPid: 16675, processStartedAt: new Date() });
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issue.id));
|
|
const resumeAt = new Date(Date.now() + 30_000);
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
|
|
attempt: 1, nextAttemptAt: resumeAt, failureCode: "native_session_interrupted",
|
|
failureDetail: { recoverable: true, recoveryMode: "exact_checkpoint_resume", checkpointExists: true } });
|
|
for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
|
|
await taskInterruptionPresentation(db, issue)]) {
|
|
expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
|
|
}
|
|
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, issue.id));
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issue.id));
|
|
await db.update(nativeRunFinalizations).set({ failureDetail: { recoverable: true, recoveryMode: "fresh_session" } })
|
|
.where(eq(nativeRunFinalizations.runId, runId));
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("explains a saved Retry held by unfinished blockers instead of offering Retry again (restart chaos t05)", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Waiting parent", issuePrefix: `W${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Parent agent", adapterType: "paperclip_runner" });
|
|
const [parent] = await db.insert(issues).values({ companyId, title: "Write the brief", identifier: "CHA-5",
|
|
status: "blocked", assigneeAgentId: agentId }).returning();
|
|
const [child] = await db.insert(issues).values({ companyId, title: "Part 2", identifier: "CHA-10", parentId: parent.id,
|
|
status: "blocked", assigneeAgentId: agentId }).returning();
|
|
await db.insert(issueRelations).values({ companyId, issueId: child.id, relatedIssueId: parent.id, type: "blocks" });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: parent.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date(Date.now() - 60_000) });
|
|
const [retry] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "task_retry",
|
|
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 30_000),
|
|
payload: { issueId: parent.id, taskRetry: { sourceRunId: runId } } }).returning();
|
|
const presentation = await taskInterruptionPresentation(db, parent);
|
|
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: retry.id,
|
|
retryWait: { reason: "dependency_blocked" } });
|
|
expect(presentation?.retryWait?.message).toContain("blocked task this one depends on");
|
|
expect(JSON.stringify(presentation)).not.toContain("Part 2");
|
|
expect(JSON.stringify(presentation)).not.toContain("CHA-10");
|
|
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, child.id));
|
|
expect((await taskInterruptionPresentations(db, companyId, [parent])).get(parent.id)?.retryWait?.message)
|
|
.toContain("Remove or replace that dependency");
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, child.id));
|
|
expect((await taskInterruptionPresentation(db, parent))?.retryWait?.reason).not.toBe("dependency_blocked");
|
|
});
|
|
|
|
it("does not report a superseded Retry as failure after a newer conversation succeeds", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), oldRunId = randomUUID(), newRunId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Continued conversation", issuePrefix: `C${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "paperclip_runner" });
|
|
const [task] = await db.insert(issues).values({ companyId, title: "Continue setup", status: "in_review",
|
|
assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values([
|
|
{ id: oldRunId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue", runtimeMode: "native",
|
|
status: "failed", createdAt: new Date(Date.now() - 120_000), finishedAt: new Date(Date.now() - 110_000) },
|
|
{ id: newRunId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue", runtimeMode: "native",
|
|
status: "succeeded", createdAt: new Date(Date.now() - 60_000), finishedAt: new Date(Date.now() - 30_000) },
|
|
]);
|
|
const [retry] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "task_retry",
|
|
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 90_000),
|
|
payload: { issueId: task.id, taskRetry: { sourceRunId: oldRunId } } }).returning();
|
|
for (const presentation of [await taskInterruptionPresentation(db, task),
|
|
(await taskInterruptionPresentations(db, companyId, [task])).get(task.id)]) {
|
|
expect(presentation).toMatchObject({ state: "resolved", canRetry: false, intentId: null, observedRunId: newRunId });
|
|
}
|
|
// Reads do not acknowledge input or retire execution authority.
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.id)))[0]?.status)
|
|
.toBe("deferred_issue_execution");
|
|
// A later provider failure still belongs to the current run.
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "native_provider_terminal_failed" })
|
|
.where(eq(heartbeatRuns.id, newRunId));
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true, observedRunId: newRunId });
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null }).where(eq(heartbeatRuns.id, newRunId));
|
|
// Unconsumed ordinary input is independent of the obsolete Retry.
|
|
const [message] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand",
|
|
reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user",
|
|
requestedAt: new Date(Date.now() - 20_000), payload: { issueId: task.id } }).returning();
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", intentId: message.id });
|
|
});
|
|
|
|
// G1 wave s9802 t02: an accepted Retry held by the unpriced-usage refusal
|
|
// read as a budget wait. It is an accounting matter; name it so.
|
|
it("labels a Retry held by unpriced usage as accounting, not budget", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Unpriced", issuePrefix: `U${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "paperclip_runner" });
|
|
const [task] = await db.insert(issues).values({ companyId, title: "Task", identifier: "CHA-2",
|
|
status: "blocked", assigneeAgentId: agentId }).returning();
|
|
const [todo] = await db.insert(issues).values({ companyId, title: "Not started", identifier: "CHA-3",
|
|
status: "todo", assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_cleanup_quarantined", finishedAt: new Date(Date.now() - 60_000) });
|
|
const refusal = "Company cannot start work because recorded usage has no reliable price.";
|
|
await db.insert(agentWakeupRequests).values([{ companyId, agentId, source: "on_demand", reason: "task_retry",
|
|
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 30_000),
|
|
payload: { issueId: task.id, taskRetry: { sourceRunId: runId }, executionWait: { reason: "budget.blocked", message: refusal } } },
|
|
{ companyId, agentId, source: "assignment", reason: "budget.blocked", status: "skipped", error: refusal,
|
|
requestedAt: new Date(), payload: { issueId: todo.id } }]);
|
|
const held = await taskInterruptionPresentation(db, task);
|
|
expect(held?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
|
|
expect(held?.retryWait?.message).not.toContain("budget to allow");
|
|
const waiting = (await taskInterruptionPresentations(db, companyId, [todo])).get(todo.id);
|
|
expect(waiting?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
|
|
// A budget hard stop keeps its budget label.
|
|
await db.update(agentWakeupRequests).set({ payload: { issueId: task.id, taskRetry: { sourceRunId: runId },
|
|
executionWait: { reason: "budget.blocked", message: "Company is paused because its budget hard-stop was reached." } } })
|
|
.where(eq(agentWakeupRequests.reason, "task_retry"));
|
|
expect((await taskInterruptionPresentation(db, task))?.retryWait?.reason).toBe("budget.blocked");
|
|
});
|
|
|
|
// Shape saved by the previous release for a remote native run: the provider
|
|
// turn completed (checkpoint terminal), a same-run recovery was then refused by
|
|
// the cleanup gate. No migration rewrites it; the projection reads it as is.
|
|
it("reads a quarantine saved after a completed turn as settled work, not a failed attempt", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), otherRunId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Completed turn", issuePrefix: `K${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Codie", adapterType: "paperclip_runner" });
|
|
const [task] = await db.insert(issues).values({ companyId, title: "Set up Storie", identifier: "CMP-7",
|
|
status: "blocked", assigneeAgentId: agentId }).returning();
|
|
const staged = (id: string, turnTerminalState: string) => ({ id, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_cleanup_quarantined",
|
|
error: "native_session_cleanup_quarantined: prior session cleanup requires operator recovery; verify its retained process ownership and checkpoint before a controlled restart. Clearing a task session does not resolve this quarantine.",
|
|
finishedAt: new Date(Date.now() - 60_000),
|
|
resultJson: { apiToolReceipts: { receipt: { state: "completed" } }, semanticToolReceipts: { doc: { state: "completed" } } },
|
|
runnerProfileJson: { mode: "native", backend: "codex_app_server", sessionCheckpoint: {
|
|
cursor: "818", activeTurnId: null, semanticResult: null, driverKind: "codex_app_server", backendKind: "runner",
|
|
identity: { runId: id, agentId, issueId: task.id, companyId, sessionId: randomUUID() },
|
|
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState,
|
|
reportedWorkDisposition: "yielded" },
|
|
terminalTurns: [{ turnId: randomUUID(), fingerprint: "{}" }],
|
|
} } }) as typeof heartbeatRuns.$inferInsert;
|
|
await db.insert(heartbeatRuns).values(staged(runId, "completed"));
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId: task.id, runId, phase: "terminal_failure",
|
|
attempt: 2, failureCode: "native_session_cleanup_quarantined" });
|
|
// Nothing settles a gate without an owner (s9107 t04): Retry is offered.
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true });
|
|
// An unreleased remote environment is that owner: its termination receipt retires the gate.
|
|
const leaseId = randomUUID();
|
|
await db.insert(environmentLeases).values({ id: leaseId, companyId, heartbeatRunId: runId, provider: "daytona",
|
|
providerLeaseId: "sandbox" });
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: false,
|
|
retryWait: { reason: "cleanup_quarantined" } });
|
|
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date() }).where(eq(environmentLeases.id, leaseId));
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true });
|
|
// The task waits on an active child: it shows that wait, not a failed attempt.
|
|
const [child] = await db.insert(issues).values({ companyId, title: "Verify review", identifier: "CMP-8",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
await db.insert(issueRelations).values({ companyId, issueId: child.id, relatedIssueId: task.id, type: "blocks" });
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "resolved", canRetry: false });
|
|
// A quarantine whose turn did not complete is still unfinished; with no
|
|
// owner settling its gate, Retry is the way forward.
|
|
await db.update(heartbeatRuns).set({ createdAt: new Date(Date.now() - 120_000) }).where(eq(heartbeatRuns.id, runId));
|
|
await db.insert(heartbeatRuns).values(staged(otherRunId, "interrupted"));
|
|
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true,
|
|
observedRunId: otherRunId });
|
|
});
|
|
|
|
// A user's message can join an earlier automatic wake that keeps its system requester.
|
|
it.each(["user", "system"] as const)("shows a queued-message Interrupt as handing off, not as a failed attempt (queue started by %s)", async (queueStartedBy) => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), userId = "board-user";
|
|
await db.insert(companies).values({ id: companyId, name: "Interrupted task", issuePrefix: `Q${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Researcher", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Creative brief",
|
|
status: "todo", assigneeAgentId: agentId }).returning();
|
|
// The message was queued well before the user clicked Interrupt.
|
|
const [wake] = await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "on_demand", status: "deferred_issue_execution", requestedByActorType: queueStartedBy,
|
|
requestedAt: new Date(Date.now() - 60_000), payload: { issueId: issue.id },
|
|
}).returning();
|
|
const interrupt = async (requestedAt: Date) => {
|
|
await db.update(agentWakeupRequests).set({ payload: { issueId: issue.id,
|
|
queuedCommentInterrupt: { actorId: userId, requestedAt: requestedAt.toISOString() } } })
|
|
.where(eq(agentWakeupRequests.id, wake.id));
|
|
};
|
|
await interrupt(new Date(Date.now() - 20_000));
|
|
// Same resultJson the queued-comment Interrupt route passes to cancelRun.
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
|
status: "cancelled", runtimeMode: "native", errorCode: "operator_interrupted",
|
|
startedAt: new Date(Date.now() - 42_000), finishedAt: new Date(Date.now() - 20_000),
|
|
resultJson: { operatorInterrupted: true, interruptionSource: "issue_comment_interrupt", interruptedIssueId: issue.id,
|
|
interruptedByActorType: "user", interruptedByActorId: userId, cancelledByActorType: "user", cancelledByUserId: userId,
|
|
queuedCommentInterruptQueueId: wake.id } });
|
|
captureTaskInterruptionFailure.mockClear();
|
|
for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
|
|
await taskInterruptionPresentation(db, issue)]) {
|
|
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: wake.id });
|
|
}
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
// A newer queued message does not hide the wake the run was stopped for.
|
|
await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "on_demand", status: "deferred_issue_execution", requestedByActorType: "user",
|
|
requestedAt: new Date(Date.now() - 30_000), payload: { issueId: issue.id },
|
|
});
|
|
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, intentId: wake.id });
|
|
// A run stopped for a different queue is not this hand-off.
|
|
await db.update(heartbeatRuns).set({ resultJson: { operatorInterrupted: true, cancelledByActorType: "user",
|
|
cancelledByUserId: userId, queuedCommentInterruptQueueId: randomUUID() } }).where(eq(heartbeatRuns.id, runId));
|
|
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
|
await db.update(heartbeatRuns).set({ resultJson: { operatorInterrupted: true, cancelledByActorType: "user",
|
|
cancelledByUserId: userId, queuedCommentInterruptQueueId: wake.id } }).where(eq(heartbeatRuns.id, runId));
|
|
// A hand-off that never starts eventually offers one Retry.
|
|
await interrupt(new Date(Date.now() - 10 * 60_000));
|
|
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
|
});
|
|
|
|
it("decides several pending native resumes in one list read", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), otherAgentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Resuming list", issuePrefix: `L${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values([{ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" },
|
|
{ id: otherAgentId, companyId, name: "Other agent", adapterType: "paperclip_runner" }]);
|
|
const resumeAt = new Date(Date.now() + 30_000);
|
|
const list = [];
|
|
for (let index = 0; index < 6; index++) {
|
|
const [issue] = await db.insert(issues).values({ companyId, title: `Resume ${index}`, status: "in_progress",
|
|
assigneeAgentId: index % 2 ? otherAgentId : agentId }).returning();
|
|
const runId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() });
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
|
|
attempt: 1, nextAttemptAt: resumeAt });
|
|
list.push(issue);
|
|
}
|
|
const result = await taskInterruptionPresentations(db, companyId, list);
|
|
list.forEach((issue, index) => expect(result.get(issue.id)).toMatchObject(index % 2
|
|
? { state: "failed", canRetry: true }
|
|
: { state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }));
|
|
});
|
|
|
|
it("redacts workspace preparation errors on task detail and list reads", async () => {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
|
try {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const secret = `private-workspace-value-${randomUUID()}`;
|
|
const error = `Could not prepare the project folder: ${secret}.`;
|
|
await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error }).where(eq(heartbeatRuns.id, run!.id));
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
|
|
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
|
|
const detail = await taskInterruptionPresentation(db, f.input[0]!);
|
|
for (const presentation of [list.get(f.input[0]!.id), detail]) {
|
|
expect(presentation?.workspaceFailure?.message).toContain("Could not prepare the project folder");
|
|
expect(presentation?.workspaceFailure?.message).toContain("REDACTED");
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
}
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
|
|
expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("workspaceFailure");
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.error).toBe(error);
|
|
} finally { vi.unstubAllEnvs(); }
|
|
});
|
|
|
|
it.each(["valid", "numeric_user", "numeric_audit"] as const)("retains bounded typed Stop evidence in the real batched projection (%s)", async variant => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Stopped task", issuePrefix: `S${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Stopped agent", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Keep this task in progress",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const nativeCancellation = { schema: "paperclip.native-cancellation.v1", companyId, runId, issueId: issue.id,
|
|
scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true,
|
|
intentAuditId: variant === "numeric_audit" ? 123 : randomUUID(), acknowledgementAuditId: randomUUID() };
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id,
|
|
scopeKind: "issue", status: "cancelled", runtimeMode: "native", driverKind: "cursor_cloud_api",
|
|
resultJson: { cancelledByActorType: "user", cancelledByUserId: variant === "numeric_user" ? 123 : "board", nativeCancellation,
|
|
executionCancellation: { state: "requested" }, unrelatedPayload: "x".repeat(128_000) } });
|
|
const result = await taskInterruptionPresentations(db, companyId, [issue]);
|
|
expect(result.get(issue.id)).toMatchObject(variant === "valid"
|
|
? { state: "resolved", canRetry: false } : { state: "failed", canRetry: true });
|
|
const query = await captureInterruptionQuery(db, taskInterruptionPresentations, companyId, [issue]);
|
|
const [projected] = await db.$client.unsafe(query.sql, query.params as never[]);
|
|
expect(projected.result_json.nativeCancellation).toEqual({ ...nativeCancellation,
|
|
intentAuditId: variant === "numeric_audit" ? null : nativeCancellation.intentAuditId });
|
|
expect(projected.result_json.cancelledByUserId).toBe(variant === "numeric_user" ? null : "board");
|
|
expect(projected).toMatchObject({ company_id: companyId, native_issue_id: issue.id, runtime_mode: "native" });
|
|
expect(projected.result_json).not.toHaveProperty("unrelatedPayload");
|
|
expect(JSON.stringify(projected.result_json).length).toBeLessThan(4000);
|
|
});
|
|
|
|
it("redacts registered run secrets from both task detail and list provider explanations", async () => {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
|
try {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const secret = `private-runtime-value-${randomUUID()}`;
|
|
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
|
|
category: "unknown", message: `Provider rejected runtime value ${secret}.`, retryability: "unknown" };
|
|
await db.update(heartbeatRuns).set({ resultJson: { providerFailure } }).where(eq(heartbeatRuns.id, run!.id));
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
|
|
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
|
|
const detail = await taskInterruptionPresentation(db, f.input[0]!);
|
|
for (const presentation of [list.get(f.input[0]!.id), detail]) {
|
|
expect(presentation?.providerFailure?.message).toContain("Provider rejected runtime value");
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
}
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.resultJson).toEqual({ providerFailure });
|
|
} finally { vi.unstubAllEnvs(); }
|
|
});
|
|
|
|
it("redacts a scheduled retry's predecessor provider explanation with the predecessor's registry", async () => {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
|
try {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const predecessorId = randomUUID();
|
|
const secret = `private-predecessor-value-${randomUUID()}`;
|
|
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "daytona", phase: "startup",
|
|
category: "overloaded", message: `Sandbox refused lease for ${secret}.`, httpStatus: 502, retryability: "transient" };
|
|
await db.insert(heartbeatRuns).values({ id: predecessorId, companyId: f.companyId, agentId: run!.agentId,
|
|
status: "failed", errorCode: "environment_provider_unavailable", resultJson: { providerFailure },
|
|
createdAt: new Date(run!.createdAt.getTime() - 60_000) });
|
|
await db.update(heartbeatRuns).set({ status: "scheduled_retry", retryOfRunId: predecessorId, resultJson: {},
|
|
scheduledRetryAt: new Date(Date.now() + 30_000), scheduledRetryAttempt: 1 }).where(eq(heartbeatRuns.id, run!.id));
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, predecessorId, secret);
|
|
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
|
|
const detail = await taskInterruptionPresentation(db, f.input[0]!);
|
|
for (const presentation of [list.get(f.input[0]!.id), detail]) {
|
|
expect(presentation?.observedRunId).toBe(run!.id);
|
|
expect(presentation?.providerFailure).toMatchObject({ httpStatus: 502 });
|
|
expect(presentation?.providerFailure?.message).toContain("Sandbox refused lease");
|
|
expect(presentation?.providerFailure?.message).toContain("REDACTED");
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
}
|
|
} finally { vi.unstubAllEnvs(); }
|
|
});
|
|
|
|
it("names a failed setup command on task detail and list reads, redacted with the run's registry (CF1)", async () => {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
|
try {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const secret = `private-setup-value-${randomUUID()}`;
|
|
const command = "bash ./scripts/provision-worktree.sh";
|
|
const output = `bash: ./scripts/provision-worktree.sh: No such file or directory (${secret})`;
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "setup_failed",
|
|
error: `Execution workspace provision command "${command}" failed: ${output}`,
|
|
resultJson: { setupCommandFailure: { step: "workspace_provision", command, exitCode: 127, output } } }).where(eq(heartbeatRuns.id, run!.id));
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
|
|
const read = async () => [(await taskInterruptionPresentations(db, f.companyId, f.input)).get(f.input[0]!.id),
|
|
await taskInterruptionPresentation(db, f.input[0]!)];
|
|
for (const presentation of await read()) {
|
|
expect(presentation).toMatchObject({ state: "failed", canRetry: true, setupFailure: { step: "workspace_provision", command, exitCode: 127 } });
|
|
expect(presentation?.setupFailure?.message).toContain(`The project's workspace provision command "${command}" failed with exit code 127`);
|
|
expect(presentation?.setupFailure?.output).toContain("REDACTED");
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
}
|
|
// A run recorded before the structured cause existed is named from its fixed message shape. Its
|
|
// registered secret crosses the 300-character cut; it is removed whole before shortening.
|
|
const longOutput = `${secret}${"z".repeat(290)}`;
|
|
await db.update(heartbeatRuns).set({ resultJson: {}, error: `Execution workspace provision command "${command}" failed: ${longOutput}` })
|
|
.where(eq(heartbeatRuns.id, run!.id));
|
|
for (const presentation of await read()) {
|
|
expect(presentation?.setupFailure).toMatchObject({ step: "workspace_provision", command, exitCode: null });
|
|
expect(presentation?.setupFailure?.output?.length).toBeLessThanOrEqual(300);
|
|
const text = JSON.stringify(presentation);
|
|
for (let index = 0; index + 8 <= secret.length; index += 1) expect(text).not.toContain(secret.slice(index, index + 8));
|
|
}
|
|
// A long older log still shows its final error line.
|
|
await db.update(heartbeatRuns).set({ error: `Execution workspace provision command "${command}" failed: ${"filler\n".repeat(800)}No such file or directory` })
|
|
.where(eq(heartbeatRuns.id, run!.id));
|
|
for (const presentation of await read()) expect(presentation?.setupFailure?.output).toMatch(/No such file or directory$/);
|
|
// A transient retry shows its setup_failed predecessor's cause.
|
|
const predecessorId = randomUUID();
|
|
await db.insert(heartbeatRuns).values({ id: predecessorId, companyId: f.companyId, agentId: run!.agentId, status: "failed",
|
|
errorCode: "setup_failed", resultJson: { setupCommandFailure: { step: "workspace_provision", command, exitCode: 1, output } },
|
|
createdAt: new Date(run!.createdAt.getTime() - 60_000) });
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, predecessorId, secret);
|
|
await db.update(heartbeatRuns).set({ status: "scheduled_retry", retryOfRunId: predecessorId, errorCode: null, error: null, resultJson: {},
|
|
scheduledRetryAt: new Date(Date.now() + 30_000), scheduledRetryAttempt: 1 }).where(eq(heartbeatRuns.id, run!.id));
|
|
for (const presentation of await read()) {
|
|
expect(presentation?.setupFailure).toMatchObject({ command, exitCode: 1 });
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
}
|
|
} finally { vi.unstubAllEnvs(); }
|
|
});
|
|
|
|
it("names the configured model a provider rejected on task detail and list reads (CF3 model_id)", async () => {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "native_provider_model_rejected",
|
|
runnerProfileJson: { nativeExecutionInput: { provider: { kind: "codex", model: "chaos-missing-model-1234" } } } }).where(eq(heartbeatRuns.id, run!.id));
|
|
for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, f.input)).get(f.input[0]!.id),
|
|
await taskInterruptionPresentation(db, f.input[0]!)]) {
|
|
expect(presentation).toMatchObject({ responsibility: "user", state: "waiting_for_user", userAction: "choose_model",
|
|
rejectedModel: "chaos-missing-model-1234" });
|
|
}
|
|
// Only a model rejection discloses the configured model.
|
|
await db.update(heartbeatRuns).set({ errorCode: "adapter_failed" }).where(eq(heartbeatRuns.id, run!.id));
|
|
expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("rejectedModel");
|
|
});
|
|
|
|
it.each(["decryption failure", "expanded message"])("keeps task reads available after provider display %s", async scenario => {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
|
try {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
|
const secret = "pw-short";
|
|
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
|
|
category: "unknown", message: secret + "🚀".repeat(4000 - secret.length), retryability: "unknown" };
|
|
await db.update(heartbeatRuns).set({ resultJson: { providerFailure } }).where(eq(heartbeatRuns.id, run!.id));
|
|
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
|
|
if (scenario === "decryption failure") vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
|
|
const presentation = await taskInterruptionPresentation(db, f.input[0]!);
|
|
expect(presentation).toMatchObject({ state: "failed", canRetry: true });
|
|
if (scenario === "decryption failure") expect(presentation).not.toHaveProperty("providerFailure");
|
|
else {
|
|
expect(presentation?.providerFailure?.message).toContain("REDACTED");
|
|
expect([...presentation!.providerFailure!.message]).toHaveLength(4000);
|
|
}
|
|
expect(JSON.stringify(presentation)).not.toContain(secret);
|
|
} finally { vi.unstubAllEnvs(); }
|
|
});
|
|
|
|
it("does not inspect unrelated company history for a small issue page", async () => {
|
|
const f = await seedInterruptionFixture(db, { selected: 2, history: 3, unrelated: 1500, payloadBytes: 4096 });
|
|
const query = await captureInterruptionQuery(db, taskInterruptionPresentations, f.companyId, f.input);
|
|
const [explain] = await db.$client.unsafe(`explain (analyze, buffers, format json) ${query.sql}`, query.params as never[]);
|
|
const inspected: number[] = [];
|
|
const visit = (node: Record<string, unknown>) => {
|
|
if (node["Relation Name"] === "heartbeat_runs") inspected.push(
|
|
(Number(node["Actual Rows"] ?? 0) + Number(node["Rows Removed by Filter"] ?? 0)) * Number(node["Actual Loops"] ?? 1));
|
|
for (const child of node.Plans as Record<string, unknown>[] ?? []) visit(child);
|
|
};
|
|
visit(explain["QUERY PLAN"][0].Plan);
|
|
expect(inspected.length).toBeGreaterThan(0);
|
|
expect(Math.max(...inspected)).toBeLessThan(30);
|
|
await markFailuresFresh(f.companyId);
|
|
const presentations = await taskInterruptionPresentations(db, f.companyId, f.input);
|
|
for (const issue of f.input) expect(presentations.get(issue.id)).toMatchObject({ state: "failed", canRetry: true });
|
|
for (const issue of f.input) expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
|
|
companyId: f.companyId, taskId: issue.id, runId: presentations.get(issue.id)!.observedRunId,
|
|
runStatus: "failed", agentAdapter: "process",
|
|
}));
|
|
});
|
|
|
|
it("preserves native, first-class and text legacy precedence, ties, and tenant scope", async () => {
|
|
const companyId = randomUUID(), otherId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values([companyId, otherId].map(id => ({ id, name: "Selectors", issuePrefix: `S${id.slice(0, 7)}` })));
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Selector agent", adapterType: "process" });
|
|
const input = await db.insert(issues).values(Array.from({ length: 10 }, (_, n) => ({ companyId, title: `Selector ${n}`, status: "in_progress", assigneeAgentId: agentId }))).returning();
|
|
const base = { companyId, agentId, status: "failed", createdAt: new Date("2026-10-27T00:00:00Z") };
|
|
const winnerIds = Array.from({ length: 7 }, () => randomUUID());
|
|
await db.$client.unsafe("alter table heartbeat_runs disable trigger heartbeat_runs_set_scope_kind");
|
|
await db.insert(heartbeatRuns).values([
|
|
{ ...base, id: winnerIds[0], nativeIssueId: input[0]!.id, issueId: input[1]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[2]!.id } },
|
|
{ ...base, id: winnerIds[1], issueId: input[1]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[2]!.id } },
|
|
{ ...base, id: winnerIds[2], contextSnapshot: { issueId: input[2]!.id } },
|
|
{ ...base, nativeIssueId: randomUUID(), contextSnapshot: { issueId: input[3]!.id } },
|
|
{ ...base, nativeIssueId: null, issueId: input[9]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[3]!.id } },
|
|
{ ...base, contextSnapshot: { issueId: input[3]!.id.toUpperCase() } },
|
|
...[null, "", [], 5, { issueId: "" }, { issueId: {} }, { issueId: [] }].map(contextSnapshot => ({ ...base, contextSnapshot: contextSnapshot as never })),
|
|
{ ...base, id: winnerIds[3], nativeIssueId: input[4]!.id, contextSnapshot: null },
|
|
{ ...base, id: winnerIds[4], issueId: input[5]!.id, scopeKind: "issue", contextSnapshot: [input[3]!.id] as never },
|
|
{ ...base, id: "00000000-0000-0000-0000-000000000001", contextSnapshot: { issueId: input[6]!.id } },
|
|
{ ...base, id: "ffffffff-ffff-ffff-ffff-ffffffffffff", nativeIssueId: input[6]!.id, contextSnapshot: { issueId: "" } },
|
|
{ ...base, companyId: otherId, nativeIssueId: input[3]!.id, createdAt: new Date("2026-10-28T00:00:00Z") },
|
|
]);
|
|
await db.$client.unsafe("alter table heartbeat_runs enable trigger heartbeat_runs_set_scope_kind");
|
|
const result = await taskInterruptionPresentations(db, companyId, [...input, input[0]!, { ...input[0]!, companyId: otherId }]);
|
|
for (let n = 0; n < 3; n++) expect(result.get(input[n]!.id)?.observedRunId).toBe(winnerIds[n]);
|
|
expect(result.get(input[3]!.id)).toBeNull();
|
|
expect(result.get(input[4]!.id)?.observedRunId).toBe(winnerIds[3]);
|
|
expect(result.get(input[5]!.id)?.observedRunId).toBe(winnerIds[4]);
|
|
expect(result.get(input[6]!.id)?.observedRunId).toBe("ffffffff-ffff-ffff-ffff-ffffffffffff");
|
|
expect(result.get(input[7]!.id)).toBeNull();
|
|
expect(result.get(input[8]!.id)).toBeNull();
|
|
const upper = { ...input[4]!, id: input[4]!.id.toUpperCase() };
|
|
expect((await taskInterruptionPresentations(db, companyId, [upper])).get(upper.id)).toBeNull();
|
|
expect(await taskInterruptionPresentations(db, companyId, [])).toEqual(new Map());
|
|
});
|
|
|
|
it("reports the saved-message failure state without reporting work, retries or user actions", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), pausedAgentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Failure states", issuePrefix: `F${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values([
|
|
{ id: agentId, companyId, name: "Worker", adapterType: "process" },
|
|
{ id: pausedAgentId, companyId, name: "Paused worker", adapterType: "process", status: "paused" },
|
|
]);
|
|
const input = await db.insert(issues).values(Array.from({ length: 9 }, (_, n) => ({
|
|
companyId, title: `State ${n}`, status: n === 5 ? "done" : "in_progress",
|
|
assigneeAgentId: n === 8 ? pausedAgentId : agentId,
|
|
}))).returning();
|
|
const runs = await db.insert(heartbeatRuns).values(input.map((issue, n) => ({
|
|
companyId, agentId: issue.assigneeAgentId!, issueId: issue.id,
|
|
status: n === 1 || n === 4 ? "succeeded" : n === 2 ? "running" : n === 7 ? "cancelled" : "failed",
|
|
errorCode: n === 6 ? "native_provider_approval_required" : null,
|
|
scheduledRetryAt: n === 3 ? new Date(Date.now() + 60_000) : null,
|
|
}))).returning();
|
|
const [wait] = await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", payload: { issueId: input[1]!.id }, requestedAt: new Date(Date.now() - 20_000),
|
|
}).returning();
|
|
captureTaskInterruptionFailure.mockClear();
|
|
const result = await taskInterruptionPresentations(db, companyId, [...input, { ...input[0]!, companyId: randomUUID() }]);
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledTimes(3);
|
|
for (const n of [0, 1, 7]) {
|
|
expect(result.get(input[n]!.id)).toMatchObject({ responsibility: "system", state: "failed" });
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith({
|
|
companyId, agentId, taskId: input[n]!.id, runId: runs[n]!.id,
|
|
runStatus: runs[n]!.status, agentAdapter: "process", intentId: n === 1 ? wait!.id : null, runErrorCode: null,
|
|
});
|
|
}
|
|
expect(result.get(input[2]!.id)?.state).toBe("working");
|
|
expect(result.get(input[3]!.id)?.state).toBe("retry_scheduled");
|
|
for (const n of [4, 5]) expect(result.get(input[n]!.id)?.state).toBe("resolved");
|
|
for (const n of [6, 8]) expect(result.get(input[n]!.id)?.responsibility).toBe("user");
|
|
});
|
|
|
|
it("reports a retained failure once at its state change, not on every later process's read", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Retained failures", issuePrefix: `R${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "process" });
|
|
const input = await db.insert(issues).values(Array.from({ length: 3 }, (_, n) => ({
|
|
companyId, title: `Retained ${n}`, status: "in_progress", assigneeAgentId: agentId,
|
|
}))).returning();
|
|
const old = new Date(Date.now() - 22 * 60 * 60_000);
|
|
const runs = await db.insert(heartbeatRuns).values(input.map((issue, n) => ({
|
|
companyId, agentId, issueId: issue.id, status: "failed", createdAt: old,
|
|
finishedAt: n === 2 ? new Date(Date.now() - 60_000) : old,
|
|
errorCode: n === 0 ? "native_workspace_sync_out_retry_exhausted" : null,
|
|
}))).returning();
|
|
// A new saved message is a new state change for an old failed run.
|
|
const [wait] = await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", payload: { issueId: input[1]!.id }, requestedAt: new Date(Date.now() - 20_000),
|
|
}).returning();
|
|
captureTaskInterruptionFailure.mockClear();
|
|
const result = await taskInterruptionPresentations(db, companyId, input);
|
|
for (const issue of input) expect(result.get(issue.id)).toMatchObject({ responsibility: "system", state: "failed" });
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledTimes(2);
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
|
|
taskId: input[1]!.id, runId: runs[1]!.id, intentId: wait!.id }));
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
|
|
taskId: input[2]!.id, runId: runs[2]!.id, intentId: null }));
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date() }).where(eq(heartbeatRuns.id, runs[0]!.id));
|
|
captureTaskInterruptionFailure.mockClear();
|
|
await taskInterruptionPresentations(db, companyId, [input[0]!]);
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
taskId: input[0]!.id, runErrorCode: "native_workspace_sync_out_retry_exhausted" }));
|
|
});
|
|
|
|
it("reports recovery that newly fails on an old run once, then treats it as retained", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Old run recovery", issuePrefix: `O${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "process" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Recovery", status: "in_progress",
|
|
assigneeAgentId: agentId }).returning();
|
|
const old = new Date(Date.now() - 22 * 60 * 60_000);
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue!.id, status: "failed",
|
|
createdAt: old, finishedAt: old }).returning();
|
|
// Exhausted recovery clears nextAttemptAt; only the saved transition time is recent.
|
|
const exhausted = { ...createSystemRecovery(run!.id), state: "exhausted" as const, attempts: 6, retriesUsed: 5,
|
|
firstFailureAt: old.getTime(), deadlineAt: old.getTime() + 30 * 60_000, nextAttemptAt: null };
|
|
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue!.id,
|
|
kind: "workspace_recovery", status: "active", ownerType: "system", cause: "native_workspace_sync_out_failed",
|
|
fingerprint: `recovery:${run!.id}`, nextAction: "Recover saved files.", createdAt: old, updatedAt: new Date(),
|
|
evidence: { runId: run!.id, systemRecovery: exhausted } }).returning();
|
|
captureTaskInterruptionFailure.mockClear();
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
|
|
.toMatchObject({ responsibility: "system", state: "failed" });
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
taskId: issue!.id, runId: run!.id }));
|
|
await db.update(issueRecoveryActions).set({ updatedAt: old }).where(eq(issueRecoveryActions.id, action!.id));
|
|
captureTaskInterruptionFailure.mockClear();
|
|
await taskInterruptionPresentations(db, companyId, [issue!]);
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("reports a native resume that is newly exhausted on an old run once, then treats it as retained", async () => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Old native run", issuePrefix: `N${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Native worker", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Native resume", status: "in_progress",
|
|
assigneeAgentId: agentId }).returning();
|
|
const old = new Date(Date.now() - 22 * 60 * 60_000);
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
|
|
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", createdAt: old, finishedAt: old });
|
|
// Only the coordinator's saved terminal transition is recent.
|
|
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue!.id, runId, phase: "terminal_failure",
|
|
attempt: 3, nextAttemptAt: null, createdAt: old, updatedAt: new Date() });
|
|
captureTaskInterruptionFailure.mockClear();
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
|
|
.toMatchObject({ responsibility: "system", state: "failed" });
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
taskId: issue!.id, runId, runErrorCode: "native_session_interrupted" }));
|
|
await db.update(nativeRunFinalizations).set({ updatedAt: old }).where(eq(nativeRunFinalizations.runId, runId));
|
|
captureTaskInterruptionFailure.mockClear();
|
|
await taskInterruptionPresentations(db, companyId, [issue!]);
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["failed", "cancelled"])("reports a delayed Retry that newly becomes %s, then ages out", async status => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), intentId = randomUUID(), actionId = randomUUID();
|
|
const old = new Date(Date.now() - 22 * 60 * 60_000), recent = new Date(Date.now() - 60_000);
|
|
await db.insert(companies).values({ id: companyId, name: "Delayed chat Retry", issuePrefix: "D" + companyId.slice(0, 7) });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Chat worker", adapterType: "claude_local" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Deferred Retry", status: "in_progress",
|
|
assigneeAgentId: agentId }).returning();
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId: issue!.id,
|
|
status: "failed", createdAt: old, finishedAt: old });
|
|
const [application] = await db.insert(toolApplications).values({ companyId, name: "Chat",
|
|
applicationKey: randomUUID(), type: "chat" }).returning();
|
|
const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: application!.id,
|
|
name: "Chat", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning();
|
|
const [endpoint] = await db.insert(chatEndpoints).values({ companyId, connectionId: connection!.id,
|
|
assignedAgentId: agentId, provider: "slack", publicId: randomUUID(), status: "active" }).returning();
|
|
const payload = { issueId: issue!.id, agentId, failedRunId: runId, taskRetryIntentId: intentId };
|
|
await db.insert(chatActions).values({ id: actionId, companyId, endpointId: endpoint!.id,
|
|
kind: "failed_run_retry", providerActionId: randomUUID(), status, payload, createdAt: old, updatedAt: recent });
|
|
await db.insert(agentWakeupRequests).values({ id: intentId, companyId, agentId, source: "on_demand",
|
|
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: old,
|
|
payload: { issueId: issue!.id, taskRetry: { sourceRunId: runId, chatActionId: actionId } } });
|
|
captureTaskInterruptionFailure.mockClear();
|
|
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
|
|
.toMatchObject({ responsibility: "system", state: "failed", nextRetryAt: null });
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
taskId: issue!.id, runId, intentId }));
|
|
await db.update(chatActions).set({ updatedAt: old }).where(eq(chatActions.id, actionId));
|
|
captureTaskInterruptionFailure.mockClear();
|
|
await taskInterruptionPresentations(db, companyId, [issue!]);
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
await db.update(chatActions).set({ updatedAt: recent, payload: { ...payload, failedRunId: randomUUID() } })
|
|
.where(eq(chatActions.id, actionId));
|
|
captureTaskInterruptionFailure.mockClear();
|
|
await taskInterruptionPresentations(db, companyId, [issue!]);
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("keeps the failed run's adapter after the task is reassigned", async () => {
|
|
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
|
|
const [nextAgent] = await db.insert(agents).values({
|
|
companyId: f.companyId, name: "Next worker", adapterType: "codex_local",
|
|
}).returning();
|
|
const [reassigned] = await db.update(issues).set({ assigneeAgentId: nextAgent!.id })
|
|
.where(eq(issues.id, f.input[0]!.id)).returning();
|
|
await markFailuresFresh(f.companyId);
|
|
captureTaskInterruptionFailure.mockClear();
|
|
const result = await taskInterruptionPresentations(db, f.companyId, [reassigned!]);
|
|
expect(result.get(reassigned!.id)?.state).toBe("failed");
|
|
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
|
|
taskId: reassigned!.id, agentAdapter: "process",
|
|
}));
|
|
});
|
|
describe("a newer continuation dropped before it started (Gate C C1 t169)", () => {
|
|
const seed = async (newer: Partial<typeof heartbeatRuns.$inferInsert>) => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), runningId = randomUUID(), droppedId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Continuing task", issuePrefix: `C${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Continuing agent", adapterType: "paperclip_runner" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Keep working",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const now = Date.now();
|
|
await db.insert(heartbeatRuns).values({ id: runningId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
|
|
status: "running", runtimeMode: "native", createdAt: new Date(now - 60_000), startedAt: new Date(now - 59_000) });
|
|
await db.insert(heartbeatRuns).values({ id: droppedId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
|
|
status: "cancelled", runtimeMode: "native", createdAt: new Date(now - 30_000), finishedAt: new Date(now - 30_000),
|
|
contextSnapshot: { issueId: issue!.id, wakeReason: "issue_continuation_needed" }, ...newer });
|
|
return { companyId, issue: issue!, runningId, droppedId };
|
|
};
|
|
|
|
it.each(["recovery_no_longer_needed", "issue_closed_before_start"])(
|
|
"shows the running run, not a failed attempt (%s)", async errorCode => {
|
|
const f = await seed({ errorCode, error: "The task no longer needed this recovery run when its turn came." });
|
|
for (const presentation of [(await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id),
|
|
await taskInterruptionPresentation(db, f.issue)]) {
|
|
expect(presentation).toMatchObject({ state: "working", canRetry: false, observedRunId: f.runningId });
|
|
}
|
|
});
|
|
|
|
it("still reports a cancelled run that had started", async () => {
|
|
const f = await seed({ errorCode: "recovery_no_longer_needed", startedAt: new Date(Date.now() - 29_000) });
|
|
expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId });
|
|
});
|
|
|
|
it("still reports a run a person cancelled before it started", async () => {
|
|
const f = await seed({ errorCode: "cancelled", resultJson: { cancelledByActorType: "user", cancelledByUserId: "board" } });
|
|
expect((await taskInterruptionPresentations(db, f.companyId, [f.issue])).get(f.issue.id))
|
|
.toMatchObject({ state: "failed", canRetry: true, observedRunId: f.droppedId });
|
|
});
|
|
});
|
|
// N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy
|
|
// retry. Nothing failed, yet the banner offered a Retry that cannot help.
|
|
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait", "storage_unavailable_wait"])(
|
|
"does not present an overdue %s wait as a failed attempt", async reason => {
|
|
const companyId = randomUUID(), agentId = randomUUID(), parentRunId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Waiting task", issuePrefix: `W${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Waiting agent", adapterType: "process" });
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Paperclip onboarding",
|
|
status: "in_progress", assigneeAgentId: agentId }).returning();
|
|
const now = Date.now();
|
|
await db.insert(heartbeatRuns).values({ id: parentRunId, companyId, agentId, issueId: issue!.id, scopeKind: "issue",
|
|
status: "cancelled", errorCode: reason, createdAt: new Date(now - 10 * 60_000), finishedAt: new Date(now - 9 * 60_000) });
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue!.id, scopeKind: "issue",
|
|
status: "scheduled_retry", retryOfRunId: parentRunId, scheduledRetryReason: reason, scheduledRetryAttempt: 2,
|
|
scheduledRetryAt: new Date(now - 5 * 60_000), createdAt: new Date(now - 9 * 60_000) });
|
|
captureTaskInterruptionFailure.mockClear();
|
|
const presentation = (await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id);
|
|
expect(presentation).toMatchObject({ responsibility: "system", state: "working", canRetry: false });
|
|
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
|
|
});
|
|
});
|