Files
PaperClipAI/server/src/__tests__/task-interruption-query.test.ts
T
DottaandPaperclip 3cc1032f6a fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path

> - Paperclip lets people direct agent work through tasks and structured
answers.
> - Dependencies stop task execution until prerequisite work finishes.
> - A person must still be able to answer or redirect the assignee of a
blocked task.
> - Ordinary comments already get a bounded answer/triage turn. A saved
question response has no comment ID, so the same gates reject it.
> - Later dispatch also requires an active task and tries to check it
out. That conflicts with bounded interaction on a blocked task.
> - This pull request verifies the stored human response at each gate
and keeps the turn in bounded interaction mode.
> - The assignee can receive the answer without starting dependency work
or removing its blockers.

## Linked Issues or Issue Description

Refs #19923. That change covers dependency waits and comment retries.
This change covers stored interaction responses that have no comment ID.

**What happened?**

A parent task waits on a parked child. The assignee asked a setup
question. After a person answers, the response stays queued. Sending
that saved response is refused as `issue_dependencies_blocked`. Ordinary
new messages can reach the same assignee.

Passing admission alone is insufficient. The dispatch status check
rejects a blocked task, and automatic checkout then tries to start work
whose prerequisites are unfinished.

**Expected behavior**

Deliver the stored human answer or confirmation to its requesting
assignee as a bounded answer/triage turn. Keep the task blocked and keep
its dependency relationships. Preserve current-owner, pause, approval,
budget, recovery, and execution gates.

**Steps to reproduce**

1. Assign a parent task and make a backlog child block it.
2. Save a question on the parent and answer it as a person.
3. Dispatch its resolved-interaction wake, or send the saved answer from
the task queue.
4. Observe that the answer does not reach the provider while an ordinary
comment can run in answer/triage mode.

**Paperclip version or commit**

Reproduced on master at `653ab8b18f76`. This branch is rebased onto
master at `0e3fdebea8`.

## What Changed

- Verify the stored response by company, task, requesting agent, status,
kind, human resolver, and resolution time. A context flag alone is
insufficient.
- Allow a verified response through dependency admission and claim in
the existing bounded interaction mode.
- Recheck that response during scheduled retry promotion and final
provider dispatch.
- Keep bounded responses from checking out the blocked task or setting
it to `in_progress`, including when the last blocker finishes during
preparation.
- Preserve continuation classification through workspace and AI-account
resource waits, so the final response and owner checks still run.
- Correct the task-case API summary so the existing `case for task`
discovery regression passes on the first search page. This fixes a
pre-existing failure exposed by full CI, without changing search ranking
or assertions.
- Ignore obsolete Retry notices after newer conversation work, without
acknowledging saved input or changing execution authority.
- Document the response contract. Add real PostgreSQL and
assembled-heartbeat regressions.

## Verification

- Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI
passed, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2),
including typecheck, production build, server/chat tests, Runner checks,
and all 8 browser shards. PR status: 59 passing checks and 3 normal
skips.
- Red: answered questions and accepted/rejected confirmations return no
admitted run before the fix. All three new positive cases fail at the
dependency gate. A resource-wait retry also reproduces as refused before
its gate is fixed.
- Green: at delivery revision `9375a603e4`, focused heartbeat
integration and classification passed 14 tests; retry scheduling and
dispatch safety passed 241 tests. These use real PostgreSQL and
controlled adapter and resource-availability boundaries. The wider
initial dependency, native-continuation, and queue suites also passed
before the final refinements.
- Negative cases keep pending, missing, wrong-task, and agent-resolved
responses blocked. Changing the response proof immediately before
dispatch cancels the run without calling the provider. Policy controls
retain the current assignee, task status, and pending approval checks.
- Positive cases execute once, leave the parent blocked, preserve the
dependency edge, and leave the child in backlog with no child execution.
The same saved response survives a resource wait; removing its human
proof refuses promotion.
- The resource-wait provider-boundary and blocker-completion regressions
also failed before their fixes. All 14 focused integration and
classifier cases passed at delivery revision `9375a603e4`, including a
changed response during retry preparation.
- Red: a retained Retry for an older run makes a newer successful
conversation display as failed. Green: all 78 database-backed query and
interruption-presentation tests pass, including a later real failure and
independent queued input controls.
- Server layered typecheck and module-boundary checks passed at
`11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API
summary correction. That CI passed build, typecheck and every browser
shard, but exposed the pre-existing case-search assertion and a
report-assets test timeout. Final-head CI passed as linked above.
- Fresh Greptile review is 5/5 on final commit
`ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads.
- The existing API discovery suite reproduced its first-page assertion
before the summary correction; all 3 cases pass afterward. The timed-out
report-assets privacy test passed locally in isolation and the full
support job passed on the final-head rerun. Runner shutdowns interrupted
the first attempt; affected jobs were rerun once without code changes. A
separate onboarding timeout also passed on that rerun.
- `pnpm verify --explain` requests broad CI because the execution
contract is a global input. The final CI run supplied that coverage.
- Separately recovered the reported staging conversation through an
ordinary message and the existing steering control. The agent
acknowledged the message and the saved-answer queue drained. The fresh
Notebook action subsequently succeeded. This was on the previously
deployed revision; it does not claim this code is deployed.

## Risks

- This changes admission and dispatch. Each gate checks the persisted
response; no caller-supplied marker grants the exception.
- The bounded turn does not check out the task, resolve a dependency, or
approve another action. Existing pause, ownership, budget, approval, and
stopped-execution controls still apply.
- Each additional lookup is one primary-key read and runs only for a
response on a dependency-blocked path. There is no new polling worker.
- No migration is required. Older queued contexts get the exception only
when their stored human response still verifies. No prior run or action
is replayed by this change.
- This PR does not roll out the code to staging.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:27:34 -06:00

629 lines
48 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, chatActions, chatEndpoints, companies, createDb, applyPendingMigrations, heartbeatRuns, issueRecoveryActions,
issueRelations, issues,
toolApplications, toolConnections,
nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { createSystemRecovery } from "../services/system-recovery-state.js";
import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
import { readNativeSessionResumptionAgent } from "../services/native-runtime/native-session-resumption-admission.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { captureInterruptionQuery, seedInterruptionFixture } from "./helpers/task-interruption-fixture.js";
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TASK_INTERRUPTION_TEST_DATABASE_URL");
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const captureTaskInterruptionFailure = vi.hoisted(() => vi.fn());
vi.mock("../sentry.js", () => ({ captureTaskInterruptionFailure }));
describe.skipIf(!support.supported)("task interruption latest run query", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl);
database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} }
: await startEmbeddedPostgresTestDatabase("interruption-query-");
db = createDb(database.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
// The shared fixture's failures are historical; reporting tests need a current failure.
const markFailuresFresh = (companyId: string) => db.update(heartbeatRuns).set({ finishedAt: new Date() })
.where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.status, "failed")));
it("shows a pending native same-run resume instead of a competing Retry", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Resuming task", issuePrefix: `R${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Resume after attach failure",
status: "in_progress", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() });
const resumeAt = new Date(Date.now() + 30_000);
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
attempt: 1, nextAttemptAt: resumeAt });
for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
await taskInterruptionPresentation(db, issue)]) {
expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
}
// The coordinator also pushes nextAttemptAt while it defers an ineligible
// candidate. Those deferrals, or one far past the failure, never hide Retry.
const otherAgentId = randomUUID();
await db.insert(agents).values({ id: otherAgentId, companyId, name: "Other agent", adapterType: "paperclip_runner" });
const admission = async () => {
const [currentIssue] = await db.select().from(issues).where(eq(issues.id, issue.id));
const [currentRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
return readNativeSessionResumptionAgent(db, { issue: currentIssue!, run: currentRun!, coordinator: coordinator! });
};
expect(await admission()).not.toBeNull();
const otherRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: otherRunId, companyId, agentId: otherAgentId, status: "cancelled",
finishedAt: new Date(Date.now() - 60_000), createdAt: new Date(Date.now() - 60 * 60_000) });
const terminatedManagerId = randomUUID();
await db.insert(agents).values({ id: terminatedManagerId, companyId, name: "Former manager",
adapterType: "paperclip_runner", status: "terminated" });
for (const [table, change, restore] of [
["issue", { assigneeAgentId: otherAgentId }, { assigneeAgentId: agentId }],
["issue", { status: "blocked" }, { status: "in_progress" }],
["issue", { executionRunId: otherRunId }, { executionRunId: null }],
["issue", { checkoutRunId: otherRunId }, { checkoutRunId: null }],
["run", { contextSnapshot: { nativeQuestionCancellation: { requestId: "q" } } }, { contextSnapshot: {} }],
// Invokability is more than status: an idle agent under a terminated manager is refused.
["agent", { reportsTo: terminatedManagerId }, { reportsTo: null }],
] as const) {
if (table === "issue") await db.update(issues).set(change).where(eq(issues.id, issue.id));
else if (table === "run") await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId));
else await db.update(agents).set(change).where(eq(agents.id, agentId));
// The scheduler would defer this candidate, so the task offers Retry.
if (table !== "run") expect(await admission()).toBeNull();
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
.toMatchObject({ state: "failed", canRetry: true });
if (table === "issue") await db.update(issues).set(restore).where(eq(issues.id, issue.id));
else if (table === "run") await db.update(heartbeatRuns).set(restore).where(eq(heartbeatRuns.id, runId));
else await db.update(agents).set(restore).where(eq(agents.id, agentId));
}
expect(await admission()).not.toBeNull();
// A provider cooldown can schedule the resume an hour out; it stays visible.
const cooldown = new Date(Date.now() + 60 * 60_000);
await db.update(nativeRunFinalizations).set({ nextAttemptAt: cooldown }).where(eq(nativeRunFinalizations.runId, runId));
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: cooldown.toISOString() });
// Exhausted recovery is terminal: exactly one Retry returns.
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", nextAttemptAt: null })
.where(eq(nativeRunFinalizations.runId, runId));
expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
.toMatchObject({ state: "failed", canRetry: true });
});
it("explains a saved Retry held by unfinished blockers instead of offering Retry again (restart chaos t05)", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Waiting parent", issuePrefix: `W${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Parent agent", adapterType: "paperclip_runner" });
const [parent] = await db.insert(issues).values({ companyId, title: "Write the brief", identifier: "CHA-5",
status: "blocked", assigneeAgentId: agentId }).returning();
const [child] = await db.insert(issues).values({ companyId, title: "Part 2", identifier: "CHA-10", parentId: parent.id,
status: "blocked", assigneeAgentId: agentId }).returning();
await db.insert(issueRelations).values({ companyId, issueId: child.id, relatedIssueId: parent.id, type: "blocks" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: parent.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date(Date.now() - 60_000) });
const [retry] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "task_retry",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 30_000),
payload: { issueId: parent.id, taskRetry: { sourceRunId: runId } } }).returning();
const presentation = await taskInterruptionPresentation(db, parent);
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: retry.id,
retryWait: { reason: "dependency_blocked" } });
expect(presentation?.retryWait?.message).toContain("blocked task this one depends on");
expect(JSON.stringify(presentation)).not.toContain("Part 2");
expect(JSON.stringify(presentation)).not.toContain("CHA-10");
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, child.id));
expect((await taskInterruptionPresentations(db, companyId, [parent])).get(parent.id)?.retryWait?.message)
.toContain("Remove or replace that dependency");
await db.update(issues).set({ status: "done" }).where(eq(issues.id, child.id));
expect((await taskInterruptionPresentation(db, parent))?.retryWait?.reason).not.toBe("dependency_blocked");
});
it("does not report a superseded Retry as failure after a newer conversation succeeds", async () => {
const companyId = randomUUID(), agentId = randomUUID(), oldRunId = randomUUID(), newRunId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Continued conversation", issuePrefix: `C${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "paperclip_runner" });
const [task] = await db.insert(issues).values({ companyId, title: "Continue setup", status: "in_review",
assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values([
{ id: oldRunId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue", runtimeMode: "native",
status: "failed", createdAt: new Date(Date.now() - 120_000), finishedAt: new Date(Date.now() - 110_000) },
{ id: newRunId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue", runtimeMode: "native",
status: "succeeded", createdAt: new Date(Date.now() - 60_000), finishedAt: new Date(Date.now() - 30_000) },
]);
const [retry] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "task_retry",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 90_000),
payload: { issueId: task.id, taskRetry: { sourceRunId: oldRunId } } }).returning();
for (const presentation of [await taskInterruptionPresentation(db, task),
(await taskInterruptionPresentations(db, companyId, [task])).get(task.id)]) {
expect(presentation).toMatchObject({ state: "resolved", canRetry: false, intentId: null, observedRunId: newRunId });
}
// Reads do not acknowledge input or retire execution authority.
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.id)))[0]?.status)
.toBe("deferred_issue_execution");
// A later provider failure still belongs to the current run.
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "native_provider_terminal_failed" })
.where(eq(heartbeatRuns.id, newRunId));
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true, observedRunId: newRunId });
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null }).where(eq(heartbeatRuns.id, newRunId));
// Unconsumed ordinary input is independent of the obsolete Retry.
const [message] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand",
reason: "issue_commented", status: "deferred_issue_execution", requestedByActorType: "user",
requestedAt: new Date(Date.now() - 20_000), payload: { issueId: task.id } }).returning();
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", intentId: message.id });
});
// G1 wave s9802 t02: an accepted Retry held by the unpriced-usage refusal
// read as a budget wait. It is an accounting matter; name it so.
it("labels a Retry held by unpriced usage as accounting, not budget", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Unpriced", issuePrefix: `U${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "paperclip_runner" });
const [task] = await db.insert(issues).values({ companyId, title: "Task", identifier: "CHA-2",
status: "blocked", assigneeAgentId: agentId }).returning();
const [todo] = await db.insert(issues).values({ companyId, title: "Not started", identifier: "CHA-3",
status: "todo", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: task.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_cleanup_quarantined", finishedAt: new Date(Date.now() - 60_000) });
const refusal = "Company cannot start work because recorded usage has no reliable price.";
await db.insert(agentWakeupRequests).values([{ companyId, agentId, source: "on_demand", reason: "task_retry",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 30_000),
payload: { issueId: task.id, taskRetry: { sourceRunId: runId }, executionWait: { reason: "budget.blocked", message: refusal } } },
{ companyId, agentId, source: "assignment", reason: "budget.blocked", status: "skipped", error: refusal,
requestedAt: new Date(), payload: { issueId: todo.id } }]);
const held = await taskInterruptionPresentation(db, task);
expect(held?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
expect(held?.retryWait?.message).not.toContain("budget to allow");
const waiting = (await taskInterruptionPresentations(db, companyId, [todo])).get(todo.id);
expect(waiting?.retryWait).toMatchObject({ reason: "accounting.unpriced_usage", message: expect.stringContaining("has no price") });
// A budget hard stop keeps its budget label.
await db.update(agentWakeupRequests).set({ payload: { issueId: task.id, taskRetry: { sourceRunId: runId },
executionWait: { reason: "budget.blocked", message: "Company is paused because its budget hard-stop was reached." } } })
.where(eq(agentWakeupRequests.reason, "task_retry"));
expect((await taskInterruptionPresentation(db, task))?.retryWait?.reason).toBe("budget.blocked");
});
it("shows a queued-message Interrupt as handing off, not as a failed attempt", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), userId = "board-user";
await db.insert(companies).values({ id: companyId, name: "Interrupted task", issuePrefix: `Q${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Researcher", adapterType: "paperclip_runner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Creative brief",
status: "todo", assigneeAgentId: agentId }).returning();
// The message was queued well before the user clicked Interrupt.
const [wake] = await db.insert(agentWakeupRequests).values({
companyId, agentId, source: "on_demand", status: "deferred_issue_execution", requestedByActorType: "user",
requestedAt: new Date(Date.now() - 60_000), payload: { issueId: issue.id },
}).returning();
const interrupt = async (requestedAt: Date) => {
await db.update(agentWakeupRequests).set({ payload: { issueId: issue.id,
queuedCommentInterrupt: { actorId: userId, requestedAt: requestedAt.toISOString() } } })
.where(eq(agentWakeupRequests.id, wake.id));
};
await interrupt(new Date(Date.now() - 20_000));
// Same resultJson the queued-comment Interrupt route passes to cancelRun.
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "cancelled", runtimeMode: "native", errorCode: "operator_interrupted",
startedAt: new Date(Date.now() - 42_000), finishedAt: new Date(Date.now() - 20_000),
resultJson: { operatorInterrupted: true, interruptionSource: "issue_comment_interrupt", interruptedIssueId: issue.id,
interruptedByActorType: "user", interruptedByActorId: userId, cancelledByActorType: "user", cancelledByUserId: userId,
queuedCommentInterruptQueueId: wake.id } });
captureTaskInterruptionFailure.mockClear();
for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
await taskInterruptionPresentation(db, issue)]) {
expect(presentation).toMatchObject({ state: "working", canRetry: false, intentId: wake.id });
}
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
// A newer queued message does not hide the wake the run was stopped for.
await db.insert(agentWakeupRequests).values({
companyId, agentId, source: "on_demand", status: "deferred_issue_execution", requestedByActorType: "user",
requestedAt: new Date(Date.now() - 30_000), payload: { issueId: issue.id },
});
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, intentId: wake.id });
// A run stopped for a different queue is not this hand-off.
await db.update(heartbeatRuns).set({ resultJson: { operatorInterrupted: true, cancelledByActorType: "user",
cancelledByUserId: userId, queuedCommentInterruptQueueId: randomUUID() } }).where(eq(heartbeatRuns.id, runId));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
await db.update(heartbeatRuns).set({ resultJson: { operatorInterrupted: true, cancelledByActorType: "user",
cancelledByUserId: userId, queuedCommentInterruptQueueId: wake.id } }).where(eq(heartbeatRuns.id, runId));
// A hand-off that never starts eventually offers one Retry.
await interrupt(new Date(Date.now() - 10 * 60_000));
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
});
it("decides several pending native resumes in one list read", async () => {
const companyId = randomUUID(), agentId = randomUUID(), otherAgentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Resuming list", issuePrefix: `L${companyId.slice(0, 7)}` });
await db.insert(agents).values([{ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" },
{ id: otherAgentId, companyId, name: "Other agent", adapterType: "paperclip_runner" }]);
const resumeAt = new Date(Date.now() + 30_000);
const list = [];
for (let index = 0; index < 6; index++) {
const [issue] = await db.insert(issues).values({ companyId, title: `Resume ${index}`, status: "in_progress",
assigneeAgentId: index % 2 ? otherAgentId : agentId }).returning();
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() });
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
attempt: 1, nextAttemptAt: resumeAt });
list.push(issue);
}
const result = await taskInterruptionPresentations(db, companyId, list);
list.forEach((issue, index) => expect(result.get(issue.id)).toMatchObject(index % 2
? { state: "failed", canRetry: true }
: { state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }));
});
it("redacts workspace preparation errors on task detail and list reads", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
try {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const secret = `private-workspace-value-${randomUUID()}`;
const error = `Could not prepare the project folder: ${secret}.`;
await db.update(heartbeatRuns).set({ errorCode: "workspace_validation_failed", error }).where(eq(heartbeatRuns.id, run!.id));
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
const detail = await taskInterruptionPresentation(db, f.input[0]!);
for (const presentation of [list.get(f.input[0]!.id), detail]) {
expect(presentation?.workspaceFailure?.message).toContain("Could not prepare the project folder");
expect(presentation?.workspaceFailure?.message).toContain("REDACTED");
expect(JSON.stringify(presentation)).not.toContain(secret);
}
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
expect(await taskInterruptionPresentation(db, f.input[0]!)).not.toHaveProperty("workspaceFailure");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.error).toBe(error);
} finally { vi.unstubAllEnvs(); }
});
it.each(["valid", "numeric_user", "numeric_audit"] as const)("retains bounded typed Stop evidence in the real batched projection (%s)", async variant => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Stopped task", issuePrefix: `S${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Stopped agent", adapterType: "paperclip_runner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Keep this task in progress",
status: "in_progress", assigneeAgentId: agentId }).returning();
const nativeCancellation = { schema: "paperclip.native-cancellation.v1", companyId, runId, issueId: issue.id,
scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true,
intentAuditId: variant === "numeric_audit" ? 123 : randomUUID(), acknowledgementAuditId: randomUUID() };
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id,
scopeKind: "issue", status: "cancelled", runtimeMode: "native", driverKind: "cursor_cloud_api",
resultJson: { cancelledByActorType: "user", cancelledByUserId: variant === "numeric_user" ? 123 : "board", nativeCancellation,
executionCancellation: { state: "requested" }, unrelatedPayload: "x".repeat(128_000) } });
const result = await taskInterruptionPresentations(db, companyId, [issue]);
expect(result.get(issue.id)).toMatchObject(variant === "valid"
? { state: "resolved", canRetry: false } : { state: "failed", canRetry: true });
const query = await captureInterruptionQuery(db, taskInterruptionPresentations, companyId, [issue]);
const [projected] = await db.$client.unsafe(query.sql, query.params as never[]);
expect(projected.jsonb_build_object.nativeCancellation).toEqual({ ...nativeCancellation,
intentAuditId: variant === "numeric_audit" ? null : nativeCancellation.intentAuditId });
expect(projected.jsonb_build_object.cancelledByUserId).toBe(variant === "numeric_user" ? null : "board");
expect(projected).toMatchObject({ company_id: companyId, native_issue_id: issue.id, runtime_mode: "native" });
expect(projected.jsonb_build_object).not.toHaveProperty("unrelatedPayload");
expect(JSON.stringify(projected.jsonb_build_object).length).toBeLessThan(4000);
});
it("redacts registered run secrets from both task detail and list provider explanations", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
try {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const secret = `private-runtime-value-${randomUUID()}`;
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
category: "unknown", message: `Provider rejected runtime value ${secret}.`, retryability: "unknown" };
await db.update(heartbeatRuns).set({ resultJson: { providerFailure } }).where(eq(heartbeatRuns.id, run!.id));
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
const detail = await taskInterruptionPresentation(db, f.input[0]!);
for (const presentation of [list.get(f.input[0]!.id), detail]) {
expect(presentation?.providerFailure?.message).toContain("Provider rejected runtime value");
expect(JSON.stringify(presentation)).not.toContain(secret);
}
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0]?.resultJson).toEqual({ providerFailure });
} finally { vi.unstubAllEnvs(); }
});
it("redacts a scheduled retry's predecessor provider explanation with the predecessor's registry", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
try {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const predecessorId = randomUUID();
const secret = `private-predecessor-value-${randomUUID()}`;
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "daytona", phase: "startup",
category: "overloaded", message: `Sandbox refused lease for ${secret}.`, httpStatus: 502, retryability: "transient" };
await db.insert(heartbeatRuns).values({ id: predecessorId, companyId: f.companyId, agentId: run!.agentId,
status: "failed", errorCode: "environment_provider_unavailable", resultJson: { providerFailure },
createdAt: new Date(run!.createdAt.getTime() - 60_000) });
await db.update(heartbeatRuns).set({ status: "scheduled_retry", retryOfRunId: predecessorId, resultJson: {},
scheduledRetryAt: new Date(Date.now() + 30_000), scheduledRetryAttempt: 1 }).where(eq(heartbeatRuns.id, run!.id));
await createRunSecretRedactionRegistry(db).register(f.companyId, predecessorId, secret);
const list = await taskInterruptionPresentations(db, f.companyId, f.input);
const detail = await taskInterruptionPresentation(db, f.input[0]!);
for (const presentation of [list.get(f.input[0]!.id), detail]) {
expect(presentation?.observedRunId).toBe(run!.id);
expect(presentation?.providerFailure).toMatchObject({ httpStatus: 502 });
expect(presentation?.providerFailure?.message).toContain("Sandbox refused lease");
expect(presentation?.providerFailure?.message).toContain("REDACTED");
expect(JSON.stringify(presentation)).not.toContain(secret);
}
} finally { vi.unstubAllEnvs(); }
});
it.each(["decryption failure", "expanded message"])("keeps task reads available after provider display %s", async scenario => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
try {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const secret = "pw-short";
const providerFailure = { schema: "paperclip.provider-failure.v1", provider: "anthropic", phase: "startup",
category: "unknown", message: secret + "🚀".repeat(4000 - secret.length), retryability: "unknown" };
await db.update(heartbeatRuns).set({ resultJson: { providerFailure } }).where(eq(heartbeatRuns.id, run!.id));
await createRunSecretRedactionRegistry(db).register(f.companyId, run!.id, secret);
if (scenario === "decryption failure") vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "b".repeat(64));
const presentation = await taskInterruptionPresentation(db, f.input[0]!);
expect(presentation).toMatchObject({ state: "failed", canRetry: true });
if (scenario === "decryption failure") expect(presentation).not.toHaveProperty("providerFailure");
else {
expect(presentation?.providerFailure?.message).toContain("REDACTED");
expect([...presentation!.providerFailure!.message]).toHaveLength(4000);
}
expect(JSON.stringify(presentation)).not.toContain(secret);
} finally { vi.unstubAllEnvs(); }
});
it("does not inspect unrelated company history for a small issue page", async () => {
const f = await seedInterruptionFixture(db, { selected: 2, history: 3, unrelated: 1500, payloadBytes: 4096 });
const query = await captureInterruptionQuery(db, taskInterruptionPresentations, f.companyId, f.input);
const [explain] = await db.$client.unsafe(`explain (analyze, buffers, format json) ${query.sql}`, query.params as never[]);
const inspected: number[] = [];
const visit = (node: Record<string, unknown>) => {
if (node["Relation Name"] === "heartbeat_runs") inspected.push(
(Number(node["Actual Rows"] ?? 0) + Number(node["Rows Removed by Filter"] ?? 0)) * Number(node["Actual Loops"] ?? 1));
for (const child of node.Plans as Record<string, unknown>[] ?? []) visit(child);
};
visit(explain["QUERY PLAN"][0].Plan);
expect(inspected.length).toBeGreaterThan(0);
expect(Math.max(...inspected)).toBeLessThan(30);
await markFailuresFresh(f.companyId);
const presentations = await taskInterruptionPresentations(db, f.companyId, f.input);
for (const issue of f.input) expect(presentations.get(issue.id)).toMatchObject({ state: "failed", canRetry: true });
for (const issue of f.input) expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
companyId: f.companyId, taskId: issue.id, runId: presentations.get(issue.id)!.observedRunId,
runStatus: "failed", agentAdapter: "process",
}));
});
it("preserves native, first-class and text legacy precedence, ties, and tenant scope", async () => {
const companyId = randomUUID(), otherId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values([companyId, otherId].map(id => ({ id, name: "Selectors", issuePrefix: `S${id.slice(0, 7)}` })));
await db.insert(agents).values({ id: agentId, companyId, name: "Selector agent", adapterType: "process" });
const input = await db.insert(issues).values(Array.from({ length: 10 }, (_, n) => ({ companyId, title: `Selector ${n}`, status: "in_progress", assigneeAgentId: agentId }))).returning();
const base = { companyId, agentId, status: "failed", createdAt: new Date("2026-10-27T00:00:00Z") };
const winnerIds = Array.from({ length: 7 }, () => randomUUID());
await db.$client.unsafe("alter table heartbeat_runs disable trigger heartbeat_runs_set_scope_kind");
await db.insert(heartbeatRuns).values([
{ ...base, id: winnerIds[0], nativeIssueId: input[0]!.id, issueId: input[1]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[2]!.id } },
{ ...base, id: winnerIds[1], issueId: input[1]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[2]!.id } },
{ ...base, id: winnerIds[2], contextSnapshot: { issueId: input[2]!.id } },
{ ...base, nativeIssueId: randomUUID(), contextSnapshot: { issueId: input[3]!.id } },
{ ...base, nativeIssueId: null, issueId: input[9]!.id, scopeKind: "issue", contextSnapshot: { issueId: input[3]!.id } },
{ ...base, contextSnapshot: { issueId: input[3]!.id.toUpperCase() } },
...[null, "", [], 5, { issueId: "" }, { issueId: {} }, { issueId: [] }].map(contextSnapshot => ({ ...base, contextSnapshot: contextSnapshot as never })),
{ ...base, id: winnerIds[3], nativeIssueId: input[4]!.id, contextSnapshot: null },
{ ...base, id: winnerIds[4], issueId: input[5]!.id, scopeKind: "issue", contextSnapshot: [input[3]!.id] as never },
{ ...base, id: "00000000-0000-0000-0000-000000000001", contextSnapshot: { issueId: input[6]!.id } },
{ ...base, id: "ffffffff-ffff-ffff-ffff-ffffffffffff", nativeIssueId: input[6]!.id, contextSnapshot: { issueId: "" } },
{ ...base, companyId: otherId, nativeIssueId: input[3]!.id, createdAt: new Date("2026-10-28T00:00:00Z") },
]);
await db.$client.unsafe("alter table heartbeat_runs enable trigger heartbeat_runs_set_scope_kind");
const result = await taskInterruptionPresentations(db, companyId, [...input, input[0]!, { ...input[0]!, companyId: otherId }]);
for (let n = 0; n < 3; n++) expect(result.get(input[n]!.id)?.observedRunId).toBe(winnerIds[n]);
expect(result.get(input[3]!.id)).toBeNull();
expect(result.get(input[4]!.id)?.observedRunId).toBe(winnerIds[3]);
expect(result.get(input[5]!.id)?.observedRunId).toBe(winnerIds[4]);
expect(result.get(input[6]!.id)?.observedRunId).toBe("ffffffff-ffff-ffff-ffff-ffffffffffff");
expect(result.get(input[7]!.id)).toBeNull();
expect(result.get(input[8]!.id)).toBeNull();
const upper = { ...input[4]!, id: input[4]!.id.toUpperCase() };
expect((await taskInterruptionPresentations(db, companyId, [upper])).get(upper.id)).toBeNull();
expect(await taskInterruptionPresentations(db, companyId, [])).toEqual(new Map());
});
it("reports the saved-message failure state without reporting work, retries or user actions", async () => {
const companyId = randomUUID(), agentId = randomUUID(), pausedAgentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Failure states", issuePrefix: `F${companyId.slice(0, 7)}` });
await db.insert(agents).values([
{ id: agentId, companyId, name: "Worker", adapterType: "process" },
{ id: pausedAgentId, companyId, name: "Paused worker", adapterType: "process", status: "paused" },
]);
const input = await db.insert(issues).values(Array.from({ length: 9 }, (_, n) => ({
companyId, title: `State ${n}`, status: n === 5 ? "done" : "in_progress",
assigneeAgentId: n === 8 ? pausedAgentId : agentId,
}))).returning();
const runs = await db.insert(heartbeatRuns).values(input.map((issue, n) => ({
companyId, agentId: issue.assigneeAgentId!, issueId: issue.id,
status: n === 1 || n === 4 ? "succeeded" : n === 2 ? "running" : n === 7 ? "cancelled" : "failed",
errorCode: n === 6 ? "native_provider_approval_required" : null,
scheduledRetryAt: n === 3 ? new Date(Date.now() + 60_000) : null,
}))).returning();
const [wait] = await db.insert(agentWakeupRequests).values({
companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
requestedByActorType: "user", payload: { issueId: input[1]!.id }, requestedAt: new Date(Date.now() - 20_000),
}).returning();
captureTaskInterruptionFailure.mockClear();
const result = await taskInterruptionPresentations(db, companyId, [...input, { ...input[0]!, companyId: randomUUID() }]);
expect(captureTaskInterruptionFailure).toHaveBeenCalledTimes(3);
for (const n of [0, 1, 7]) {
expect(result.get(input[n]!.id)).toMatchObject({ responsibility: "system", state: "failed" });
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith({
companyId, agentId, taskId: input[n]!.id, runId: runs[n]!.id,
runStatus: runs[n]!.status, agentAdapter: "process", intentId: n === 1 ? wait!.id : null, runErrorCode: null,
});
}
expect(result.get(input[2]!.id)?.state).toBe("working");
expect(result.get(input[3]!.id)?.state).toBe("retry_scheduled");
for (const n of [4, 5]) expect(result.get(input[n]!.id)?.state).toBe("resolved");
for (const n of [6, 8]) expect(result.get(input[n]!.id)?.responsibility).toBe("user");
});
it("reports a retained failure once at its state change, not on every later process's read", async () => {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Retained failures", issuePrefix: `R${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "process" });
const input = await db.insert(issues).values(Array.from({ length: 3 }, (_, n) => ({
companyId, title: `Retained ${n}`, status: "in_progress", assigneeAgentId: agentId,
}))).returning();
const old = new Date(Date.now() - 22 * 60 * 60_000);
const runs = await db.insert(heartbeatRuns).values(input.map((issue, n) => ({
companyId, agentId, issueId: issue.id, status: "failed", createdAt: old,
finishedAt: n === 2 ? new Date(Date.now() - 60_000) : old,
errorCode: n === 0 ? "native_workspace_sync_out_retry_exhausted" : null,
}))).returning();
// A new saved message is a new state change for an old failed run.
const [wait] = await db.insert(agentWakeupRequests).values({
companyId, agentId, source: "on_demand", status: "deferred_issue_execution",
requestedByActorType: "user", payload: { issueId: input[1]!.id }, requestedAt: new Date(Date.now() - 20_000),
}).returning();
captureTaskInterruptionFailure.mockClear();
const result = await taskInterruptionPresentations(db, companyId, input);
for (const issue of input) expect(result.get(issue.id)).toMatchObject({ responsibility: "system", state: "failed" });
expect(captureTaskInterruptionFailure).toHaveBeenCalledTimes(2);
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
taskId: input[1]!.id, runId: runs[1]!.id, intentId: wait!.id }));
expect(captureTaskInterruptionFailure).toHaveBeenCalledWith(expect.objectContaining({
taskId: input[2]!.id, runId: runs[2]!.id, intentId: null }));
await db.update(heartbeatRuns).set({ finishedAt: new Date() }).where(eq(heartbeatRuns.id, runs[0]!.id));
captureTaskInterruptionFailure.mockClear();
await taskInterruptionPresentations(db, companyId, [input[0]!]);
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
taskId: input[0]!.id, runErrorCode: "native_workspace_sync_out_retry_exhausted" }));
});
it("reports recovery that newly fails on an old run once, then treats it as retained", async () => {
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Old run recovery", issuePrefix: `O${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "process" });
const [issue] = await db.insert(issues).values({ companyId, title: "Recovery", status: "in_progress",
assigneeAgentId: agentId }).returning();
const old = new Date(Date.now() - 22 * 60 * 60_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue!.id, status: "failed",
createdAt: old, finishedAt: old }).returning();
// Exhausted recovery clears nextAttemptAt; only the saved transition time is recent.
const exhausted = { ...createSystemRecovery(run!.id), state: "exhausted" as const, attempts: 6, retriesUsed: 5,
firstFailureAt: old.getTime(), deadlineAt: old.getTime() + 30 * 60_000, nextAttemptAt: null };
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue!.id,
kind: "workspace_recovery", status: "active", ownerType: "system", cause: "native_workspace_sync_out_failed",
fingerprint: `recovery:${run!.id}`, nextAction: "Recover saved files.", createdAt: old, updatedAt: new Date(),
evidence: { runId: run!.id, systemRecovery: exhausted } }).returning();
captureTaskInterruptionFailure.mockClear();
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
.toMatchObject({ responsibility: "system", state: "failed" });
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
taskId: issue!.id, runId: run!.id }));
await db.update(issueRecoveryActions).set({ updatedAt: old }).where(eq(issueRecoveryActions.id, action!.id));
captureTaskInterruptionFailure.mockClear();
await taskInterruptionPresentations(db, companyId, [issue!]);
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
});
it("reports a native resume that is newly exhausted on an old run once, then treats it as retained", async () => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Old native run", issuePrefix: `N${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Native worker", adapterType: "paperclip_runner" });
const [issue] = await db.insert(issues).values({ companyId, title: "Native resume", status: "in_progress",
assigneeAgentId: agentId }).returning();
const old = new Date(Date.now() - 22 * 60 * 60_000);
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue!.id, scopeKind: "issue",
status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", createdAt: old, finishedAt: old });
// Only the coordinator's saved terminal transition is recent.
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue!.id, runId, phase: "terminal_failure",
attempt: 3, nextAttemptAt: null, createdAt: old, updatedAt: new Date() });
captureTaskInterruptionFailure.mockClear();
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
.toMatchObject({ responsibility: "system", state: "failed" });
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
taskId: issue!.id, runId, runErrorCode: "native_session_interrupted" }));
await db.update(nativeRunFinalizations).set({ updatedAt: old }).where(eq(nativeRunFinalizations.runId, runId));
captureTaskInterruptionFailure.mockClear();
await taskInterruptionPresentations(db, companyId, [issue!]);
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
});
it.each(["failed", "cancelled"])("reports a delayed Retry that newly becomes %s, then ages out", async status => {
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), intentId = randomUUID(), actionId = randomUUID();
const old = new Date(Date.now() - 22 * 60 * 60_000), recent = new Date(Date.now() - 60_000);
await db.insert(companies).values({ id: companyId, name: "Delayed chat Retry", issuePrefix: "D" + companyId.slice(0, 7) });
await db.insert(agents).values({ id: agentId, companyId, name: "Chat worker", adapterType: "claude_local" });
const [issue] = await db.insert(issues).values({ companyId, title: "Deferred Retry", status: "in_progress",
assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId: issue!.id,
status: "failed", createdAt: old, finishedAt: old });
const [application] = await db.insert(toolApplications).values({ companyId, name: "Chat",
applicationKey: randomUUID(), type: "chat" }).returning();
const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: application!.id,
name: "Chat", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning();
const [endpoint] = await db.insert(chatEndpoints).values({ companyId, connectionId: connection!.id,
assignedAgentId: agentId, provider: "slack", publicId: randomUUID(), status: "active" }).returning();
const payload = { issueId: issue!.id, agentId, failedRunId: runId, taskRetryIntentId: intentId };
await db.insert(chatActions).values({ id: actionId, companyId, endpointId: endpoint!.id,
kind: "failed_run_retry", providerActionId: randomUUID(), status, payload, createdAt: old, updatedAt: recent });
await db.insert(agentWakeupRequests).values({ id: intentId, companyId, agentId, source: "on_demand",
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: old,
payload: { issueId: issue!.id, taskRetry: { sourceRunId: runId, chatActionId: actionId } } });
captureTaskInterruptionFailure.mockClear();
expect((await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id))
.toMatchObject({ responsibility: "system", state: "failed", nextRetryAt: null });
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
taskId: issue!.id, runId, intentId }));
await db.update(chatActions).set({ updatedAt: old }).where(eq(chatActions.id, actionId));
captureTaskInterruptionFailure.mockClear();
await taskInterruptionPresentations(db, companyId, [issue!]);
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
await db.update(chatActions).set({ updatedAt: recent, payload: { ...payload, failedRunId: randomUUID() } })
.where(eq(chatActions.id, actionId));
captureTaskInterruptionFailure.mockClear();
await taskInterruptionPresentations(db, companyId, [issue!]);
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
});
it("keeps the failed run's adapter after the task is reassigned", async () => {
const f = await seedInterruptionFixture(db, { selected: 1, history: 1, unrelated: 0, payloadBytes: 0 });
const [nextAgent] = await db.insert(agents).values({
companyId: f.companyId, name: "Next worker", adapterType: "codex_local",
}).returning();
const [reassigned] = await db.update(issues).set({ assigneeAgentId: nextAgent!.id })
.where(eq(issues.id, f.input[0]!.id)).returning();
await markFailuresFresh(f.companyId);
captureTaskInterruptionFailure.mockClear();
const result = await taskInterruptionPresentations(db, f.companyId, [reassigned!]);
expect(result.get(reassigned!.id)?.state).toBe("failed");
expect(captureTaskInterruptionFailure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
taskId: reassigned!.id, agentAdapter: "process",
}));
});
// N-03: ONB-1 on staging waited 16 minutes on an overdue workspace_busy
// retry. Nothing failed, yet the banner offered a Retry that cannot help.
it.each(["workspace_busy", "ai_connection_busy", "ai_connection_pool_wait", "runner_revision_wait"])(
"does not present an overdue %s wait as a failed attempt", async reason => {
const companyId = randomUUID(), agentId = randomUUID(), parentRunId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Waiting task", issuePrefix: `W${companyId.slice(0, 7)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Waiting agent", adapterType: "process" });
const [issue] = await db.insert(issues).values({ companyId, title: "Paperclip onboarding",
status: "in_progress", assigneeAgentId: agentId }).returning();
const now = Date.now();
await db.insert(heartbeatRuns).values({ id: parentRunId, companyId, agentId, issueId: issue!.id, scopeKind: "issue",
status: "cancelled", errorCode: reason, createdAt: new Date(now - 10 * 60_000), finishedAt: new Date(now - 9 * 60_000) });
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: issue!.id, scopeKind: "issue",
status: "scheduled_retry", retryOfRunId: parentRunId, scheduledRetryReason: reason, scheduledRetryAttempt: 2,
scheduledRetryAt: new Date(now - 5 * 60_000), createdAt: new Date(now - 9 * 60_000) });
captureTaskInterruptionFailure.mockClear();
const presentation = (await taskInterruptionPresentations(db, companyId, [issue!])).get(issue!.id);
expect(presentation).toMatchObject({ responsibility: "system", state: "working", canRetry: false });
expect(captureTaskInterruptionFailure).not.toHaveBeenCalled();
});
});