mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at2756be2f32## What Changed - `fresh-continuation-fallback.ts`, `interruptionEligibility`: the interruption counts as one the system continues when the failed run's cause is system-owned and a later run that the dispatch gate refused before it started is a system recovery run (`recoveryRunSql`, which excludes a person's Retry). A failed run that was itself a person's Retry (`task_retry` or `explicitUserContinuation`) never qualifies this way. The 24-hour recency bound is unchanged. The claim-gate hand-off, the backlog pass and its deadline query all use this one predicate. - `recovery/continuation-ledger.ts`, `recovery/task-automatic-attempts.ts`: one shared rule, `startedOrPendingAutomaticRetry`. An automatic retry that the dispatch gate cancelled before it started (`cancelled`, no `startedAt`, `execution_reconciliation_required`) does not count toward the hourly cap in either ledger. - `recovery/assigned-work-sweep.ts`: - When the latest run was refused at dispatch, the sweep continues from the run it was refused for through the fallback. Before, it gave up. - A blocked task with an active hold and a recent refused-at-dispatch latest run is now listed as a suspect. The check hands it to the fallback before it answers `left_to_hold_owner`. A pending recovery becomes `waiting_on_recovery` with its recheck time. - Holds that wait for a person's configuration or confirmation are still excluded. - Review follow-up (`recovery/task-automatic-attempts.ts`, `recovery/assigned-work-sweep.ts`): - The task ledger counts a sweep's fresh-continuation record (`trigger: assigned_work`) itself. It no longer depends on a companion activity entry, and an activity entry marked `continuedFromSavedWork` is not counted a second time. Every sweep continuation counts once, whatever path recorded it. - When the sweep claims a continuation, it marks its claim entry `continuedFromSavedWork` inside the transaction that records the fresh continuation. A restart between the two writes can no longer leave one attempt counted twice. - The sweep's continuation behind a refused run spends the per-minute repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is deferred with a `retryAt`. - `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's execution blocker offers no Retry, the escalation notice and the recovery action's next action leave out every Retry instruction. They say that Paperclip is holding the task until its saved work is settled. - No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold predicate in `run-dispatch/adapters/postgres.ts`, or `execution-blocker.ts`). The recorded continuation settles the hold, so the delivered run is not refused again. ## Verification Red before the change, green after (embedded Postgres, real services: `terminalizeLegacyExecution`, the dispatch adapter's `cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`, `deliverReconciledExecutions`, `continueHeldExecutions`, `recoveryService().assignedWork`): - `server/src/services/fresh-continuation-fallback.test.ts` - "continues a follow-up … when the saved work was recovered before the system retry was refused at claim": red (`continuationDelivery: "invalidated"`, no continuation). Green: the continuation is recorded at the claim, and the stranded reconciler does not block the task. The delivered run is queued and passes the claim gate (`not_stale`). - "… recovered after the system retry was refused at claim": red (no pending check). Green: the claim records `exact_recovery_pending` / `saved_work_recovery_running`. When the recovery completes, the backlog pass continues the task, and the delivered run passes the claim gate. - "continues a follow-up … with its files in place, after its system retry was refused at claim": red, then green. This is the control without a restore failure. - "still leaves a person's failed Retry of a follow-up for that person …": stays `person_owned` / `retry_needed`, before and after. - "counts only automatic retries that started, or still may, against the task's hourly budget": new. - `server/src/__tests__/failed-continuation-continues-again.test.ts`, "the sweep continues a blocked task whose system retry was refused at claim behind a recovered saved-work hold": red (the task was not a suspect). Green: `repaired`, a pending continuation in the hold, the task back in `todo`, and no wake enqueued. The next check reports a pending intent. - Same file, "a continuation the sweep records behind a refused run counts once toward the task's hourly cap": red (5 counted after 5 attempts plus the new continuation), green (6, and the transient-retry cap check refuses the next attempt). - Same file, "continuations behind refused runs spend the sweep's per-minute repair budget": 20 tasks. Red (20 continued in one pass), green (10 continue, 10 are deferred with a `retryAt`, and the 10 continue in the next pass). - Same file, "a sweep continuation interrupted right after it was recorded still counts once": the continuation commits, then the post-commit activity write fails (a simulated server stop). Red: 6 counted for 5 real attempts. Green: 5 counted, and the next automatic retry is still allowed. - `server/src/services/recovery/stranded-notice.test.ts`, "never tells a person to press Retry when the task offers no Retry": new. Local runs on this branch, one file at a time: - `fresh-continuation-fallback.test.ts`: 77 passed. The suite's `afterAll` teardown hit the 30 s hook timeout on a heavily loaded machine. - `failed-continuation-continues-again.test.ts`: 32 passed - `assigned-work-invariant.test.ts`: 31 passed - `stranded-notice.test.ts`: 33 passed - CI on the first head (ce4431410b): every required check passed - `server` `tsc --noEmit`: clean - `pnpm verify --explain` selects most of the repository (2666 files), so required CI supplies the broad coverage. ## Risks - More tasks continue automatically. Each continuation goes through the existing fallback. The 24-hour recency bound, the per-task hourly cap (6), isolation of the next run, and every person-owned gate (person-owned failure codes, a person's Stop, review, a person-owned hold, a person's failed Retry) are unchanged. - The hourly cap now ignores automatic retries that never started. A task whose retries are refused at dispatch again and again is still bounded: the refusal hands the hold to the fallback, and the fallback records one continuation per source run. - Known separate gap, not changed here: after an interrupted preparation's holds are resolved (`settleInterruptedNativeBootstrap`), `scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing follows that up. The assigned-work sweep may now pick such a task up through the refused-at-dispatch path, but this pull request does not prove that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
517 lines
36 KiB
TypeScript
517 lines
36 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { activityLog, agentWakeupRequests, agents, companies, createDb, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
|
|
import { recoveryService } from "../services/recovery/service.js";
|
|
import { ASSIGNED_WORK_CONTINUED_ACTION, ASSIGNED_WORK_REPAIRS_PER_MINUTE, assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js";
|
|
import { taskAutomaticAttempts, transientRetryWithinTaskCap } from "../services/recovery/task-automatic-attempts.js";
|
|
import { FRESH_CONTINUATION_SCHEMA, continueHeldExecutions, ensureFreshContinuation, resetBacklogContinuationPacingForTests } from "../services/fresh-continuation-fallback.js";
|
|
import { PERMANENT_FAILURE_CODES, PERSON_OWNED_FAILURE_CODES } from "../services/continuation-cause.js";
|
|
import { AUTOMATIC_RETRY_REASONS, SHARED_CONTINUATION_WINDOW_MS, sharedContinuationAttempts } from "../services/recovery/continuation-ledger.js";
|
|
import { SERVER_RESTART_RETRY_MAX_ATTEMPTS, SERVER_RESTART_RETRY_REASON, SERVER_RESTART_RETRY_WINDOW_MS } from "../services/execution-recovery-attempt.js";
|
|
import { FRESH_CONTINUATIONS_PER_HOUR } from "../services/fresh-continuation-fallback.js";
|
|
import { createSystemRecovery } from "../services/system-recovery-state.js";
|
|
import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
|
|
|
|
const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
|
|
if (!recoveryPostgresSupport.supported) console.warn(`Skipping failed continuation tests: ${recoveryPostgresSupport.reason}`);
|
|
|
|
const MINUTE = 60_000;
|
|
const HOUR = 60 * MINUTE;
|
|
const DAYTONA_TIMEOUT = "HTTP request timed out after 300000ms waiting for a response. This is a client-side deadline (DaytonaConfig.timeout)";
|
|
|
|
describe("shared continuation ledger", () => {
|
|
it("uses the server-restart lane's window, cap and retry reason", () => {
|
|
expect(SHARED_CONTINUATION_WINDOW_MS).toBe(SERVER_RESTART_RETRY_WINDOW_MS);
|
|
expect(FRESH_CONTINUATIONS_PER_HOUR).toBe(SERVER_RESTART_RETRY_MAX_ATTEMPTS);
|
|
expect(AUTOMATIC_RETRY_REASONS).toContain(SERVER_RESTART_RETRY_REASON);
|
|
});
|
|
});
|
|
|
|
describePostgres("a continuation that fails before doing any work is continued again", () => {
|
|
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startRecoveryPostgres("failed-continuation-");
|
|
db = createDb(temporary.connectionString);
|
|
}, 240_000);
|
|
|
|
afterAll(async () => {
|
|
await db?.$client.end({ timeout: 0 });
|
|
await temporary?.cleanup();
|
|
});
|
|
|
|
function wakeQueue() {
|
|
const calls: string[] = [];
|
|
const enqueueWakeup = vi.fn(async (agentId: string, opts?: { payload?: Record<string, unknown> | null; reason?: string | null }) => {
|
|
const issueId = String(opts?.payload?.issueId);
|
|
const [issue] = await db.select({ companyId: issues.companyId }).from(issues).where(eq(issues.id, issueId));
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: issue!.companyId, agentId, source: "automation",
|
|
triggerDetail: "system", reason: opts?.reason ?? null, payload: opts?.payload ?? null, status: "queued" }).returning();
|
|
calls.push(issueId);
|
|
return { id: wake!.id } as never;
|
|
});
|
|
return { calls, enqueueWakeup };
|
|
}
|
|
|
|
async function seed() {
|
|
const companyId = randomUUID(), agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Continuation fixture", issuePrefix: `C${companyId.slice(0, 7)}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Builder", adapterType: "paperclip_runner", status: "idle", adapterConfig: {} });
|
|
let number = 0;
|
|
async function task(values: Partial<typeof issues.$inferInsert> = {}) {
|
|
number += 1;
|
|
const [row] = await db.insert(issues).values({ companyId, title: `Task ${number}`, status: "in_progress", assigneeAgentId: agentId,
|
|
issueNumber: number, identifier: `C${companyId.slice(0, 7)}-${number}`, updatedAt: new Date(Date.now() - 2 * HOUR),
|
|
createdAt: new Date(Date.now() - 5 * HOUR), ...values }).returning();
|
|
return row!;
|
|
}
|
|
async function run(issueId: string, values: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
|
const finishedAt = values.finishedAt ?? new Date(Date.now() - 10 * MINUTE);
|
|
const [row] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "automation", status: "failed",
|
|
runtimeMode: "legacy", startedAt: new Date(finishedAt.getTime() - MINUTE), finishedAt,
|
|
createdAt: new Date(finishedAt.getTime() - 2 * MINUTE), livenessState: "failed",
|
|
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
|
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } }, ...values,
|
|
contextSnapshot: { issueId, taskId: issueId, ...(values.contextSnapshot ?? {}) } }).returning();
|
|
return row!;
|
|
}
|
|
async function continued(issueId: string, values: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
|
const source = await run(issueId, { errorCode: "native_restart_recovery_blocked", error: "native_interruption_deadline_exceeded",
|
|
finishedAt: new Date(Date.now() - 70 * MINUTE) });
|
|
const recordId = randomUUID();
|
|
const continuation = await run(issueId, { errorCode: "setup_failed", error: DAYTONA_TIMEOUT, retryOfRunId: source.id, ...values,
|
|
contextSnapshot: { wakeReason: "issue_recovery_action_restored", previousRunId: source.id, forceFreshSession: true,
|
|
freshContinuationIsolation: recordId, ...(values.contextSnapshot ?? {}) } });
|
|
await db.insert(issueRecoveryActions).values({ id: recordId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId, cause: "fresh_continuation",
|
|
fingerprint: `fresh-continuation:${source.id}`, nextAction: "Continuing.", resolutionNote: "continued_fresh_from_last_saved",
|
|
resolvedAt: new Date(Date.now() - 60 * MINUTE), attemptCount: 1,
|
|
evidence: { runId: source.id, continuationDelivery: "delivered", continuationRunId: continuation.id,
|
|
freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, sourceRunId: source.id, isolation: "remote_epoch",
|
|
recordedAt: new Date(Date.now() - 60 * MINUTE).toISOString() } } });
|
|
return continuation;
|
|
}
|
|
return { companyId, agentId, task, run, continued };
|
|
}
|
|
|
|
async function heldRun(f: Awaited<ReturnType<typeof seed>>, errorCode: string, contextSnapshot: Record<string, unknown>,
|
|
endedAgo = 2 * HOUR) {
|
|
const task = await f.task({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 30 * MINUTE), statusVersion: 4 });
|
|
const failed = await f.run(task.id, { runtimeMode: "native", nativeIssueId: task.id, issueId: task.id, errorCode, error: errorCode,
|
|
finishedAt: new Date(Date.now() - endedAgo), contextSnapshot });
|
|
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "blocked", resolvedAt: new Date(Date.now() - endedAgo), ownerType: "board", returnOwnerAgentId: f.agentId,
|
|
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${failed.id}`, nextAction: "Held.",
|
|
evidence: { runId: failed.id, originalFailureCode: errorCode, nativeFailureBlock: { runId: failed.id, statusVersion: 4 },
|
|
automaticRecovery: { policy: "preserve_without_replay_v1", runId: failed.id, replay: "blocked", actionOutcome: "unknown",
|
|
recordedAt: new Date(Date.now() - endedAgo).toISOString() } } }).returning();
|
|
return { task, failed, hold: hold! };
|
|
}
|
|
const recoveryContext = { wakeReason: "transient_failure_retry", retryReason: "transient_failure", previousRunId: randomUUID() };
|
|
const personRetryContext = { wakeReason: "task_retry", previousRunId: randomUUID(), explicitUserContinuation: { previousRunId: randomUUID() } };
|
|
|
|
const taskRow = async (id: string) => (await db.select().from(issues).where(eq(issues.id, id)))[0]!;
|
|
const activeStrandedHolds = async (issueId: string) => db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.kind, "stranded_assigned_issue"),
|
|
inArray(issueRecoveryActions.status, ["active", "escalated"])));
|
|
const continuationsOf = async (runId: string) => db.select().from(issueRecoveryActions)
|
|
.where(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${runId}`);
|
|
|
|
it("a recorded continuation whose run failed in setup is not escalated to the board, and the sweep continues it again from saved work", async () => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const failed = await f.continued(task.id);
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: task.id, skipRelatedRecovery: true });
|
|
expect((await taskRow(task.id)).status).toBe("in_progress");
|
|
expect(await activeStrandedHolds(task.id)).toHaveLength(0);
|
|
|
|
const check = await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id });
|
|
expect(check).toMatchObject({ outcome: "repaired", verdict: { kind: "violation" } });
|
|
const [record] = await continuationsOf(failed.id);
|
|
expect(record?.evidence).toMatchObject({ continuationDelivery: "pending", freshContinuation: { sourceRunId: failed.id } });
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
|
|
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent" } });
|
|
}, 60_000);
|
|
|
|
it("a todo task whose assignment recovery run failed in setup is continued, not blocked", async () => {
|
|
const f = await seed();
|
|
const task = await f.task({ status: "todo" });
|
|
const earlier = await f.run(task.id, { errorCode: "controller_lease_lost", finishedAt: new Date(Date.now() - 80 * MINUTE) });
|
|
const failed = await f.run(task.id, { errorCode: "setup_failed", error: DAYTONA_TIMEOUT, retryOfRunId: earlier.id,
|
|
contextSnapshot: { wakeReason: "issue_assignment_recovery", retryReason: "assignment_recovery" } });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: task.id, skipRelatedRecovery: true });
|
|
expect((await taskRow(task.id)).status).toBe("todo");
|
|
expect(await activeStrandedHolds(task.id)).toHaveLength(0);
|
|
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "repaired" });
|
|
expect(await continuationsOf(failed.id)).toHaveLength(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("a task an older release already escalated to the board for a failed continuation is continued on the first sweep after deploy", async () => {
|
|
const f = await seed();
|
|
const blockedAt = new Date(Date.now() - 20 * MINUTE);
|
|
const task = await f.task({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
const failed = await f.continued(task.id, { errorCode: "setup_failed",
|
|
error: "remote_workspace_sandbox_disk_full: the sandbox disk is full; free space in the sandbox before saving the work",
|
|
finishedAt: new Date(blockedAt.getTime() - 30_000) });
|
|
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "stranded_assigned_issue",
|
|
status: "active", ownerType: "board", cause: "stranded_assigned_issue",
|
|
fingerprint: `source_scoped_recovery:${f.companyId}:${task.id}:stranded_assigned_issue`,
|
|
evidence: { previousStatus: "in_progress", latestRunId: failed.id, latestRunErrorCode: "setup_failed",
|
|
routingPolicy: "board_escalation_no_takeover_v1" }, nextAction: "Board operator: inspect the evidence.", attemptCount: 1 }).returning();
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
|
|
entityType: "issue", entityId: task.id, details: { status: "blocked", previousStatus: "in_progress",
|
|
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup });
|
|
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(task.id);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { reason: "recovery_block" } });
|
|
expect((await taskRow(task.id)).status).not.toBe("blocked");
|
|
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, hold!.id)))[0])
|
|
.toMatchObject({ status: "resolved", ownerType: "system" });
|
|
expect(await continuationsOf(failed.id)).toHaveLength(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it.each([
|
|
["setup_failed", "Database query failed (40P01)"],
|
|
["adapter_failed", "native_remote_recovery_lease_mismatch: stage=runner_verification field=replacement_sandbox"],
|
|
["runner_remote_provider_artifact_incompatible", "provider pack verification failed (exitCode=1; timedOut=false)"],
|
|
])("the held backlog continues a recovery run that failed with %s, and still leaves an ordinary run's provider failure", async (errorCode, error) => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
async function held(contextSnapshot: Record<string, unknown>) {
|
|
const task = await f.task({ status: "blocked", blockedTransitionAt: new Date(Date.now() - 30 * MINUTE), statusVersion: 4 });
|
|
const failed = await f.run(task.id, { runtimeMode: "native", nativeIssueId: task.id, issueId: task.id, errorCode, error,
|
|
finishedAt: new Date(Date.now() - 2 * HOUR), contextSnapshot });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "blocked", resolvedAt: new Date(Date.now() - 2 * HOUR), ownerType: "board", returnOwnerAgentId: f.agentId,
|
|
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${failed.id}`, nextAction: "Held.",
|
|
evidence: { runId: failed.id, originalFailureCode: errorCode, nativeFailureBlock: { runId: failed.id, statusVersion: 4 },
|
|
automaticRecovery: { policy: "preserve_without_replay_v1", runId: failed.id, replay: "blocked", actionOutcome: "unknown",
|
|
recordedAt: new Date(Date.now() - 2 * HOUR).toISOString() } } });
|
|
return failed;
|
|
}
|
|
const recoveryRun = await held({ wakeReason: "transient_failure_retry", retryReason: "transient_failure", previousRunId: randomUUID() });
|
|
const ordinary = await held({ wakeReason: "issue_assigned" });
|
|
for (let pass = 0; pass < 3; pass += 1) await continueHeldExecutions(db, new Date());
|
|
expect(await continuationsOf(recoveryRun.id)).toHaveLength(1);
|
|
expect(await continuationsOf(ordinary.id)).toHaveLength(0);
|
|
}, 60_000);
|
|
|
|
it("a continuation run that failed for a reason a person owns is still escalated as before", async () => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
await f.continued(task.id, { errorCode: "provider_tool_definition_invalid", error: "The tool definition was rejected by the provider." });
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: task.id, skipRelatedRecovery: true });
|
|
expect((await taskRow(task.id)).status).toBe("blocked");
|
|
}, 60_000);
|
|
|
|
it("past the hourly cap the task waits for the next window, unblocked, and no attempt is spent", async () => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const failed = await f.continued(task.id);
|
|
for (let index = 0; index < 6; index += 1) {
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION,
|
|
entityType: "issue", entityId: task.id, details: { outcome: "queued" }, createdAt: new Date(Date.now() - (50 - index * 5) * MINUTE) });
|
|
}
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: task.id, skipRelatedRecovery: true });
|
|
const check = await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id });
|
|
expect(check).toMatchObject({ outcome: "deferred_cap" });
|
|
expect(check!.retryAt!.getTime()).toBeGreaterThan(Date.now());
|
|
expect((await taskRow(task.id)).status).toBe("in_progress");
|
|
expect(await continuationsOf(failed.id)).toHaveLength(0);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
it("a recovery run stopped during preparation with no hold left is continued once the wait bound passes, not waited on forever", async () => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const endedAt = new Date(Date.now() - 5 * MINUTE);
|
|
const stopped = await f.run(task.id, { status: "interrupted", errorCode: "server_shutdown_interrupted", executionStage: "preparing",
|
|
error: "Interrupted by graceful server shutdown (SIGTERM)", finishedAt: endedAt,
|
|
contextSnapshot: { wakeReason: "issue_assignment_recovery", retryReason: "assignment_recovery" } });
|
|
const queue = wakeQueue();
|
|
let clock = new Date();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, now: () => clock });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "waiting_on_recovery" });
|
|
expect(await continuationsOf(stopped.id)).toHaveLength(0);
|
|
for (const step of [3, 6, 12]) {
|
|
clock = new Date(Date.now() + step * MINUTE);
|
|
await sweep.checkIssue({ companyId: f.companyId, issueId: task.id });
|
|
}
|
|
expect(await continuationsOf(stopped.id)).toHaveLength(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
}, 60_000);
|
|
it.each([["20 minutes", 20 * MINUTE, "continued"], ["30 hours", 30 * HOUR, "retry_needed"]] as const)(
|
|
"a saved-work wait on a hold nobody records a check on ends in a named state; run ended %s ago, checked by two servers", async (_label, endedAgo, ending) => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const failed = await f.run(task.id, { errorCode: "setup_failed", error: DAYTONA_TIMEOUT, finishedAt: new Date(Date.now() - endedAgo),
|
|
contextSnapshot: { wakeReason: "transient_failure_retry", retryReason: "transient_failure" } });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "active_run_watchdog",
|
|
status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: f.agentId,
|
|
cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${failed.id}`, nextAction: "Recovering saved work.",
|
|
resolvedAt: new Date(Date.now() - endedAgo), evidence: { runId: failed.id, systemRecovery: createSystemRecovery(failed.id) } });
|
|
const queue = wakeQueue();
|
|
let clock = new Date();
|
|
const servers = [0, 1].map(() => assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, now: () => clock }));
|
|
for (const [index, step] of [0, 4, 9, 17, 33].entries()) {
|
|
clock = new Date(Date.now() + step * MINUTE);
|
|
await servers[index % 2]!.checkIssue({ companyId: f.companyId, issueId: task.id });
|
|
if ((await continuationsOf(failed.id)).length || queue.calls.length) break;
|
|
}
|
|
if (ending === "continued") {
|
|
expect(await continuationsOf(failed.id)).toHaveLength(1);
|
|
expect(queue.calls).toHaveLength(0);
|
|
} else {
|
|
expect(await continuationsOf(failed.id)).toHaveLength(0);
|
|
const [entry] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, task.id),
|
|
eq(activityLog.action, ASSIGNED_WORK_CONTINUED_ACTION)));
|
|
expect(entry?.details).toMatchObject({ outcome: "queued" });
|
|
expect(queue.calls).toHaveLength(1);
|
|
}
|
|
}, 60_000);
|
|
it("the held backlog leaves each permanent cause blocked", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const left = [];
|
|
for (const code of PERMANENT_FAILURE_CODES) left.push(await heldRun(f, code, recoveryContext));
|
|
for (let pass = 0; pass < 4; pass += 1) await continueHeldExecutions(db, new Date());
|
|
for (const entry of left) {
|
|
expect(await continuationsOf(entry.failed.id)).toHaveLength(0);
|
|
expect((await taskRow(entry.task.id)).status).toBe("blocked");
|
|
}
|
|
}, 120_000);
|
|
|
|
const runKinds = {
|
|
personRetry: personRetryContext,
|
|
automatic: recoveryContext,
|
|
firstAssignment: { wakeReason: "issue_assigned" },
|
|
} as const;
|
|
const causes = { setup: "setup_failed", restart: "process_lost", permanent: "low_trust_boundary_mismatch", person: "codex_auth_required" } as const;
|
|
const decision = (kind: keyof typeof runKinds, cause: keyof typeof causes) =>
|
|
cause === "restart" || (kind === "automatic" && cause === "setup");
|
|
it.each(Object.keys(runKinds).flatMap(kind => Object.keys(causes).map(cause => [kind, cause] as [keyof typeof runKinds, keyof typeof causes])))(
|
|
"every entry point makes the same decision for a %s run that failed with the %s cause", async (kind, cause) => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const errorCode = causes[cause];
|
|
const continues = decision(kind, cause);
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
|
|
const held = await heldRun(f, errorCode, runKinds[kind]);
|
|
await continueHeldExecutions(db, new Date());
|
|
expect(await continuationsOf(held.failed.id)).toHaveLength(continues ? 1 : 0);
|
|
|
|
const blockedAt = new Date(Date.now() - 20 * MINUTE);
|
|
const escalated = await f.task({ status: "blocked", blockedTransitionAt: blockedAt });
|
|
const escalatedRun = await f.run(escalated.id, { errorCode, error: errorCode, finishedAt: new Date(blockedAt.getTime() - 30_000),
|
|
contextSnapshot: runKinds[kind] });
|
|
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: escalated.id, kind: "stranded_assigned_issue",
|
|
status: "active", ownerType: "board", cause: "stranded_assigned_issue",
|
|
fingerprint: `source_scoped_recovery:${f.companyId}:${escalated.id}:stranded_assigned_issue`,
|
|
evidence: { previousStatus: "in_progress", latestRunId: escalatedRun.id, latestRunErrorCode: errorCode }, nextAction: "Held.", attemptCount: 1 });
|
|
await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system", action: "issue.updated",
|
|
entityType: "issue", entityId: escalated.id, details: { status: "blocked", previousStatus: "in_progress",
|
|
source: "recovery.reconcile_stranded_assigned_issue" }, createdAt: blockedAt });
|
|
const sweep = await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: escalated.id });
|
|
expect(sweep?.outcome === "left_to_hold_owner").toBe(!continues);
|
|
const fresh = await continuationsOf(escalatedRun.id);
|
|
if (continues && kind === "automatic") {
|
|
expect(fresh).toHaveLength(1);
|
|
expect(queue.calls).not.toContain(escalated.id);
|
|
}
|
|
if (!continues) {
|
|
expect(fresh).toHaveLength(0);
|
|
expect(queue.calls).not.toContain(escalated.id);
|
|
expect((await taskRow(escalated.id)).status).toBe("blocked");
|
|
}
|
|
|
|
if (kind === "automatic" && cause !== "restart") {
|
|
const open = await f.task();
|
|
const failed = await f.run(open.id, { errorCode, error: errorCode, contextSnapshot: runKinds[kind] });
|
|
await recovery.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: open.id, skipRelatedRecovery: true });
|
|
if (continues) {
|
|
expect((await taskRow(open.id)).status).toBe("in_progress");
|
|
await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: open.id });
|
|
expect(await continuationsOf(failed.id)).toHaveLength(1);
|
|
}
|
|
}
|
|
}, 120_000);
|
|
|
|
it("no entry point continues a person-owned or permanent cause", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
for (const code of [...PERSON_OWNED_FAILURE_CODES, ...PERMANENT_FAILURE_CODES]) {
|
|
const held = await heldRun(f, code, recoveryContext);
|
|
expect(await ensureFreshContinuation(db, { companyId: f.companyId, runId: held.failed.id, trigger: "held_recovery" }))
|
|
.toMatchObject({ kind: "person_owned" });
|
|
const task = await f.task();
|
|
const failed = await f.continued(task.id, { errorCode: code, error: code });
|
|
await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id });
|
|
expect(await continuationsOf(failed.id)).toHaveLength(0);
|
|
}
|
|
await continueHeldExecutions(db, new Date());
|
|
expect(await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, f.companyId),
|
|
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,trigger}' in ('held_recovery', 'assigned_work')`))).toHaveLength(0);
|
|
}, 240_000);
|
|
|
|
it("six automatic attempts of mixed kinds in the hour stop the held backlog until the next window", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const held = await heldRun(f, "setup_failed", recoveryContext);
|
|
const at = (minutesAgo: number) => new Date(Date.now() - minutesAgo * MINUTE);
|
|
for (const minutesAgo of [50, 40]) await f.run(held.task.id, { errorCode: "process_lost", scheduledRetryReason: "transient_failure",
|
|
createdAt: at(minutesAgo), finishedAt: at(minutesAgo - 1) });
|
|
for (const minutesAgo of [35, 30]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system", actorId: "system",
|
|
action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: held.task.id, details: { outcome: "queued" }, createdAt: at(minutesAgo) });
|
|
for (const minutesAgo of [25, 20]) await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: held.task.id,
|
|
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", cause: "fresh_continuation",
|
|
fingerprint: `fresh-continuation:${randomUUID()}`, nextAction: "Continuing.", resolvedAt: at(minutesAgo),
|
|
evidence: { runId: randomUUID(), continuationDelivery: "delivered", freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA,
|
|
trigger: "held_recovery", recordedAt: at(minutesAgo).toISOString() } } });
|
|
await db.update(heartbeatRuns).set({ createdAt: at(10), finishedAt: at(9) }).where(eq(heartbeatRuns.id, held.failed.id));
|
|
expect(await continueHeldExecutions(db, new Date())).toMatchObject({ continued: 0 });
|
|
const [checked] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, held.hold.id));
|
|
const nextCheckAt = Number((checked!.evidence.freshContinuationCheck as { nextCheckAt?: number }).nextCheckAt);
|
|
expect(nextCheckAt).toBeGreaterThan(Date.now() + 9 * MINUTE);
|
|
expect(nextCheckAt).toBeLessThanOrEqual(Date.now() + 10 * MINUTE + 5_000);
|
|
expect(await continuationsOf(held.failed.id)).toHaveLength(0);
|
|
expect(await continueHeldExecutions(db, new Date(Date.now() + 11 * MINUTE))).toMatchObject({ continued: 1 });
|
|
}, 60_000);
|
|
it("checks that only waited for exact recovery spend no attempt, so the fallback is reached at the bound", async () => {
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const stopped = await f.run(task.id, { status: "interrupted", errorCode: "server_shutdown_interrupted", executionStage: "preparing",
|
|
error: "Interrupted by graceful server shutdown (SIGTERM)", finishedAt: new Date(Date.now() - 2 * MINUTE),
|
|
contextSnapshot: { wakeReason: "issue_assignment_recovery", retryReason: "assignment_recovery" } });
|
|
for (const minutesAgo of [40, 36, 32, 28, 24]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system",
|
|
actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: task.id,
|
|
details: { outcome: "queued" }, createdAt: new Date(Date.now() - minutesAgo * MINUTE) });
|
|
const queue = wakeQueue();
|
|
let clock = new Date();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: queue.enqueueWakeup, now: () => clock });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "waiting_on_recovery" });
|
|
clock = new Date(Date.now() + 16 * MINUTE);
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "repaired" });
|
|
expect(await continuationsOf(stopped.id)).toHaveLength(1);
|
|
}, 60_000);
|
|
|
|
async function blockedBehindRefusedRun(f: Awaited<ReturnType<typeof seed>>) {
|
|
const blockedAt = new Date(Date.now() - 20 * MINUTE);
|
|
const task = await f.task({ status: "blocked", blockedTransitionAt: blockedAt, statusVersion: 3 });
|
|
const source = await f.run(task.id, { errorCode: "adapter_failed", error: "native_instruction_materialization_handoff_unavailable",
|
|
finishedAt: new Date(blockedAt.getTime() - 500), resultJson: { workspaceRestoreFailure: "restore_failed" },
|
|
contextSnapshot: { wakeReason: "issue_children_completed" } });
|
|
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task.id, kind: "active_run_watchdog",
|
|
status: "active", ownerType: "board", returnOwnerAgentId: f.agentId, cause: "legacy_execution_requires_reconciliation",
|
|
fingerprint: `legacy-execution:${source.id}`, nextAction: "Held.", createdAt: blockedAt,
|
|
evidence: { runId: source.id, originalFailureCode: "adapter_failed", workspaceRestoreFailure: "restore_failed",
|
|
systemRecovery: { ...createSystemRecovery(source.id), state: "complete" } } }).returning();
|
|
const refused = await f.run(task.id, { status: "cancelled", errorCode: "execution_reconciliation_required", startedAt: null,
|
|
retryOfRunId: source.id, finishedAt: new Date(blockedAt.getTime() + 1_000), createdAt: new Date(blockedAt.getTime() - 100), resultJson: {},
|
|
contextSnapshot: { wakeReason: "issue_continuation_needed", retryReason: "issue_continuation_needed" } });
|
|
return { task, source, hold: hold!, refused };
|
|
}
|
|
|
|
it("the sweep continues a blocked task whose system retry was refused at claim behind a recovered saved-work hold", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const { task, source, hold, refused } = await blockedBehindRefusedRun(f);
|
|
const queue = wakeQueue();
|
|
const recovery = recoveryService(db, { enqueueWakeup: queue.enqueueWakeup, scheduleRecoveryRetry: async () => null });
|
|
|
|
expect((await recovery.assignedWork.listSuspects({ companyId: f.companyId }, null, 100)).map(row => row.id)).toContain(task.id);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
|
|
.toMatchObject({ outcome: "repaired", verdict: { kind: "violation" } });
|
|
const [record] = await continuationsOf(source.id);
|
|
expect(record?.id).toBe(hold.id);
|
|
expect(record?.evidence).toMatchObject({ continuationDelivery: "pending", freshContinuation: { sourceRunId: source.id, trigger: "assigned_work" },
|
|
freshContinuationSettlement: { refusedAtDispatchRunId: refused.id, requestedBy: "system" } });
|
|
expect((await taskRow(task.id)).status).toBe("todo");
|
|
expect(queue.calls).toHaveLength(0);
|
|
expect(await recovery.assignedWork.checkIssue({ companyId: f.companyId, issueId: task.id }))
|
|
.toMatchObject({ outcome: "ok", verdict: { clause: "pending_intent" } });
|
|
}, 60_000);
|
|
|
|
it("a continuation the sweep records behind a refused run counts once toward the task's hourly cap", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const { task } = await blockedBehindRefusedRun(f);
|
|
for (const minutesAgo of [50, 45, 40, 35, 30]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system",
|
|
actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: task.id,
|
|
details: { outcome: "queued" }, createdAt: new Date(Date.now() - minutesAgo * MINUTE) });
|
|
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(true);
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup });
|
|
expect(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).toMatchObject({ outcome: "repaired" });
|
|
const attempts = await taskAutomaticAttempts(db, f.companyId, task.id, new Date());
|
|
expect(attempts.started).toHaveLength(FRESH_CONTINUATIONS_PER_HOUR);
|
|
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(false);
|
|
}, 60_000);
|
|
|
|
it("continuations behind refused runs spend the sweep's per-minute repair budget", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const tasks = [];
|
|
for (let index = 0; index < 20; index++) tasks.push((await blockedBehindRefusedRun(f)).task);
|
|
let clock = new Date();
|
|
const sweep = assignedWorkSweep(db, { enqueueWakeup: wakeQueue().enqueueWakeup, now: () => clock });
|
|
const first = [];
|
|
for (const task of tasks) first.push(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id }));
|
|
expect(first.filter(check => check?.outcome === "repaired")).toHaveLength(ASSIGNED_WORK_REPAIRS_PER_MINUTE);
|
|
const deferred = first.filter(check => check?.outcome === "deferred_rate_limit");
|
|
expect(deferred).toHaveLength(20 - ASSIGNED_WORK_REPAIRS_PER_MINUTE);
|
|
expect(deferred.every(check => check!.retryAt!.getTime() > clock.getTime())).toBe(true);
|
|
clock = new Date(clock.getTime() + 66_000);
|
|
const second = [];
|
|
for (const task of tasks) second.push(await sweep.checkIssue({ companyId: f.companyId, issueId: task.id }));
|
|
expect(second.filter(check => check?.outcome === "repaired")).toHaveLength(20 - ASSIGNED_WORK_REPAIRS_PER_MINUTE);
|
|
}, 120_000);
|
|
|
|
it("a sweep continuation interrupted right after it was recorded still counts once", async () => {
|
|
resetBacklogContinuationPacingForTests(db);
|
|
const f = await seed();
|
|
const task = await f.task();
|
|
const failed = await f.continued(task.id);
|
|
for (const minutesAgo of [50, 45, 40, 35]) await db.insert(activityLog).values({ companyId: f.companyId, actorType: "system",
|
|
actorId: "system", action: ASSIGNED_WORK_CONTINUED_ACTION, entityType: "issue", entityId: task.id,
|
|
details: { outcome: "queued" }, createdAt: new Date(Date.now() - minutesAgo * MINUTE) });
|
|
const before = (await taskAutomaticAttempts(db, f.companyId, task.id, new Date())).started.length;
|
|
const crashAfterCommit = new Proxy(db, { get(target, property) {
|
|
if (property === "update") return (table: unknown) => {
|
|
if (table === activityLog) throw new Error("server stopped");
|
|
return target.update(table as typeof activityLog);
|
|
};
|
|
const value = Reflect.get(target, property, target);
|
|
return typeof value === "function" ? value.bind(target) : value;
|
|
} });
|
|
const sweep = assignedWorkSweep(crashAfterCommit, { enqueueWakeup: wakeQueue().enqueueWakeup });
|
|
await expect(sweep.checkIssue({ companyId: f.companyId, issueId: task.id })).rejects.toThrow("server stopped");
|
|
expect(await continuationsOf(failed.id)).toHaveLength(1);
|
|
const attempts = await taskAutomaticAttempts(db, f.companyId, task.id, new Date());
|
|
expect(attempts.started).toHaveLength(before + 1);
|
|
expect(attempts.started).toHaveLength(5);
|
|
expect(await sharedContinuationAttempts(db, f.companyId, task.id, new Date())).toHaveLength(before + 1);
|
|
expect(await transientRetryWithinTaskCap(db, f.companyId, task.id, new Date())).toBe(true);
|
|
}, 60_000);
|
|
});
|