fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When Paperclip cannot resume a cut-off run exactly, the
fresh-continuation fallback records a continuation for the task. A
delivery pass later starts it as a run. Staging shows this pending state
can last 30 to 110 minutes for backlog items.
> - The fallback also decides how the next run is isolated from the old
writer. For `fresh_isolated_workspace`, the next run must prepare a new
workspace.
> - A person's Retry pressed while the continuation was recorded but not
yet delivered started its own run at once. That run did not carry the
recorded isolation, so it could use the shared directory next to a
writer that is not proven stopped. It also did not carry the
continuation's recovery context.
> - This pull request makes wake admission deliver the recorded
continuation as the person's run, under the task lock. It also folds a
saved Retry into a delivered continuation for every record kind.
> - The benefit is one run per interruption, whatever arrives first, and
the recorded isolation decision is never lost.

## Linked Issues or Issue Description

Refs #21348

**What happened?**
A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while
a fresh continuation was recorded but not delivered, was admitted at
once as an ordinary `task_retry` run. That run had no
`freshContinuationIsolation` and no `recoveryActionId`. On a projectless
task the run used the shared directory that the fallback had decided to
isolate from. The later delivery pass marked the continuation as
superseded by that run. The reproduction showed exactly one run, so the
reported "two runs" did not happen in that order. The isolation was lost
for all three record kinds tested. For `writer_ended` and native
records, a Retry saved before delivery was not folded into the delivered
continuation, because the fold only covered `fresh_isolated_workspace`.

**Expected behavior**
With a recorded, undelivered continuation, the next run admitted for the
task is that continuation. It has the recorded isolation and the
person's Retry is recorded as answered by it.

**Steps to reproduce**
1. Record a fresh continuation (`ensureFreshContinuation`, delivery
`recorded`) for a projectless task whose legacy writer is on another
host.
2. Call `requestTaskRetry` and then `resumeTaskRetry`.
3. Before this change: one `task_retry` run without
`freshContinuationIsolation` or `recoveryActionId`, while the record is
still `pending`.

## What Changed

- `fresh-continuation-retry-fold.ts`:
- `undeliveredContinuation` reads the task's pending fresh-continuation
record under the task lock (`for update`). It returns nothing when a run
of the task was created after the record.
- `undeliveredContinuationContext` gives the run the continuation's
`recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for
`fresh_isolated_workspace`, `freshContinuationIsolation`.
- `deliverContinuationAsRun` marks the record `delivered` to the new run
in the same transaction, settles its coalesced holds, and folds other
waiting Retries.
- `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry
into any delivered fresh continuation, not only an isolated one.
- `heartbeat/queue.ts` (wake admission, issue transaction): a
non-system, non-delivery wake with no execution blocker takes the
pending continuation. The run is inserted as before, then delivered as
the continuation. The fold after a delivery wake now applies to every
fresh-continuation kind. No change to ranking or to the claim loop.
- `execution-recovery-resolution.ts`: the delivery pass folds a saved
Retry for every fresh-continuation kind. The superseded-run fold keys on
`recoveryActionId`.
- The person's run keeps its `task_retry` wake reason and the user's
wake receipt, so the existing #21349 rules rank it as person work.
- Limits: the person-triggered delivery does not spend an automatic
attempt. It adds no `issue.continued_automatically` entry and no
scheduled transient retry, and it is not gated by the hourly cap or by
delivery-pass pacing. The record itself was already counted when it was
recorded. The agent's concurrency limit still applies at claim time.

## Verification

New file
`server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12
tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and
native (`remote_epoch`) records, through the real `requestTaskRetry` +
`resumeTaskRetry` path:
- A Retry during pending delivery gives exactly one run. That run is the
continuation, carries the recorded isolation (the workspace readiness
reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as
person work. Later delivery passes add nothing.
- Delivery first, then a saved Retry: one run, and the Retry is
coalesced. A Retry on the delivered run joins it through the busy path.
- A Retry and two delivery passes race (concurrent transactions): one
run.
- After the continuation's own failure, a Retry gets its own run.
- The delivery pass is held by pacing (backlog source, agent queue
full), and the hourly cap and the shared attempt ledger are exhausted.
The person's Retry still starts the run.

Red before the fix: 8 of 12 failed. Green after: 12 of 12.

Local runs, one file at a time:
- `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs
a person's Retry pressed before delivery…", a deadlock in `afterEach`
truncate. It passes alone, and it fails the same way on unmodified
master under this machine's load.
- `__tests__/recovery-wake-herd.test.ts`: 29 passed
- `__tests__/execution-control-work.test.ts`: 27 passed
- `__tests__/issue-recovery-actions.test.ts`: 112 passed
- `workspace-recovery-fallback.test.ts`: 19 passed
- `heartbeat/queue.test.ts`: 101 passed
- `task-retry.test.ts`: 117 passed;
`legacy-preparation-retry-hold.test.ts`: 5 passed
- `server` `tsc --noEmit`: clean

## Risks

- A person's message, chat input or manual wake that arrives while a
continuation is pending also becomes that continuation, with a fresh
session and the recorded isolation. Before, such a wake ran without
isolation and superseded the record. This is the same rule for every
admission path, and it closes the shared-directory gap there too.
- A wake that takes the context and is then deferred by a later gate can
keep the isolation marker in its saved context. If the record is
delivered meanwhile, that later run prepares a new isolated workspace.
That is the safe direction.
- No schema change and no migration. Nothing is added before turn start
for healthy runs: the new read runs only at wake admission.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent with tool use (shell, file edit,
test runs). Model ID `claude-opus-5-5`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-13 07:13:49 -06:00
1 parent 5ba26600ef
commit 3f030971dc
4 files changed
+277 -8

No files matched your search

@@ -534,7 +534,7 @@ export async function deliverReconciledExecutions(
.where(pendingDecision);
await settleCoalescedFreshContinuationHolds(db, { companyId: action.companyId, primaryId: action.id,
delivery: "delivered", continuationRunId: newer.id });
if (newer.contextSnapshot?.freshContinuationIsolation === action.id)
if (newer.contextSnapshot?.recoveryActionId === action.id)
await db.transaction(async tx => {
await tx.execute(sql`select id from issues where company_id = ${action.companyId} and id = ${task.id} for update`);
await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: action.companyId, issueId: task.id, runId: newer.id,
@@ -600,7 +600,7 @@ export async function deliverReconciledExecutions(
.where(pendingDecision);
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
delivery: "delivered", continuationRunId: run.id });
if ((action.evidence.freshContinuation as { isolation?: unknown } | undefined)?.isolation === "fresh_isolated_workspace")
if (fresh)
await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: action.companyId, issueId: task.id, runId: run.id,
sourceRunId: decision.runId });
});
@@ -0,0 +1,203 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
activityLog, agentWakeupRequests, agents, authUsers, companies, createDb, heartbeatRuns, issueRecoveryActions, issues,
nativeRunFinalizations,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { deliverReconciledExecutions } from "./execution-recovery-resolution.js";
import { FRESH_CONTINUATIONS_PER_HOUR, FRESH_CONTINUATION_SCHEMA, ensureFreshContinuation,
resetBacklogContinuationPacingForTests } from "./fresh-continuation-fallback.js";
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
import { heartbeatService } from "./heartbeat.js";
import { queuedRunDispatchClass } from "./heartbeat/queued-run-dispatch-order.js";
import { ASSIGNED_WORK_CONTINUED_ACTION } from "./recovery/task-automatic-attempts.js";
import { CONTINUATION_CAP } from "./recovery/assigned-work-invariant.js";
import { requestTaskRetry } from "./task-retry.js";
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
const KINDS = ["fresh_isolated_workspace", "writer_ended", "native"] as const;
type Kind = (typeof KINDS)[number];
const RECORDED_ISOLATION: Record<Kind, string> = {
fresh_isolated_workspace: "fresh_isolated_workspace", writer_ended: "writer_ended", native: "remote_epoch" };
suite("a person's Retry while a continuation waits for delivery", () => {
let db!: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let companyId = "", agentId = "", userId = "";
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-retry-delivery-");
db = createDb(temporary.connectionString);
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "FRD",
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId });
await db.insert(agents).values({ id: agentId, companyId, name: "Coder", role: "engineer", status: "idle",
adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } }, permissions: {} });
}, 60_000);
afterEach(async () => {
resetBacklogContinuationPacingForTests(db);
await db.execute(sql`truncate table issue_recovery_actions, native_run_finalizations, activity_log,
agent_wakeup_requests, heartbeat_runs, issues cascade`);
});
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
async function pendingContinuation(kind: Kind, endedAgoMs = 60_000) {
const issueId = randomUUID(), runId = randomUUID();
const finishedAt = new Date(Date.now() - endedAgoMs);
const native = kind === "native";
await db.insert(issues).values({ id: issueId, companyId, title: "Ship the release notes", status: "blocked",
assigneeAgentId: agentId, statusVersion: 4, blockedTransitionAt: finishedAt });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", issueId,
nativeIssueId: native ? issueId : null, runtimeMode: native ? "native" : "legacy",
status: "failed", errorCode: native ? "native_session_interrupted" : kind === "writer_ended" ? "process_lost" : "controller_lease_lost",
error: "cut off", startedAt: new Date(finishedAt.getTime() - 10 * 60_000), finishedAt,
createdAt: new Date(finishedAt.getTime() - 11 * 60_000), responsibleUserId: userId,
contextSnapshot: { issueId, wakeReason: "issue_assigned" },
...(native ? { nativePhase: "terminal_failure", runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "sandbox-1", remoteCwd: "/home/daytona/work", epoch: 3,
state: "prepared", snapshotId: null } } }
: { runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } } }),
...(kind === "fresh_isolated_workspace"
? { processPid: 4242, processHostHash: "replaced-server", processStartedAt: new Date(Date.now() - 600_000) } : {}) });
if (native) {
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "terminal_failure",
recoveryState: "blocked", failureCode: "native_continuation_requires_reconciliation", leaseOwner: null });
}
const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
status: "active", ownerType: "board", returnOwnerAgentId: agentId,
cause: native ? "native_continuation_requires_reconciliation" : "legacy_execution_requires_reconciliation",
fingerprint: `${native ? "native-continuation" : "legacy-execution"}:${runId}`, nextAction: "Held.", maxAttempts: 3,
createdAt: finishedAt, evidence: { runId, originalFailureCode: "cut_off", nativeFailureBlock: { runId, statusVersion: 4 } } })
.returning();
await db.update(issues).set({ projectId: null, projectWorkspaceId: null, executionWorkspaceId: null }).where(eq(issues.id, issueId));
expect(await ensureFreshContinuation(db, { companyId, runId, trigger: "recovery_refused" }))
.toMatchObject({ kind: "continued", delivery: "recorded", isolation: RECORDED_ISOLATION[kind] });
return { issueId, runId, actionId: action!.id, isolation: RECORDED_ISOLATION[kind] };
}
async function occupySlot() {
const [other] = await db.insert(issues).values({ companyId, title: "Other work", status: "in_progress", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "running", contextSnapshot: { issueId: other!.id } });
}
const taskRuns = (f: { issueId: string; runId: string }) => db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`, sql`${heartbeatRuns.id} <> ${f.runId}`));
const record = async (actionId: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, actionId)))[0]!;
const wake = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0]!;
const deliveryPass = () => deliverReconciledExecutions(db, heartbeatService(db).wakeup);
async function deliver() { await deliveryPass(); await deliveryPass(); }
async function requestRetry(issueId: string, observedRunId: string) {
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: userId, observedRunId,
idempotencyKey: randomUUID(), actorSource: "local_implicit" });
expect(receipt).toMatchObject({ accepted: true });
return receipt.requestId!;
}
async function pressRetry(f: { issueId: string; runId: string }) {
const requestId = await requestRetry(f.issueId, f.runId);
await heartbeatService(db).resumeTaskRetry(companyId, requestId);
return requestId;
}
async function expectSingleContinuationRun(f: Awaited<ReturnType<typeof pendingContinuation>>, requestId: string) {
const runs = await taskRuns(f);
expect(runs).toHaveLength(1);
const run = runs[0]!;
expect(run.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId, previousRunId: f.runId, forceFreshSession: true });
expect(run.retryOfRunId).toBe(f.runId);
if (f.isolation === "fresh_isolated_workspace") {
expect(run.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
expect(await readFreshWorkspaceReadiness(db, run)).toMatchObject({ mode: "fresh_isolated_workspace", retryIntentId: f.actionId });
} else expect(run.contextSnapshot?.freshContinuationIsolation).toBeUndefined();
expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: run.id });
expect((await record(f.actionId)).evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: run.id });
return run;
}
it.each(KINDS)("runs a Retry pressed before delivery as the recorded continuation itself, ranked as person work (%s)", async kind => {
const f = await pendingContinuation(kind);
await occupySlot();
const requestId = await pressRetry(f);
const run = await expectSingleContinuationRun(f, requestId);
expect(run.contextSnapshot).toMatchObject({ wakeReason: "task_retry", taskRetryRequestId: requestId });
const [runWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, run.wakeupRequestId!));
expect(runWake!.requestedByActorType).toBe("user");
expect(queuedRunDispatchClass({ id: run.id, createdAt: run.createdAt, wakeReason: "task_retry", requestedByUser: true,
sourceEndedAt: null, readinessRank: 1, priorityRank: 0 }, new Date())).toBe("person");
await deliver();
expect((await taskRuns(f)).map(row => row.id)).toEqual([run.id]);
}, 60_000);
it.each(KINDS)("answers a Retry saved before delivery with the delivered continuation, never a second run (%s)", async kind => {
const f = await pendingContinuation(kind);
await occupySlot();
const requestId = await requestRetry(f.issueId, f.runId);
await deliver();
await heartbeatService(db).resumeTaskRetry(companyId, requestId);
const runs = await taskRuns(f);
expect(runs).toHaveLength(1);
expect(runs[0]!.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId, wakeReason: "issue_recovery_action_restored" });
expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: runs[0]!.id });
const joined = await requestRetry(f.issueId, runs[0]!.id);
expect(await wake(joined)).toMatchObject({ status: "coalesced", runId: runs[0]!.id });
expect(await taskRuns(f)).toHaveLength(1);
}, 60_000);
it.each(KINDS)("starts exactly one run when a Retry and the delivery pass race (%s)", async kind => {
const f = await pendingContinuation(kind);
await occupySlot();
const requestId = await requestRetry(f.issueId, f.runId);
await Promise.all([heartbeatService(db).resumeTaskRetry(companyId, requestId), deliveryPass(), deliveryPass()]);
await deliver();
await heartbeatService(db).resumeTaskRetry(companyId, requestId);
const runs = await taskRuns(f);
expect(runs).toHaveLength(1);
expect(runs[0]!.contextSnapshot).toMatchObject({ recoveryActionId: f.actionId });
if (f.isolation === "fresh_isolated_workspace") expect(runs[0]!.contextSnapshot).toMatchObject({ freshContinuationIsolation: f.actionId });
expect(await wake(requestId)).toMatchObject({ status: "coalesced", runId: runs[0]!.id });
expect((await record(f.actionId)).evidence).toMatchObject({ continuationDelivery: "delivered", continuationRunId: runs[0]!.id });
}, 60_000);
it.each(["fresh_isolated_workspace", "writer_ended"] as const)("gives a Retry of the continuation's own failure its own run (%s)", async kind => {
const f = await pendingContinuation(kind);
await occupySlot();
await deliver();
const [continuation] = await taskRuns(f);
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed", startedAt: new Date(), finishedAt: new Date() })
.where(eq(heartbeatRuns.id, continuation!.id));
const requestId = await pressRetry({ issueId: f.issueId, runId: continuation!.id });
const retryWake = await wake(requestId);
expect(retryWake.runId, JSON.stringify(retryWake)).toBeTruthy();
expect(retryWake.runId).not.toBe(continuation!.id);
const [own] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retryWake.runId!));
expect(own!.contextSnapshot).toMatchObject({ wakeReason: "task_retry", previousRunId: continuation!.id });
expect(own!.contextSnapshot?.recoveryActionId).toBeUndefined();
expect((await record(f.actionId)).evidence).toMatchObject({ continuationRunId: continuation!.id });
}, 60_000);
it("starts the person's run while the delivery is held by pacing and the task's automatic attempts are spent", async () => {
const f = await pendingContinuation("fresh_isolated_workspace", 45 * 60_000);
const [other] = await db.insert(issues).values({ companyId, title: "Queued work", status: "todo", assigneeAgentId: agentId }).returning();
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: other!.id, status: "queued", contextSnapshot: { issueId: other!.id } });
const now = new Date();
for (let index = 0; index < FRESH_CONTINUATIONS_PER_HOUR; index += 1)
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog", status: "resolved",
ownerType: "system", cause: "fresh_continuation", fingerprint: `spent:${index}:${f.issueId}`, nextAction: "Continued.",
evidence: { freshContinuation: { schema: FRESH_CONTINUATION_SCHEMA, recordedAt: new Date(now.getTime() - 60_000).toISOString() },
continuationDelivery: "delivered" } });
for (let index = 0; index < CONTINUATION_CAP; index += 1)
await db.insert(activityLog).values({ companyId, actorType: "system", actorId: "assigned-work", action: ASSIGNED_WORK_CONTINUED_ACTION,
entityType: "issue", entityId: f.issueId });
await deliver();
expect(await taskRuns(f)).toHaveLength(0);
expect((await record(f.actionId)).evidence.continuationDelivery).toBe("pending");
const requestId = await pressRetry(f);
await expectSingleContinuationRun(f, requestId);
}, 60_000);
});
@@ -1,6 +1,7 @@
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { FRESH_CONTINUATION_SCHEMA, settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js";
const LIVE_CONTINUATION_STATUSES = ["queued", "running"];
@@ -29,3 +30,60 @@ export async function foldWaitingRetriesIntoContinuation(tx: Db, input: {
.where(and(eq(agentWakeupRequests.companyId, input.companyId), inArray(agentWakeupRequests.id, retries),
eq(agentWakeupRequests.status, "deferred_issue_execution")));
}
export async function foldWaitingRetriesIntoRecordedContinuation(tx: Db, input: {
companyId: string; issueId: string; runId: string; recoveryActionId: string | null; sourceRunId: string | null;
}) {
if (!input.recoveryActionId || !input.sourceRunId) return;
const [record] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
sql`${issueRecoveryActions.id}::text = ${input.recoveryActionId}`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${input.sourceRunId}`)).limit(1);
if (record) await foldWaitingRetriesIntoContinuation(tx, input);
}
export type UndeliveredContinuation = { recoveryActionId: string; sourceRunId: string; isolation: string | null };
/** Caller holds the task row lock. The task's recorded continuation that no run took yet: a run admitted now is that continuation. */
export async function undeliveredContinuation(tx: Db, input: { companyId: string; issueId: string }): Promise<UndeliveredContinuation | null> {
const [record] = await tx.select({ id: issueRecoveryActions.id, evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`))
.orderBy(desc(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`)).limit(1).for("update");
if (!record) return null;
const fresh = record.evidence.freshContinuation as { recordedAt?: unknown; isolation?: unknown } | undefined;
const sourceRunId = (record.evidence.executionReconciliation as { runId?: unknown } | undefined)?.runId;
if (typeof sourceRunId !== "string" || typeof fresh?.recordedAt !== "string") return null;
const [intervening] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), sql`${heartbeatRuns.id}::text <> ${sourceRunId}`,
or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}::text`),
sql`${heartbeatRuns.createdAt} >= ${fresh.recordedAt}::timestamptz`)).limit(1);
if (intervening) return null;
return { recoveryActionId: record.id, sourceRunId, isolation: typeof fresh.isolation === "string" ? fresh.isolation : null };
}
export function undeliveredContinuationContext(continuation: UndeliveredContinuation, context: Record<string, unknown>) {
return { recoveryActionId: continuation.recoveryActionId, retryOfRunId: continuation.sourceRunId, forceFreshSession: true,
...(typeof context.previousRunId === "string" ? {} : { previousRunId: continuation.sourceRunId }),
...(continuation.isolation === "fresh_isolated_workspace" ? { freshContinuationIsolation: continuation.recoveryActionId } : {}) };
}
/** In the admitting transaction: the new run is the recorded continuation, so the delivery pass finds nothing left to start. */
export async function deliverContinuationAsRun(tx: Db, input: { companyId: string; issueId: string; runId: string;
continuation: UndeliveredContinuation }) {
const { continuation } = input;
await tx.update(heartbeatRuns).set({ retryOfRunId: continuation.sourceRunId })
.where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
await tx.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
continuationDelivery: "delivered", continuationRunId: input.runId, continuationDeliveredByPersonRun: true })}::jsonb` })
.where(and(eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.id, continuation.recoveryActionId),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
await settleCoalescedFreshContinuationHolds(tx, { companyId: input.companyId, primaryId: continuation.recoveryActionId,
delivery: "delivered", continuationRunId: input.runId });
await foldWaitingRetriesIntoContinuation(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId,
sourceRunId: continuation.sourceRunId });
}
+12 -4
View File
@@ -4,7 +4,8 @@ import { verifyBlockerStallWake } from "../recovery/blocker-stall-wakeups.js";
import { CONTEXT_COMPACTION_WAKE_REASON, ContextCompactionError, isContextCompactionRun, readContextCompactionRequest } from "../context-compaction.js";
import { isProcessRunAdmissionClosed } from "./run-admission.js";
import { continueFreshInTransaction, hasPendingFreshContinuation, isFreshIsolatedContinuationWake, settleCoalescedFreshContinuationHolds } from "../fresh-continuation-fallback.js";
import { foldWaitingRetriesIntoContinuation } from "../fresh-continuation-retry-fold.js";
import { deliverContinuationAsRun, foldWaitingRetriesIntoContinuation, foldWaitingRetriesIntoRecordedContinuation, undeliveredContinuation,
undeliveredContinuationContext } from "../fresh-continuation-retry-fold.js";
import { refreshRunStartYieldIfStale } from "../run-start-handover.js";
import { deliveredQueuedCommentIds } from "../queued-comment-delivery.js";
import { hasAutomaticInteractionDeliveryHold, INTERACTION_DELIVERY_DENIED, recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "../agent-interaction-delivery.js";
@@ -4028,6 +4029,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
});
return { kind: "skipped" as const };
}
const continuationToDeliver = !systemWake && !executionReconciliationWake && !executionBlocker
? await undeliveredContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id }) : null;
if (continuationToDeliver)
Object.assign(enrichedContextSnapshot, undeliveredContinuationContext(continuationToDeliver, enrichedContextSnapshot));
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
@@ -5083,9 +5088,12 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
eq(issueRecoveryActions.id, String(enrichedContextSnapshot.recoveryActionId)),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`));
}
if (readNonEmptyString(enrichedContextSnapshot.freshContinuationIsolation))
await foldWaitingRetriesIntoContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id,
sourceRunId: readNonEmptyString(enrichedContextSnapshot.previousRunId) });
if (continuationToDeliver)
await deliverContinuationAsRun(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id, runId: newRun.id,
continuation: continuationToDeliver });
else if (reconciledSourceRunId)
await foldWaitingRetriesIntoRecordedContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id,
runId: newRun.id, recoveryActionId: readNonEmptyString(enrichedContextSnapshot.recoveryActionId), sourceRunId: reconciledSourceRunId });
await tx
.update(agentWakeupRequests)