fix(recovery): settle a late workspace hold when a remote-epoch continuation already took over (#22387)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Restart recovery must continue a task after a server is killed. A
person must never need to press Retry.
> - When Paperclip cannot resume a cut-off run exactly, it records a
"fresh continuation". For a remote workspace, the continuation abandons
the old run's workspace epoch and restores the last save into a new
root. It does not apply the old run's unsaved files.
> - Workspace recovery can try to save the old run's files after that.
The epoch is already abandoned, so the source is "unavailable". Recovery
then wrote an active hold that asked a person to press Retry to continue
in a new workspace, which the continuation had already arranged.
> - That hold made the dispatch gate withhold every queued run of the
task, and the task rested on a Retry banner.
> - This pull request settles that hold into the recorded continuation
at the moment workspace recovery would leave it exhausted. Dispatch and
continuation delivery are unchanged.
> - The benefit is that the queued continuation starts automatically
after a hard kill, with no Retry banner.

## Linked Issues or Issue Description

**What happened?**

A restart-chaos release-swap wave hard-killed the old server. One task
(of 6) then rested on a Retry banner for about 5 minutes. The other 5
tasks passed. The sequence for the failed task:

1. A run started on the incoming server. The wave's injected database
delay held its process-identity write for 480 s, so its runner attach
timed out. The run saved a same-run resume.
2. Restart recovery found a newer queued run for the task (a
child-completion wake). It treated the same-run resume as superseded,
and the run ended `failed` with `native_restart_recovery_blocked`.
3. The dispatch gate withheld the queued run
(`execution_reconciliation_required`). In the same transaction, it
recorded a fresh continuation with `remote_epoch` isolation, which
abandoned the old run's remote workspace epoch.
4. 4 s later, workspace preservation for the old run found the abandoned
epoch (`source_unavailable`, `use_alternative_strategy`). It left an
active hold: "Paperclip cannot recover the unsaved files from the
previous run ... Retry continues the task in a new workspace". The
retained-source fallback also refused it later, with
`workspace_authority_changed`, because the continuation owned the epoch.
5. The gate withheld both bounded retries because of that hold. The
retries were spent, and the task showed a Retry banner.
6. A person pressed Retry. Retry started a run in a new workspace, which
the recorded continuation had already decided to do.

**Expected behavior**

The queued continuation starts automatically. No Retry banner shows.

**Steps to reproduce**

1. Run the restart-chaos K1 kill variant: 2 release swaps, and the old
server is hard-killed 120 s after SIGTERM.
2. Use an OpenCode task on a remote sandbox whose run starts just before
the second swap and that gets a child-completion wake while it runs.
3. Watch the task after the kill. Workspace recovery writes the
unsaved-files hold, the gate cancels each queued run with
`execution_reconciliation_required`, and the task rests on a Retry
banner.

`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`
reproduces the database state at step 5.

**Paperclip version or commit**

`master` at 25b5cad7f9 (wave) and c41145e72a (base of this branch).

## What Changed

- `server/src/services/fresh-continuation-fallback.ts`: new
`settleHoldIntoAbandonedEpochContinuation()`. It settles one
workspace-recovery hold with the existing `settleLateHolds()` when all
of these are true:
- the hold is active or escalated, and its saved-work recovery is
exhausted;
- no person owns it, it has no human-gate evidence, and it is not an
unsafe-archive hold;
- a fresh-continuation record for the same run has `remote_epoch`
isolation and delivery `pending`, `delivered`, `delegated` or
`not_requested`;
- durable proof shows the run's prepared epoch was abandoned
(`isAbandonedPreparedRemoteWorkspace`: an abandoned-epoch receipt for
this run, epoch and location, and an authority that moved past it).
The hold becomes `resolved` / `restored`
(`continued_fresh_from_last_saved`) and is coalesced into the
continuation record. Its files are not applied.
- `server/src/services/workspace-recovery.ts` calls it at two points:
when an attempt records an exhausted failure, and when a recovery pass
starts on an already exhausted hold (holds written before this change).
- `server/src/services/recovery/human-gate-evidence.ts`: one shared list
of human-gate evidence keys (`executionReconciliation`,
`OWNER_GONE_ATTESTATION_KEY`, `explicitUserContinuation`), with a SQL
helper and a TypeScript helper. Both treat a present key as a person's
decision, whatever its value. `isHumanGate`,
`completed-turn-cleanup-hold-settlement.ts` and
`interrupted-preparation-hold-settlement.ts` use them.
- New tests:
`server/src/__tests__/heartbeat-isolated-continuation-hold.test.ts`.

The dispatch gate, the execution-blocker predicate, the wake-queue lock
release, and continuation delivery are unchanged from `master`. An
earlier revision of this pull request changed them. That approach was
removed because each review round found another dispatch case.

`fresh_isolated_workspace` continuations are out of scope on purpose.
They do not abandon a remote epoch, so they do not cause this failure.
Only a run that carries the isolation marker is protected from the old
local directory, so a late hold for such a run keeps blocking.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-isolated-continuation-hold.test.ts` gives 20 of
20 passing.
- Recovery settles the late hold for each `remote_epoch` delivery state
(`pending`, `delivered`, `delegated`, `not_requested`). The blocker is
then empty, and the hold is coalesced into the continuation.
- Assembled heartbeat test: after recovery settles the hold, the queued
run is the only run that starts. The old run stays failed. With a hold
that a person owns, the gate still withholds the queued run.
- The hold stays for each of these cases: no continuation, a
`remote_epoch` continuation while the run still owns its epoch, a run
with no remote workspace, `fresh_isolated_workspace`, `writer_ended`,
`invalidated`, `coalesced`, a hold that a person owns, an unsafe
archive, each human-gate key, and a null-valued human-gate key.
- Red/green: with the two `workspace-recovery.ts` calls disabled, 5 of
20 fail; without the abandoned-epoch check, the 2 epoch cases fail: the
4 settle cases and the assembled start test.
- A focused regression run of the server test files for the blocker,
fresh continuations, workspace recovery, both hold settlements,
owner-gone attestation, and the run-dispatch and wake-queue adapters.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Only exhausted, system-owned workspace-recovery holds of a run whose
remote epoch a recorded continuation replaced are settled. All other
holds keep blocking.
- After settlement, workspace recovery stops for that hold. Its files
were already unreachable, because the epoch was abandoned.
- A human-gate key with a null or false value now counts as a person's
decision in TypeScript too. Before, TypeScript ignored it and SQL
counted it. This makes the two sides agree and is more conservative.
- No schema change, no migration, and no Runner change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edit, code search) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-20 04:34:53 -06:00
1 parent 414abc2dbf
commit fe0c4aa95d
7 files changed
+304 -9

No files matched your search

@@ -0,0 +1,250 @@
import { randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
agentWakeupRequests, agents, authUsers, companies, companyMemberships, createDb, executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues,
nativeRunFinalizations,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { getExecutionBlocker } from "../services/execution-blocker.ts";
import { settleHoldIntoAbandonedEpochContinuation } from "../services/fresh-continuation-fallback.ts";
import { HUMAN_GATE_EVIDENCE_KEYS, hasHumanGateEvidence } from "../services/recovery/human-gate-evidence.ts";
import { recoverRetainedWorkspace } from "../services/workspace-recovery.ts";
import { abandonUnverifiedRemoteWorkspaceEpoch } from "../services/remote-workspace-epoch-abandonment.ts";
import { heartbeatService } from "../services/heartbeat.ts";
import { runningProcesses } from "../adapters/index.ts";
import { createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../services/system-recovery-state.ts";
const held = vi.hoisted(() => {
const state = {
release: () => undefined as void,
released: Promise.resolve(),
reset() {
state.released = new Promise<void>((resolve) => { state.release = resolve; });
},
};
return state;
});
vi.mock("../adapters/index.ts", async () => {
const actual = await vi.importActual<typeof import("../adapters/index.ts")>("../adapters/index.ts");
return {
...actual,
getServerAdapter: vi.fn(() => ({
supportsLocalAgentJwt: false,
execute: async () => {
await held.released;
return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "done", provider: "test", model: "test" };
},
})),
};
});
const support = await getEmbeddedPostgresTestSupport();
const suite = support.supported ? describe : describe.skip;
const minutesAgo = (minutes: number) => new Date(Date.now() - minutes * 60_000);
const RESTORE_SCHEMA = "paperclip.workspace-restore-recovery.v1";
suite("a hold on a run that a fresh continuation already isolated", () => {
let db!: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-isolated-continuation-hold-");
db = createDb(temporary.connectionString);
}, 60_000);
afterEach(async () => {
held.release();
runningProcesses.clear();
await db.execute(sql`truncate table ${companies} cascade`);
});
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
type Shape = {
isolation?: "remote_epoch" | "fresh_isolated_workspace" | "writer_ended" | null;
personOwned?: boolean;
unsafeArchive?: boolean;
delivery?: "pending" | "delivered" | "delegated" | "invalidated" | "not_requested" | "coalesced";
holdEvidence?: Record<string, unknown>;
epoch?: "abandoned" | "still_owned" | "none";
};
async function killedRunAfterFreshContinuation(shape: Shape = {}) {
const { isolation = "remote_epoch", personOwned = false, unsafeArchive = false, delivery = "delivered", holdEvidence = {}, epoch = "abandoned" } = shape;
const userId = randomUUID(), companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
const sourceRunId = randomUUID(), continuationId = randomUUID(), holdId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Board", email: `${userId}@example.test`,
emailVerified: true, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companies).values({ id: companyId, name: "Kill wave", issuePrefix: `K${companyId.replaceAll("-", "").slice(0, 6)}`,
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: userId });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "idle",
adapterType: "codex_local", adapterConfig: {},
runtimeConfig: { heartbeat: { enabled: true, wakeOnDemand: true, maxConcurrentRuns: 1 } }, permissions: {} });
await db.insert(issues).values({ id: issueId, companyId, title: "Long task across a hard kill", status: "in_progress",
assigneeAgentId: agentId, createdByUserId: userId });
const recordedAt = minutesAgo(4);
const restoreRecovery = { schema: RESTORE_SCHEMA, leaseIds: [randomUUID()] };
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, nativeIssueId: issueId, scopeKind: "issue",
runtimeMode: "native", nativePhase: "terminal_failure", status: "failed", errorCode: "native_restart_recovery_blocked",
invocationSource: "automation", startedAt: minutesAgo(15), finishedAt: minutesAgo(5), createdAt: minutesAgo(16),
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_commented" },
resultJson: { workspaceRestoreFailure: unsafeArchive ? "restore_unsafe_archive" : "restore_failed",
workspaceRestoreRecovery: restoreRecovery,
...(isolation ? { freshContinuationSupersededAt: recordedAt.toISOString() } : {}) } });
if (epoch !== "none") {
const workspaceId = randomUUID(), leaseId = randomUUID(), snapshotId = randomUUID();
const location = { environmentId: randomUUID(), leaseId, providerLeaseId: `sandbox-${sourceRunId}`,
remoteCwd: `/home/daytona/paperclip-workspace/.paperclip-workspaces/${sourceRunId}` };
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, mode: "isolated_workspace",
strategyType: "project_primary", name: "Task workspace", status: "active", providerType: "cloud_sandbox",
cwd: `/workspaces/${workspaceId}`, sourceIssueId: issueId, metadata: { remoteWorkspace: {
schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId, source: { kind: "empty" },
writerRunId: sourceRunId, dirty: true, location } } });
await db.update(heartbeatRuns).set({ runnerProfileJson: { remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1",
workspaceId, leaseId, providerLeaseId: location.providerLeaseId, remoteCwd: location.remoteCwd, epoch: 1,
state: "prepared", snapshotId } } }).where(eq(heartbeatRuns.id, sourceRunId));
if (epoch === "abandoned") await db.transaction(tx => abandonUnverifiedRemoteWorkspaceEpoch(tx, { companyId, workspaceId,
runId: sourceRunId, epoch: 1, snapshotId, location }));
}
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: sourceRunId, phase: "terminal_failure",
attempt: 1, failureCode: "native_restart_recovery_blocked",
failureDetail: isolation ? { replacementDenied: "fresh_continuation" } : {} });
const queuedRunId = randomUUID(), wakeId = randomUUID();
if (isolation) {
const freshContinuation = { schema: "paperclip.fresh-continuation.v1", sourceRunId, trigger: "continuation_wake",
failureCode: "native_restart_recovery_blocked", delivery: "recorded", recordedAt: recordedAt.toISOString(), isolation,
stopProof: isolation === "writer_ended" ? "writer_ended" : "unproven_epoch_abandoned", unverifiedActions: [], createdObjects: [] };
const reconciliation = { runId: sourceRunId, providerStopped: true,
providerStopProof: isolation === "writer_ended" ? "writer_ended" : "isolated", actionOutcome: "mixed",
actorId: "fresh-continuation", recordedAt: recordedAt.toISOString() };
await db.insert(issueRecoveryActions).values({ id: continuationId, companyId, sourceIssueId: issueId,
kind: "active_run_watchdog", status: "resolved", outcome: "restored", ownerType: "system", returnOwnerAgentId: agentId,
cause: "native_continuation_requires_reconciliation", fingerprint: `native-continuation:${sourceRunId}`,
nextAction: "Paperclip continues the task in a fresh run from its last saved work.",
resolutionNote: "continued_fresh_from_last_saved", resolvedAt: recordedAt, createdAt: recordedAt, updatedAt: recordedAt,
evidence: { runId: sourceRunId, executionReconciliation: reconciliation, freshContinuation,
continuationDelivery: delivery,
...(delivery === "delivered" ? { continuationRunId: queuedRunId, continuationSupersededByRun: true } : {}) } });
}
const spent = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(sourceRunId, randomUUID()))!,
new SystemRecoveryFailure("source_unavailable", "use_alternative_strategy"), { now: Date.now() - 3 * 60_000 });
await db.insert(issueRecoveryActions).values({ id: holdId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
status: "active", ownerType: "board", ownerUserId: personOwned ? userId : null, returnOwnerAgentId: agentId,
cause: "native_continuation_requires_reconciliation", fingerprint: `workspace-restore:${sourceRunId}`,
nextAction: "Paperclip cannot recover the unsaved files from the previous run.",
createdAt: minutesAgo(3), updatedAt: minutesAgo(3),
evidence: { runId: sourceRunId, workspaceRestoreFailure: unsafeArchive ? "restore_unsafe_archive" : "restore_failed",
workspaceRestoreRecovery: restoreRecovery, systemRecovery: spent,
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: sourceRunId, status: "blocked", stage: "verifying_stop",
attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] }, ...holdEvidence } });
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId, agentId, source: "automation", triggerDetail: "system",
reason: "issue_children_completed", status: "queued", requestedByActorType: "system", requestedByActorId: "issue-children",
payload: { issueId } });
await db.insert(heartbeatRuns).values({ id: queuedRunId, companyId, agentId, issueId, invocationSource: "automation",
triggerDetail: "system", status: "queued", wakeupRequestId: wakeId, createdAt: minutesAgo(2),
scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure",
contextSnapshot: { issueId, taskId: issueId, wakeReason: "issue_children_completed" } });
await db.update(agentWakeupRequests).set({ runId: queuedRunId }).where(eq(agentWakeupRequests.id, wakeId));
return { userId, companyId, agentId, issueId, sourceRunId, continuationId, holdId, queuedRunId };
}
async function runRow(id: string) {
return (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!;
}
async function startedRunIds(companyId: string) {
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.startedAt} is not null`,
inArray(heartbeatRuns.status, ["running", "succeeded"])));
return rows.map(row => row.id);
}
async function holdRow(id: string) {
return (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
}
async function recoverHold(f: { companyId: string; issueId: string; sourceRunId: string; holdId: string }) {
await recoverRetainedWorkspace({ db, companyId: f.companyId, issueId: f.issueId, runId: f.sourceRunId, actionId: f.holdId,
environmentRuntime: {} as Parameters<typeof recoverRetainedWorkspace>[0]["environmentRuntime"] });
}
it.each(["pending", "delivered", "delegated", "not_requested"] as const)(
"workspace recovery settles its late hold into a %s remote_epoch continuation of the same run", async delivery => {
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch", delivery });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.holdId });
await recoverHold(f);
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
expect(await holdRow(f.holdId)).toMatchObject({ status: "resolved", outcome: "restored",
resolutionNote: "continued_fresh_from_last_saved", evidence: { coalescedIntoRecoveryActionId: f.continuationId } });
});
it.each([
["no continuation was recorded", { isolation: null }],
["the continuation is remote_epoch but the run still owns its epoch", { epoch: "still_owned" }],
["the continuation is remote_epoch but the run had no remote workspace", { epoch: "none" }],
["the continuation is a local fresh_isolated_workspace", { isolation: "fresh_isolated_workspace" }],
["the continuation reuses the old workspace", { isolation: "writer_ended" }],
["the continuation was invalidated", { delivery: "invalidated" }],
["the continuation was coalesced elsewhere", { delivery: "coalesced" }],
["a person owns the hold", { personOwned: true }],
["the saved archive is unsafe", { unsafeArchive: true }],
["the hold records an owner-gone attestation", { holdEvidence: { ownerGoneAttestation: { attestedBy: "board" } } }],
["the hold records an explicit continuation", { holdEvidence: { explicitUserContinuation: { runId: randomUUID() } } }],
["the hold records a reconciliation", { holdEvidence: { executionReconciliation: { runId: randomUUID() } } }],
["the hold records a null-valued owner-gone attestation", { holdEvidence: { ownerGoneAttestation: null } }],
] as const)("keeps the hold when %s", async (_name, shape) => {
const f = await killedRunAfterFreshContinuation(shape);
expect(await settleHoldIntoAbandonedEpochContinuation(db, { companyId: f.companyId, issueId: f.issueId,
runId: f.sourceRunId, actionId: f.holdId })).toBe(false);
expect(await holdRow(f.holdId)).toMatchObject({ status: "active" });
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ recoveryActionId: f.holdId });
});
it("treats a present human-gate key as a person's decision in both SQL and TypeScript, whatever its value", () => {
for (const key of HUMAN_GATE_EVIDENCE_KEYS) {
expect(hasHumanGateEvidence({ [key]: null })).toBe(true);
expect(hasHumanGateEvidence({ [key]: false })).toBe(true);
}
expect(hasHumanGateEvidence({})).toBe(false);
expect(hasHumanGateEvidence(null)).toBe(false);
});
it("starts the queued continuation at the execution gate once recovery settled its late hold", async () => {
held.reset();
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch" });
const heartbeat = heartbeatService(db);
try {
await recoverHold(f);
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
await vi.waitFor(async () => expect((await runRow(f.queuedRunId)).status).not.toBe("queued"), { timeout: 20_000 });
expect((await runRow(f.queuedRunId)).errorCode).not.toBe("execution_reconciliation_required");
expect(await startedRunIds(f.companyId)).toEqual([f.queuedRunId]);
expect(await runRow(f.sourceRunId)).toMatchObject({ status: "failed", errorCode: "native_restart_recovery_blocked" });
} finally {
held.release();
await heartbeat.drainActiveRunExecutions();
}
}, 60_000);
it("still withholds the queued run when a person owns the late hold", async () => {
held.reset();
const f = await killedRunAfterFreshContinuation({ isolation: "remote_epoch", personOwned: true });
const heartbeat = heartbeatService(db);
try {
await recoverHold(f);
await heartbeat.resumeQueuedRunsForAgent(f.companyId, f.agentId);
await vi.waitFor(async () => expect((await runRow(f.queuedRunId)).status).toBe("cancelled"), { timeout: 20_000 });
expect(await runRow(f.queuedRunId)).toMatchObject({ startedAt: null, errorCode: "execution_reconciliation_required" });
expect(await startedRunIds(f.companyId)).toEqual([]);
} finally {
held.release();
await heartbeat.drainActiveRunExecutions();
}
}, 60_000);
});
@@ -21,7 +21,9 @@ import { issueService } from "./issues.js";
import { holdOwnsBlockedStatus } from "./recovery-hold-blocked-status.js";
import { proveRunProcessGone } from "./run-process-owner.js";
import { hasPendingRunCleanup } from "./execution-blocker.js";
import { hasHumanGateEvidence } from "./recovery/human-gate-evidence.js";
import { abandonUnverifiedRemoteWorkspaceEpoch } from "./remote-workspace-epoch-abandonment.js";
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { readRunCancellation } from "./run-cancellation.js";
@@ -294,8 +296,7 @@ async function fenceSourceWorkspace(tx: Db, run: typeof heartbeatRuns.$inferSele
* gate again.
*/
export function isHumanGate(hold: Pick<typeof issueRecoveryActions.$inferSelect, "ownerUserId" | "evidence">): boolean {
return Boolean(hold.ownerUserId || hold.evidence.ownerGoneAttestation || hold.evidence.explicitUserContinuation
|| hold.evidence.executionReconciliation);
return Boolean(hold.ownerUserId) || hasHumanGateEvidence(hold.evidence);
}
const isRestoreHold = (hold: typeof issueRecoveryActions.$inferSelect) =>
@@ -702,6 +703,36 @@ export async function continueFreshInTransaction(tx: Db, input: Input): Promise<
return { kind: "continued", recoveryActionId: primaryId, delivery, isolation };
}
export async function settleHoldIntoAbandonedEpochContinuation(db: Db, input: {
companyId: string; issueId: string; runId: string; actionId: string; now?: Date;
}): Promise<boolean> {
const now = input.now ?? new Date();
return db.transaction(async raw => {
const tx = raw as unknown as Db;
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.id, input.runId)));
if (!issue || !run) return false;
const [hold] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
eq(issueRecoveryActions.sourceIssueId, issue.id), eq(issueRecoveryActions.id, input.actionId))).for("update");
if (!hold || !["active", "escalated"].includes(hold.status) || isHumanGate(hold) || hold.evidence.runId !== run.id
|| hold.evidence.workspaceRestoreFailure === "restore_unsafe_archive"
|| readSystemRecovery(hold.evidence)?.state !== "exhausted") return false;
const [prior] = await tx.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, input.companyId),
eq(issueRecoveryActions.sourceIssueId, issue.id), sql`${issueRecoveryActions.id} <> ${hold.id}`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,schema}' = ${FRESH_CONTINUATION_SCHEMA}`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,sourceRunId}' = ${run.id}`,
sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,isolation}' = 'remote_epoch'`,
inArray(sql<string>`${issueRecoveryActions.evidence}->>'continuationDelivery'`, ["pending", "delivered", "delegated", "not_requested"])))
.orderBy(desc(sql`${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}'`)).limit(1).for("update");
if (!prior || !await isAbandonedPreparedRemoteWorkspace(tx, run)) return false;
await settleLateHolds(tx, { companyId: input.companyId, issue, run, prior, holds: [hold], now });
logger.info({ runId: run.id, issueId: issue.id, recoveryActionId: hold.id, continuationId: prior.id },
"Workspace recovery hold settled into the continuation that already moved the task to a new workspace epoch");
return true;
});
}
async function settleLateHolds(tx: Db, input: { companyId: string; issue: typeof issues.$inferSelect; run: typeof heartbeatRuns.$inferSelect;
prior: typeof issueRecoveryActions.$inferSelect; holds: Array<typeof issueRecoveryActions.$inferSelect>; now: Date }) {
const { companyId, issue, run, prior, holds, now } = input;
@@ -5,7 +5,7 @@ import { LEGACY_RECOVERY_CAUSE, settleInterruptedNativeBootstrap } from "./legac
import { logger } from "../middleware/logger.js";
import type { OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
import { humanGateEvidencePredicate } from "./recovery/human-gate-evidence.js";
import { abandonedPreparedRemoteWorkspaceExists } from "./remote-workspace-abandoned-preparation.js";
/**
@@ -35,7 +35,7 @@ export function heldInterruptedPreparationWhere() {
and hold.status = 'resolved' and hold.outcome = 'blocked' and hold.owner_user_id is null
and hold.evidence->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'
and hold.evidence->'automaticRecovery'->>'replay' = 'blocked'
and not (hold.evidence ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])
and not ${humanGateEvidencePredicate(sql`hold.evidence`)}
and (not (hold.evidence ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
or ${abandonedPreparedRemoteWorkspaceExists()}))`,
sql`not exists (select 1 from environment_leases lease where lease.company_id = ${heartbeatRuns.companyId}
@@ -7,7 +7,7 @@ import { issueService } from "../issues.js";
import { logger } from "../../middleware/logger.js";
import type { InProgressRecord, SettleOutcome } from "../recovery/ownership-registry.js";
import type { OwnerProof } from "../owner-proof.js";
import { OWNER_GONE_ATTESTATION_KEY } from "../recovery/owner-gone-attestation.js";
import { humanGateEvidencePredicate } from "../recovery/human-gate-evidence.js";
import { holdOwnsBlockedStatus } from "../recovery-hold-blocked-status.js";
import { notifyRoutineExecutionStatus } from "../routine-execution-status-notifications.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
@@ -66,7 +66,7 @@ function settleableHoldCondition(runId: SQL | string): SQL {
isNull(issueRecoveryActions.ownerAgentId),
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'policy' = 'preserve_without_replay_v1'`,
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
sql`not (${issueRecoveryActions.evidence} ?| array['executionReconciliation', ${OWNER_GONE_ATTESTATION_KEY}, 'explicitUserContinuation'])`,
sql`not ${humanGateEvidencePredicate(issueRecoveryActions.evidence)}`,
sql`(not (${issueRecoveryActions.evidence} ?| array['workspaceRestoreFailure', 'workspaceRestoreRecovery'])
or ${completedWorkspaceRecoveryCondition(runId)})`,
)!;
@@ -0,0 +1,13 @@
import { sql, type SQL } from "drizzle-orm";
export const OWNER_GONE_ATTESTATION_KEY = "ownerGoneAttestation";
export const HUMAN_GATE_EVIDENCE_KEYS = ["executionReconciliation", OWNER_GONE_ATTESTATION_KEY, "explicitUserContinuation"] as const;
export function humanGateEvidencePredicate(evidence: SQL | unknown): SQL {
return sql`(${evidence} ?| array[${sql.join(HUMAN_GATE_EVIDENCE_KEYS.map(key => sql`${key}`), sql`, `)}])`;
}
export function hasHumanGateEvidence(evidence: Record<string, unknown> | null | undefined): boolean {
return Boolean(evidence) && HUMAN_GATE_EVIDENCE_KEYS.some(key => Object.hasOwn(evidence!, key));
}
@@ -9,8 +9,7 @@ import type { InProgressRecord, OwnableKind } from "./ownership-registry.js";
/** Recovery action cause: a stuck record whose owner Paperclip cannot judge. */
export const OWNER_UNPROVABLE_CAUSE = "ownership_owner_unprovable";
/** Lease metadata key the board's confirmation writes; the owner proof reads it as gone. */
export const OWNER_GONE_ATTESTATION_KEY = "ownerGoneAttestation";
export { OWNER_GONE_ATTESTATION_KEY } from "./human-gate-evidence.js";
/** Unprovable because the record names no usable owner: saved before owners were recorded. */
export const LEGACY_OWNER_REASONS: ReadonlySet<string> = new Set(["no_owner_stamp", "process_space_unrecorded", "missing_identity"]);
+3 -1
View File
@@ -41,7 +41,7 @@ import { capturePolicyRearmDueCondition, retainedSourceFallbackDueCondition, sto
import { rearmExhaustedCapturePolicy } from "./workspace-recovery-capture-policy.js";
import { rearmExhaustedStopProof } from "./workspace-recovery-stop-proof.js";
import { settleRecoveredWorkspaceRestoreHolds } from "./workspace-restore-hold-settlement.js";
import { expediteFreshContinuationChecks } from "./fresh-continuation-fallback.js";
import { expediteFreshContinuationChecks, settleHoldIntoAbandonedEpochContinuation } from "./fresh-continuation-fallback.js";
import { reusableSandboxNotKeptAfterReplacementSql } from "./reusable-sandbox-kept-marker.js";
const exportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
@@ -206,6 +206,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
const { db } = input;
await adoptSystemRecovery(db, input);
let initial = await loadSystemRecovery(db, input);
if (initial?.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
// A hold whose files a sibling saved, or whose source is refused permanently
// (including holds saved by older releases), settles without a person.
// An adoptable native source keeps its retention path; only the sibling
@@ -510,6 +511,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
...(cause ? { cause } : {}),
});
const recorded = await updateSystemRecovery(db, input, current!, failed);
if (recorded && failed.state === "exhausted" && await settleHoldIntoAbandonedEpochContinuation(db, input)) return;
// A permanent refusal of the recorded source repeats on every retry.
// Continue from the last save in this same pass, under this ownership.
if (recorded && failed.state === "exhausted" && isUnverifiableSourceFailure(failed.failure)) {