Files
PaperClipAI/server/src/services/fresh-workspace-continuation.ts
T
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00

464 lines
34 KiB
TypeScript

import { lstat, realpath } from "node:fs/promises";
import path from "node:path";
import { and, eq, inArray, isNull, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, environmentLeases, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db";
import { verifiedWorkspaceRepairReceipt } from "./workspace-recovery-receipt.js";
import { persistActivity } from "./activity-log.js";
import { logger } from "../middleware/logger.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { readTaskRetryIntent } from "./task-retry.js";
import { z } from "zod";
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { hasPendingRunCleanup } from "./execution-blocker.js";
import { toExecutionWorkspace } from "./execution-workspaces.js";
import { workspaceGitOperationScheduler } from "./workspace-git-operation-scheduler.js";
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
import { isBoundIsolatedTaskDirectory } from "./isolated-task-directory.js";
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema } from "@paperclipai/shared";
import { readFailedPreparationSettlement } from "./failed-preparation-settlement.js";
import { taskRetryRunAncestry } from "./recovery-run-ancestry.js";
import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
import { nativeExecutionUsesTaskWorkspace } from "./heartbeat/external-provider.js";
export type ContinuationReadiness =
| { mode: "resume_preserved_workspace"; preservationReceiptId: string }
| { mode: "reuse_current_workspace"; retryIntentId: string; sourceRunId: string; workspaceId: string; cwd: string }
| { mode: "resume_stopped_native_workspace"; retryIntentId: string; sourceRunId: string; workspaceId: string; cwd: string }
| { mode: "reuse_issue_workspace"; retryIntentId: string; sourceRunId: string; workspaceId: string; cwd: string }
| { mode: "fresh_isolated_workspace"; retryIntentId: string; sourceRunId: string; retainedSourceIds: string[] };
const freshReadinessSchema = z.discriminatedUnion("mode", [z.object({
mode: z.literal("fresh_isolated_workspace"), retryIntentId: z.string().uuid(),
sourceRunId: z.string().uuid(), retainedSourceIds: z.array(z.string().uuid()),
}), z.object({
mode: z.literal("reuse_issue_workspace"), retryIntentId: z.string().uuid(),
sourceRunId: z.string().uuid(), workspaceId: z.string().uuid(), cwd: z.string().min(1),
})]);
const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"];
const terminalRunStatuses = ["failed", "cancelled", "timed_out", "succeeded", "interrupted"];
const reusableWorkspaceStatuses = ["active", "idle", "in_review"];
// Admission can hold the issue lock. All holds share one probe per workspace,
// and that probe has one short wall-clock budget, including scheduler queueing.
const RETAINED_WORKSPACE_GIT_BUDGET_MS = 5_000;
const processStartedAt = new Date(Date.now() - process.uptime() * 1_000);
/** Why another writer may still hold the workspace, or null. The successor's
* own claims do not count. A restart orphan is ignored only when all of these
* hold: the stopped source owns it, it started before that source settled, and
* it started before this server process, so this process cannot be running it.
* Operations with no run, of another run, or from this process still block. */
export async function workspaceHasOtherWriter(db: Db, input: {
companyId: string; workspaceId: string; successorRunId: string;
stoppedSource?: { id: string; finishedAt: Date | null } | null;
}) {
const [lease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.executionWorkspaceId, input.workspaceId),
or(isNull(environmentLeases.heartbeatRunId), ne(environmentLeases.heartbeatRunId, input.successorRunId)),
or(isNull(environmentLeases.releasedAt), inArray(environmentLeases.status, ["active", "pending_cleanup"]),
eq(environmentLeases.cleanupStatus, "failed")),
)).limit(1);
const source = input.stoppedSource?.finishedAt ? input.stoppedSource : null;
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.executionWorkspaceId, input.workspaceId),
eq(workspaceOperations.status, "running"),
or(isNull(workspaceOperations.heartbeatRunId), ne(workspaceOperations.heartbeatRunId, input.successorRunId)),
// coalesce keeps an operation with no run: NULL = id must not hide it.
source ? sql`not coalesce(${workspaceOperations.heartbeatRunId} = ${source.id}
and ${workspaceOperations.startedAt} <= ${source.finishedAt!.toISOString()}::timestamptz
and ${workspaceOperations.startedAt} < ${processStartedAt.toISOString()}::timestamptz, false)` : undefined,
)).limit(1);
return lease ? "lease_held" as const : operation ? "operation_running" as const : null;
}
/** A task-owned Git workspace must read cleanly from its own root. A broken
* worktree link must not resolve to an enclosing repository. */
export async function readableOwnGitRoot(workspace: typeof executionWorkspaces.$inferSelect, real: string) {
const marker = await lstat(path.join(real, ".git")).catch(() => null);
if (!marker && workspace.providerType !== "git_worktree") return true;
const signal = AbortSignal.timeout(RETAINED_WORKSPACE_GIT_BUDGET_MS);
const scan = (args: string[], operation: string) => workspaceGitOperationScheduler.run({
workspacePath: real, args, operation, cacheTtlMs: 0, bypassSingleFlight: true, signal,
timeoutMs: RETAINED_WORKSPACE_GIT_BUDGET_MS, fairnessKeys: [`company:${workspace.companyId}`, `workspace:${workspace.id}`],
});
try {
const root = (await scan(["rev-parse", "--show-toplevel"], "retry.retained_workspace_root")).stdout.trim();
if (!root || await realpath(root).catch(() => null) !== real) return false;
await scan(["--no-optional-locks", "status", "--porcelain=v1", "--untracked-files=no"], "retry.retained_workspace_status");
return true;
} catch {
return false;
}
}
/** Whether a workspace row is the task's own local directory, outside remote
* storage authority and not closed. Shared by reuse and the prior-files hint. */
export function isTaskOwnedLocalWorkspace(workspace: typeof executionWorkspaces.$inferSelect, issueId: string) {
return Boolean(workspace.cwd) && workspace.sourceIssueId === issueId && workspace.mode !== "shared_workspace" &&
reusableWorkspaceStatuses.includes(workspace.status) && ["local_fs", "git_worktree"].includes(workspace.providerType) &&
(workspace.metadata?.remoteWorkspace ?? null) === null;
}
/** A recovery generation that is scheduled or running may still export or
* repair the source tree. A settled projection is only status text. */
/** Workspace-export failures and restore repairs keep their own repair path;
* their directory is neither reused nor offered as a recovery source. */
export function sourceNeedsWorkspaceRepair(cause: string | null, resultJson: Record<string, unknown> | null) {
return Boolean((cause && workspaceExportCauses.includes(cause)) || resultJson?.workspaceRestoreFailure ||
hasRequiredWorkspaceRecovery(resultJson));
}
export function recoveryGenerationSettled(evidence: Record<string, unknown> | null, sourceRunId: string) {
if (evidence?.systemRecovery === undefined) return true;
const recovery = readSystemRecovery(evidence);
return Boolean(recovery && recovery.sourceRunId === sourceRunId && ["complete", "exhausted"].includes(recovery.state));
}
/** R2: a Retry that would otherwise start a fresh attempt keeps the task's own
* local workspace when that is provably safe, so uncommitted files and Git
* history stay with the task. Admission has already proven that the source
* executor stopped. This adds the workspace facts: it is the one local
* directory the source used, still bound to and owned by this task, with no
* other lease, running operation, unsettled recovery generation, export
* obligation or remote owner, and its Git state can still be read. Otherwise
* the caller keeps the fresh isolated attempt and the directory stays untouched.
* Pass one `gitProbes` map when checking several holds for the same run. */
export async function retainedIssueWorkspaceForRetry(db: Db, input: {
companyId: string; issueId: string; agentId: string; successorRunId: string; cause: string | null;
source: Pick<typeof heartbeatRuns.$inferSelect, "id" | "resultJson" | "status" | "finishedAt" | "controllerLeaseExpiresAt">;
evidence: Record<string, unknown> | null; gitProbes?: Map<string, Promise<boolean>>;
}): Promise<{ id: string; cwd: string } | null> {
const keepFresh = (reason: string) => {
logger.info({ companyId: input.companyId, issueId: input.issueId, sourceRunId: input.source.id,
successorRunId: input.successorRunId, reason }, "Retry keeps a fresh attempt; the task workspace is not reusable");
return null;
};
if (!recoveryGenerationSettled(input.evidence, input.source.id)) return keepFresh("recovery_generation_unsettled");
if (!terminalRunStatuses.includes(input.source.status) || !input.source.finishedAt ||
adapterExecutionControls.has(input.source.id) ||
(input.source.controllerLeaseExpiresAt && input.source.controllerLeaseExpiresAt > new Date())) return keepFresh("source_not_settled");
if (sourceNeedsWorkspaceRepair(input.cause, input.source.resultJson)) return keepFresh("workspace_repair_required");
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.heartbeatRunId, input.source.id),
));
if (!leases.length || leases.some(lease => lease.provider !== "local" || !lease.releasedAt ||
lease.cleanupStatus === "failed" || !lease.executionWorkspaceId)) return keepFresh("source_lease_not_local_released");
const sourceWorkspaceIds = new Set(leases.map(lease => lease.executionWorkspaceId));
const [bound] = await db.select({ workspace: executionWorkspaces }).from(issues).innerJoin(executionWorkspaces, and(
eq(executionWorkspaces.companyId, issues.companyId), eq(executionWorkspaces.id, issues.executionWorkspaceId),
)).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId),
eq(issues.assigneeAgentId, input.agentId))).limit(1);
const workspace = bound?.workspace;
if (!workspace?.cwd || sourceWorkspaceIds.size !== 1 || !sourceWorkspaceIds.has(workspace.id))
return keepFresh("binding_not_source_workspace");
if (!isTaskOwnedLocalWorkspace(workspace, input.issueId)) return keepFresh("workspace_not_task_owned_local");
const writer = await workspaceHasOtherWriter(db, { companyId: input.companyId, workspaceId: workspace.id,
successorRunId: input.successorRunId, stoppedSource: input.source });
if (writer) return keepFresh(writer);
// The directory must still be the recorded one, not a replacement link.
const stat = await lstat(workspace.cwd).catch(() => null);
const real = stat?.isDirectory() && !stat.isSymbolicLink() ? await realpath(workspace.cwd).catch(() => null) : null;
if (!real) return keepFresh("directory_missing");
let probe = input.gitProbes?.get(real);
if (!probe) {
probe = readableOwnGitRoot(workspace, real);
input.gitProbes?.set(real, probe);
}
if (!await probe) return keepFresh("git_unreadable");
return { id: workspace.id, cwd: workspace.cwd };
}
/** This chooses a strategy, never establishes termination or effect safety.
* Admission verifies those independently before it can persist this receipt. */
export async function canSelectFreshWorkspace(input: {
db: Db; companyId: string; issueId: string; agentId: string; actorId: string;
sourceRunId: string; evidence: Record<string, unknown>; taskRetryRequestId?: string;
queuedRequestId?: string;
}): Promise<string | null> {
const system = readSystemRecovery(input.evidence);
const requestId = input.taskRetryRequestId ?? input.queuedRequestId;
if (!requestId || !system || system.sourceRunId !== input.sourceRunId || system.freshAttemptRunId) return null;
if (system.providerCooldownAt !== null && system.providerCooldownAt > Date.now()) return null;
const [wake] = await input.db.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.id, requestId),
eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`,
));
if (!wake) return null;
if (input.taskRetryRequestId) {
const retry = readTaskRetryIntent(wake.payload);
if (retry?.generationId !== wake.id ||
retry.actorId !== input.actorId || wake.requestedByActorId !== input.actorId) return null;
if (retry.sourceRunId !== input.sourceRunId && (system.intentId !== wake.id ||
!(await taskRetryRunAncestry({ db: input.db, companyId: input.companyId, issueId: input.issueId,
agentId: input.agentId, sourceRunId: retry.sourceRunId }))?.includes(input.sourceRunId))) return null;
return wake.id;
}
const [source] = await input.db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.sourceRunId),
));
const recoveryPolicy = source?.context?.automaticRecovery as Record<string, unknown> | undefined;
if (recoveryPolicy?.allowFreshWorkspace === false) return null;
// Saved input can use the automatic fallback only after maintenance settled.
// A pending export keeps its current attempt; it cannot spawn another agent.
return ["complete", "exhausted"].includes(system.state) ? wake.id : null;
}
async function reusableDirectoryAfterPreparationFailure(db: Db, source: typeof heartbeatRuns.$inferSelect, issueId: string) {
const validation = source.resultJson?.workspaceValidation as Record<string, unknown> | undefined;
const recovery = source.resultJson?.executionRecovery as Record<string, unknown> | undefined;
if (source.status !== "failed" || !source.finishedAt || source.runtimeMode !== "legacy" ||
source.errorCode !== "workspace_validation_failed" || source.executionStage !== "preparing" ||
validation?.reason !== "git_worktree_base_not_git_checkout" || validation.issueId !== issueId ||
validation.resolvedWorkspaceSource !== "task_session" || validation.requestedExecutionWorkspaceMode !== "isolated_workspace" ||
validation.issueExecutionWorkspacePreference !== "reuse_existing" || recovery?.kind !== "bootstrap" ||
recovery.providerWorkStarted !== false || Object.keys(recovery).some(key => !["kind", "providerWorkStarted"].includes(key)) ||
source.resultJson?.stopReason !== "adapter_failed" ||
adapterExecutionControls.has(source.id) || (source.controllerLeaseExpiresAt && source.controllerLeaseExpiresAt > new Date()) ||
[source.runtimeModeResolvedAt, source.runtimeModeResolverVersion, source.runtimeModeReason, source.runnerInstanceId,
source.nativeIssueId, source.nativePhase, source.nativeSessionId, source.driverKind, source.driverVersion,
source.completionContractId, source.completionContractSha256, source.sessionIdAfter, source.externalRunId,
source.processPid, source.processGroupId, source.processStartedAt, source.logStore, source.logRef,
source.logBytes, source.logSha256, source.stdoutExcerpt, source.stderrExcerpt, source.lastOutputAt,
source.lastOutputStream, source.lastOutputBytes, source.exitCode, source.signal].some(value => value !== null) ||
source.lastOutputSeq !== 0 || source.logCompressed ||
Object.keys(source.usageJson ?? {}).some(key => key !== "ledgerScope") ||
Object.keys(source.runnerProfileJson ?? {}).some(key => !["adapterDispatch", "chatControlRecoveryAdmission", "aiConnectionNonAssigneeCommentWake"].includes(key)) ||
Object.keys(source.resultJson ?? {}).some(key => !["effectiveTimeoutSec", "executionRecovery", "stopReason", "timeoutConfigured", "timeoutFired", "timeoutSource", "workspaceValidation", "failureCause"].includes(key))) return null;
const settlement = await readFailedPreparationSettlement(db, source);
const warning = settlement?.unsupportedHarnessWarning;
if (!settlement || warning && (warning.message !== "Company observability was not applied: this harness is not supported."
|| !z.string().uuid().safeParse(warning.revision).success)) return null;
const [executionEvidence] = await db.execute<{ present: boolean }>(sql`select
exists(select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${source.companyId} and ${environmentLeases.heartbeatRunId} = ${source.id})
or exists(select 1 from ${nativeRunResults} where ${nativeRunResults.companyId} = ${source.companyId} and ${nativeRunResults.runId} = ${source.id})
or exists(select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.companyId} = ${source.companyId} and ${nativeRunFinalizations.runId} = ${source.id}) as present`);
if (executionEvidence?.present !== false) return null;
const [task] = await db.select().from(issues).where(and(eq(issues.companyId, source.companyId), eq(issues.id, issueId)));
if (!task || task.assigneeAgentId !== source.agentId || task.executionWorkspacePreference !== "reuse_existing" ||
task.executionWorkspaceId !== validation.issueExecutionWorkspaceId || task.projectId !== validation.issueProjectId ||
task.projectWorkspaceId !== validation.issueProjectWorkspaceId) return null;
const [workspace] = await db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, source.companyId), eq(executionWorkspaces.id, task.executionWorkspaceId!),
));
if (!workspace || workspace.cwd !== validation.resolvedWorkspaceCwd || !await isBoundIsolatedTaskDirectory({
companyId: source.companyId, issueId, workspaceId: task.executionWorkspaceId, workspace: toExecutionWorkspace(workspace) })) return null;
const prior = await db.select({ runId: environmentLeases.heartbeatRunId }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, source.companyId), eq(environmentLeases.executionWorkspaceId, workspace.id),
));
for (const { runId } of prior) if (runId && await hasPendingRunCleanup(db, source.companyId, runId)) return null;
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, source.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
eq(workspaceOperations.status, "running"),
)).limit(1);
return operation ? null : workspace;
}
async function stoppedNativeContinuationWorkspace(db: Db, source: typeof heartbeatRuns.$inferSelect,
input: { runId: string; issueId: string; retryIntentId: string }) {
if (source.runtimeMode !== "native" || source.status !== "failed" ||
source.errorCode !== "native_session_cleanup_quarantined" || !source.nativeSessionId || !source.runnerInstanceId) return null;
const [proof] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, source.companyId), eq(heartbeatRunEvents.runId, source.id),
eq(heartbeatRunEvents.agentId, source.agentId), eq(heartbeatRunEvents.eventType, "native.stopped_conversation_verified"),
isNull(heartbeatRunEvents.sourceEventId),
sql`${heartbeatRunEvents.payload}->>'successorRunId' = ${input.runId}`,
sql`${heartbeatRunEvents.payload}->>'taskRetryRequestId' = ${input.retryIntentId}`,
)).limit(1);
if (!proof) return null;
const parsed = z.object({ schema: z.literal("paperclip.stopped_native_conversation.v1"),
runId: z.literal(source.id), nativeSessionId: z.literal(source.nativeSessionId), runnerInstanceId: z.literal(source.runnerInstanceId),
executionWorkspaceId: z.string().uuid(), workspaceCwd: z.string().min(1),
}).safeParse(proof.payload);
if (!parsed.success) throw new Error("Verified native continuation workspace is unavailable");
if (parsed.data.executionWorkspaceId === source.id) return null;
const [bound] = await db.select({ id: executionWorkspaces.id, cwd: executionWorkspaces.cwd }).from(issues).innerJoin(executionWorkspaces, and(
eq(executionWorkspaces.companyId, issues.companyId), eq(executionWorkspaces.id, issues.executionWorkspaceId),
)).where(and(eq(issues.companyId, source.companyId), eq(issues.id, input.issueId),
eq(issues.assigneeAgentId, source.agentId), eq(executionWorkspaces.id, parsed.data.executionWorkspaceId),
eq(executionWorkspaces.cwd, parsed.data.workspaceCwd), eq(executionWorkspaces.status, "active"),
eq(executionWorkspaces.providerType, "local_fs"),
or(isNull(executionWorkspaces.sourceIssueId), eq(executionWorkspaces.sourceIssueId, input.issueId)),
)).limit(1);
if (!bound) throw new Error("Verified native continuation workspace binding changed");
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, source.companyId), eq(workspaceOperations.executionWorkspaceId, bound.id),
eq(workspaceOperations.status, "running"), or(isNull(workspaceOperations.heartbeatRunId), ne(workspaceOperations.heartbeatRunId, input.runId)),
)).limit(1);
const [lease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, source.companyId), eq(environmentLeases.executionWorkspaceId, bound.id),
or(isNull(environmentLeases.heartbeatRunId), ne(environmentLeases.heartbeatRunId, input.runId)),
or(isNull(environmentLeases.releasedAt), inArray(environmentLeases.status, ["active", "pending_cleanup"])),
)).limit(1);
if (operation || lease) throw new Error("Verified native continuation workspace cleanup is pending");
return { mode: "resume_stopped_native_workspace" as const, retryIntentId: input.retryIntentId,
sourceRunId: source.id, workspaceId: bound.id, cwd: parsed.data.workspaceCwd };
}
/** Preparation trusts the committed admission receipt, not a raw wake payload. */
export async function readFreshWorkspaceReadiness(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "issueId" | "nativeIssueId" | "contextSnapshot" | "runtimeMode" | "runnerProfileJson">): Promise<ContinuationReadiness | null> {
const issueId = run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId;
if (typeof issueId !== "string") return null;
const admittedInput = run.runnerProfileJson?.nativeExecutionInput;
if (run.runtimeMode === "native" && admittedInput !== undefined) {
const admitted = parseNativeExecutionInput(admittedInput);
if (admitted.binding.companyId !== run.companyId || admitted.binding.runId !== run.id
|| admitted.binding.agentId !== run.agentId || admitted.binding.issueId !== issueId) {
throw new Error("native_execution_workspace_admission_binding_mismatch");
}
if (!nativeExecutionUsesTaskWorkspace(admitted)) return null;
if (!z.string().uuid().safeParse(admitted.binding.executionWorkspaceId).success) {
throw new Error("native_execution_workspace_admission_missing");
}
const [workspace] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, run.companyId), eq(executionWorkspaces.id, admitted.binding.executionWorkspaceId),
)).limit(1);
if (workspace) return null;
if (admitted.binding.executionWorkspaceId !== run.id) throw new Error("native_execution_workspace_admission_missing");
}
const [action] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
sql`${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,runId}' = ${run.id}`,
sql`${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,readiness,mode}' in ('fresh_isolated_workspace', 'reuse_issue_workspace')`,
// Historical sibling holds also record the continuation, but only the
// recovery generation owns its committed fresh attempt. Select that proof.
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,freshAttemptRunId}' = ${run.id}`,
sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,sourceRunId}' = ${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,readiness,sourceRunId}'`,
sql`${issueRecoveryActions.evidence}->>'runId' = ${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,readiness,sourceRunId}'`,
)).limit(1);
if (!action) {
const [wake] = await db.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
eq(agentWakeupRequests.runId, run.id), eq(agentWakeupRequests.reason, "task_retry"),
eq(agentWakeupRequests.status, "coalesced"), eq(agentWakeupRequests.requestedByActorType, "user"),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
)).limit(1);
const intent = readTaskRetryIntent(wake?.payload);
if (!wake || !intent || intent.canonicalIntentId ||
intent.generationId !== wake.id || intent.actorId !== wake.requestedByActorId ||
run.contextSnapshot?.taskRetryRequestId !== wake.id ||
!["evaluate", "continue", "fresh_isolated_workspace"].includes(intent.strategy)) return null;
if (intent.chatActionId) {
const [chatWake] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId),
eq(agentWakeupRequests.id, intent.chatActionId), eq(agentWakeupRequests.runId, run.id),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
));
if (!chatWake || (run.contextSnapshot?.chatFailedRunRetry as Record<string, unknown> | undefined)?.actionId !== chatWake.id)
return null;
}
const [source] = await db.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.agentId, run.agentId),
eq(heartbeatRuns.id, intent.sourceRunId), sql`coalesce(${heartbeatRuns.nativeIssueId}::text,
${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueId}`,
)).limit(1);
if (!source) throw new Error("Fresh retry source changed");
if (intent.strategy !== "fresh_isolated_workspace") {
const stoppedWorkspace = await stoppedNativeContinuationWorkspace(db, source, { runId: run.id, issueId, retryIntentId: wake.id });
if (stoppedWorkspace) return stoppedWorkspace;
const preservationReceiptId = await verifiedWorkspaceRepairReceipt(db, source);
if (preservationReceiptId) return { mode: "resume_preserved_workspace", preservationReceiptId };
const [bound] = await db.select({ id: executionWorkspaces.id, metadata: executionWorkspaces.metadata }).from(issues)
.innerJoin(executionWorkspaces, and(eq(executionWorkspaces.companyId, issues.companyId),
eq(executionWorkspaces.id, issues.executionWorkspaceId)))
.where(and(eq(issues.companyId, run.companyId), eq(issues.id, issueId)));
const authority = readRemoteWorkspaceAuthority(bound?.metadata);
if (bound && authority && !authority.dirty && !authority.writerRunId && authority.snapshotId) {
const recovered = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId,
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId')`,
runnerProfileJson: sql<Record<string, unknown>>`jsonb_build_object('remoteWorkspace', ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace',
'nativeWorkspaceSync', ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync')`,
}).from(issueRecoveryActions)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId),
sql`${heartbeatRuns.id}::text = ${issueRecoveryActions.evidence}->>'runId'`))
.where(and(eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
eq(heartbeatRuns.agentId, run.agentId),
sql`${issueRecoveryActions.evidence}#>>'{explicitUserContinuation,runId}' = ${run.id}`,
sql`${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,workspaceId}' = ${bound.id}`,
sql`${heartbeatRuns.runnerProfileJson}#>>'{remoteWorkspace,snapshotId}' = ${authority.snapshotId}`)).limit(1);
for (const writer of recovered) {
const remote = remoteWorkspaceRunReferenceSchema.safeParse(writer.runnerProfileJson?.remoteWorkspace);
if (!remote.success || remote.data.state !== "saved" || remote.data.workspaceId !== bound.id
|| remote.data.epoch !== authority.epoch || remote.data.snapshotId !== authority.snapshotId) continue;
const receipt = await verifiedWorkspaceRepairReceipt(db, writer);
if (receipt) return { mode: "resume_preserved_workspace", preservationReceiptId: receipt };
}
}
const workspace = run.contextSnapshot?.previousRunId === source.id
? await reusableDirectoryAfterPreparationFailure(db, source, issueId) : null;
if (workspace?.cwd) {
const readiness: ContinuationReadiness = { mode: "reuse_current_workspace", retryIntentId: wake.id,
sourceRunId: source.id, workspaceId: workspace.id, cwd: workspace.cwd };
await db.transaction(async tx => {
const changed = await tx.update(agentWakeupRequests).set({
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"continue"'::jsonb)`,
}).where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.runId, run.id),
sql`${agentWakeupRequests.payload}#>>'{taskRetry,strategy}' = 'evaluate'`)).returning({ id: agentWakeupRequests.id });
if (changed.length) await persistActivity(tx as unknown as Db, { companyId: run.companyId,
actorType: "system", actorId: "task-recovery", action: "issue.retry_workspace_reused",
entityType: "issue", entityId: issueId, runId: run.id, details: readiness });
});
return readiness;
}
// R2: a Retry whose source has no admitted fresh receipt (for example a
// blocked recovery without a generation) also keeps a safe task workspace.
// Every hold recorded for the source must permit the reuse.
const holds = await db.select({ cause: issueRecoveryActions.cause, evidence: issueRecoveryActions.evidence })
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, run.companyId),
eq(issueRecoveryActions.sourceIssueId, issueId), sql`${issueRecoveryActions.evidence}->>'runId' = ${source.id}`));
let retained: { id: string; cwd: string } | null = null;
const gitProbes = new Map<string, Promise<boolean>>();
for (const hold of holds.length ? holds : [{ cause: null, evidence: null }]) {
const checked = await retainedIssueWorkspaceForRetry(db, { companyId: run.companyId, issueId,
agentId: run.agentId, successorRunId: run.id, cause: hold.cause, source, evidence: hold.evidence, gitProbes });
if (!checked || (retained && retained.id !== checked.id)) { retained = null; break; }
retained = checked;
}
if (retained) {
const readiness: ContinuationReadiness = { mode: "reuse_issue_workspace", retryIntentId: wake.id,
sourceRunId: source.id, workspaceId: retained.id, cwd: retained.cwd };
await db.transaction(async tx => {
const changed = await tx.update(agentWakeupRequests).set({
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"continue"'::jsonb)`,
}).where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.runId, run.id),
sql`${agentWakeupRequests.payload}#>>'{taskRetry,strategy}' = 'evaluate'`)).returning({ id: agentWakeupRequests.id });
if (changed.length) await persistActivity(tx as unknown as Db, { companyId: run.companyId,
actorType: "system", actorId: "task-recovery", action: "issue.retry_workspace_reused",
entityType: "issue", entityId: issueId, runId: run.id, details: readiness });
});
return readiness;
}
}
const leases = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, source.id),
));
const readiness: ContinuationReadiness = { mode: "fresh_isolated_workspace",
retryIntentId: wake.id, sourceRunId: source.id, retainedSourceIds: leases.map(lease => lease.id) };
await db.transaction(async tx => {
const changed = await tx.update(agentWakeupRequests).set({
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"fresh_isolated_workspace"'::jsonb)`,
}).where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.runId, run.id),
sql`${agentWakeupRequests.payload}#>>'{taskRetry,strategy}' in ('evaluate', 'continue')`)).returning({ id: agentWakeupRequests.id });
if (changed.length) await persistActivity(tx as unknown as Db, { companyId: run.companyId,
actorType: "system", actorId: "task-recovery", action: "issue.fresh_workspace_selected",
entityType: "issue", entityId: issueId, runId: run.id, details: { ...readiness } });
});
return readiness;
}
const authorization = action?.evidence.explicitUserContinuation as Record<string, unknown> | undefined;
const parsed = freshReadinessSchema.safeParse(authorization?.readiness);
if (!parsed.success) return null;
const readiness = parsed.data;
const recovery = readSystemRecovery(action.evidence);
if (action.evidence.runId !== readiness.sourceRunId || recovery?.sourceRunId !== readiness.sourceRunId ||
recovery.freshAttemptRunId !== run.id) return null;
const [wake] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.id, readiness.retryIntentId),
eq(agentWakeupRequests.agentId, run.agentId), eq(agentWakeupRequests.runId, run.id),
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
)).limit(1);
if (!wake) throw new Error("Fresh workspace admission lost its input receipt");
return readiness;
}