fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.

## Linked Issues or Issue Description

**What happened?**

On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.

- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.

**Expected behavior**

The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.

**Steps to reproduce**

1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.

Or:

1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.

**Paperclip version or commit**

Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).

**Deployment mode**

Hosted cloud (staging), remote Daytona sandbox runner.

**Agent adapter(s) involved**

`paperclip_runner` (legacy runtime runs, native agent).

Refs #21267, #21295.

## What Changed

- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.

## Verification

- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 00:52:25 -06:00
1 parent b555087f0f
commit bbafa78fe2
10 files changed
+546 -52

No files matched your search

+17 -11
View File
@@ -8,13 +8,14 @@ import { z } from "zod";
import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, toolConnections, workspaceOperations, type Db } from "@paperclipai/db"; import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js"; import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js";
import { canRetryStoppedRun } from "./cancelled-native-startup.js"; import { canRetryStoppedRun } from "./cancelled-native-startup.js";
import { heldRetryOffersRetry, heldTaskRetryRefusals } from "./task-retry-eligibility.js";
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js"; import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
import { EXECUTION_RECONCILIATION_CAUSES, readRunFailureCause, withoutSqlStatementText, type ExecutionBlocker } from "@paperclipai/shared"; import { EXECUTION_RECONCILIATION_CAUSES, readRunFailureCause, withoutSqlStatementText, type ExecutionBlocker } from "@paperclipai/shared";
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js"; import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js"; import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js";
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js"; import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, recoveredWorkMessage, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js"; import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, recoveredWorkMessage, savedWorkOffersRetry, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js"; import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js";
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js"; import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js";
@@ -312,6 +313,14 @@ function ownNativeResumePredicate(runId: string) {
and ${issueRecoveryActions.evidence}->>'runId' = ${runId}), false)`; and ${issueRecoveryActions.evidence}->>'runId' = ${runId}), false)`;
} }
export function restoreHoldPredicate() {
return or(
unreconciledNativeExportPredicate(),
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
);
}
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: {
conversationResetCommentId?: string | null; conversationResetCommentId?: string | null;
/** The native run being dispatched; its own scheduled resume does not block it. */ /** The native run being dispatched; its own scheduled resume does not block it. */
@@ -330,11 +339,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.sourceIssueId, issueId),
executionBlockerPredicate(), executionBlockerPredicate(),
not(restoreHoldReleasedForCurrentOwnerPredicate()), not(restoreHoldReleasedForCurrentOwnerPredicate()),
or( restoreHoldPredicate(),
unreconciledNativeExportPredicate(),
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
),
)).orderBy(desc(issueRecoveryActions.updatedAt)).limit(1); )).orderBy(desc(issueRecoveryActions.updatedAt)).limit(1);
// A persisted user /new is an ordered context command, not a retry of uncertain work. // A persisted user /new is an ordered context command, not a retry of uncertain work.
// The normal issue execution lock still serializes it behind any active turn. // The normal issue execution lock still serializes it behind any active turn.
@@ -411,8 +416,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
// hold to protect its files. Offer the same Retry that admission accepts, so // hold to protect its files. Offer the same Retry that admission accepts, so
// an infrastructure failure (such as a restart) is not left to a new message. // an infrastructure failure (such as a restart) is not left to a new message.
// An intentional Stop keeps "send a new message" as its only continuation. // An intentional Stop keeps "send a new message" as its only continuation.
const recoveredFailure = Boolean(restoreHold && (run?.status === "failed" || run?.status === "timed_out") && !workspaceRepairRequired && const recoveredFailure = Boolean(restoreHold && savedWorkOffersRetry(workspaceRecoveryState, run?.status, workspaceRepairRequired, action.evidence));
workspaceRecoveryState?.status === "recovered");
const canRetry = Boolean((!restoreHold || recoveredFailure) && run && conversation?.assigneeAgentId === run.agentId && const canRetry = Boolean((!restoreHold || recoveredFailure) && run && conversation?.assigneeAgentId === run.agentId &&
!["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run)); !["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run));
const runError = run && canRetry && isPreDispatchReviewWait(run) const runError = run && canRetry && isPreDispatchReviewWait(run)
@@ -435,14 +439,16 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
)).where(and( )).where(and(
eq(chatConversations.companyId, companyId), eq(chatConversations.issueId, issueId), eq(chatConversations.companyId, companyId), eq(chatConversations.issueId, issueId),
)).limit(1) : []; )).limit(1) : [];
const retryOffered = canRetry && (!chatBinding || recoveredFailure); const heldRefusal = canRetry ? (await heldTaskRetryRefusals(db, companyId, [issueId])).get(issueId) : undefined;
const retryOffered = canRetry && (!chatBinding || recoveredFailure) && heldRetryOffersRetry(heldRefusal);
const workspaceRecovery = workspaceRecoveryState?.status === "recovered" const workspaceRecovery = workspaceRecoveryState?.status === "recovered"
? { ...workspaceRecoveryState, message: recoveredWorkMessage(run?.status, retryOffered) } : workspaceRecoveryState; ? { ...workspaceRecoveryState, message: recoveredWorkMessage(run?.status, retryOffered) }
: workspaceRecoveryState && { ...workspaceRecoveryState, canRetry: workspaceRecoveryState.canRetry && retryOffered };
let nextAction = action.status === "active" && action.cause === EXECUTION_FINALIZATION_DEADLINE_EXCEEDED let nextAction = action.status === "active" && action.cause === EXECUTION_FINALIZATION_DEADLINE_EXCEEDED
? EXECUTION_FINALIZATION_DEADLINE_MESSAGE : action.nextAction; ? EXECUTION_FINALIZATION_DEADLINE_MESSAGE : action.nextAction;
// Retry is a person's control on this task. Say so, so an agent reading this // Retry is a person's control on this task. Say so, so an agent reading this
// record points the person to it instead of re-running the work itself. // record points the person to it instead of re-running the work itself.
if (canRetry && !eligibleContinuation) nextAction += ` ${EXECUTION_BLOCKER_PERSON_RETRY_ACTION}`; if (canRetry && heldRetryOffersRetry(heldRefusal) && !eligibleContinuation) nextAction += ` ${EXECUTION_BLOCKER_PERSON_RETRY_ACTION}`;
if (chatBinding && !restoreHold) { if (chatBinding && !restoreHold) {
if (chatBinding.state === "endpoint_removed" || !chatBinding.endpointStatus || chatBinding.endpointStatus === "archived") { if (chatBinding.state === "endpoint_removed" || !chatBinding.endpointStatus || chatBinding.endpointStatus === "archived") {
nextAction = "This chat connection was removed. Inspect the stopped run and create a new task to continue the work."; nextAction = "This chat connection was removed. Inspect the stopped run and create a new task to continue the work.";
@@ -77,6 +77,7 @@ const continuationRunFields = {
executionStage: heartbeatRuns.executionStage, executionStage: heartbeatRuns.executionStage,
}; };
const terminal = ["failed", "interrupted", "timed_out", "cancelled"]; const terminal = ["failed", "interrupted", "timed_out", "cancelled"];
const PREVIOUS_EXECUTION_UNVERIFIED = "Paperclip can't confirm that the previous attempt stopped before it started work, so Retry can't continue from it. Your message is saved.";
export function isUserCommentContinuationReason(reason: string | null | undefined): boolean { export function isUserCommentContinuationReason(reason: string | null | undefined): boolean {
return reason === "issue_commented" || reason === "issue_reopened_via_comment" || reason === "issue_assigned"; return reason === "issue_commented" || reason === "issue_reopened_via_comment" || reason === "issue_assigned";
@@ -193,6 +194,8 @@ export async function admitExplicitNativeContinuation(input: {
}): Promise<{ previousRunId: string; commentId: string | null; failedRunId?: string; taskRetryRequestId?: string; queuedCommentInterruptId?: string; readiness?: ContinuationReadiness } | null> { }): Promise<{ previousRunId: string; commentId: string | null; failedRunId?: string; taskRetryRequestId?: string; queuedCommentInterruptId?: string; readiness?: ContinuationReadiness } | null> {
const { db, companyId, issueId, agentId, actorId, commentId } = input; const { db, companyId, issueId, agentId, actorId, commentId } = input;
const blocked = (reason: string, message: string) => { input.onBlocked?.(reason, message); return null; }; const blocked = (reason: string, message: string) => { input.onBlocked?.(reason, message); return null; };
let blocker: Awaited<ReturnType<typeof getExecutionBlocker>> = null;
const refused = (reason: string, message: string) => blocker ? blocked(reason, message) : null;
if (input.actorType !== "user" || !actorId) return null; if (input.actorType !== "user" || !actorId) return null;
const retry = input.reason === "retry_failed_run" && const retry = input.reason === "retry_failed_run" &&
z.string().guid().safeParse(input.failedRunId).success; z.string().guid().safeParse(input.failedRunId).success;
@@ -280,7 +283,7 @@ export async function admitExplicitNativeContinuation(input: {
const authorizedAt = nativeQueuedSend ? interruptRequestedAt! const authorizedAt = nativeQueuedSend ? interruptRequestedAt!
: response?.comment.createdAt ?? comment?.createdAt ?? retryWake?.requestedAt ?? new Date(); : response?.comment.createdAt ?? comment?.createdAt ?? retryWake?.requestedAt ?? new Date();
if (!input.dryRun) await preserveTaskRemoteWorkspaceWriter(db, companyId, issueId); if (!input.dryRun) await preserveTaskRemoteWorkspaceWriter(db, companyId, issueId);
const blocker = await getExecutionBlocker(db, companyId, issueId); blocker = await getExecutionBlocker(db, companyId, issueId);
const [boundary] = await db.select({ createdAt: issueComments.createdAt }).from(issues).innerJoin(issueComments, and( const [boundary] = await db.select({ createdAt: issueComments.createdAt }).from(issues).innerJoin(issueComments, and(
eq(issueComments.companyId, issues.companyId), eq(issueComments.issueId, issues.id), eq(issueComments.companyId, issues.companyId), eq(issueComments.issueId, issues.id),
eq(issueComments.id, issues.conversationBoundaryCommentId), eq(issueComments.id, issues.conversationBoundaryCommentId),
@@ -349,7 +352,7 @@ export async function admitExplicitNativeContinuation(input: {
)).for("update"); )).for("update");
if (!source || (source.nativeIssueId ?? source.issueId ?? source.contextSnapshot?.issueId) !== issueId if (!source || (source.nativeIssueId ?? source.issueId ?? source.contextSnapshot?.issueId) !== issueId
|| adapterExecutionControls.has(source.id) || || adapterExecutionControls.has(source.id) ||
!(await isFailedNativePreparation(db, source, coordinator) || await isPreDispatchExecutionWaitVerified(db, source))) return null; !(await isFailedNativePreparation(db, source, coordinator) || await isPreDispatchExecutionWaitVerified(db, source))) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
sourceChecks.push({ cause: "legacy_execution_requires_reconciliation", evidence: { runId: source.id } }); sourceChecks.push({ cause: "legacy_execution_requires_reconciliation", evidence: { runId: source.id } });
} }
if (!sourceChecks.length && !nativeQueuedSend) return null; if (!sourceChecks.length && !nativeQueuedSend) return null;
@@ -385,7 +388,7 @@ export async function admitExplicitNativeContinuation(input: {
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, run.id),
)).for("update"); )).for("update");
if (!lockedRun || lockedRun.status !== run.status || lockedRun.agentId !== run.agentId || if (!lockedRun || lockedRun.status !== run.status || lockedRun.agentId !== run.agentId ||
lockedRun.finishedAt?.getTime() !== run.finishedAt.getTime()) return null; lockedRun.finishedAt?.getTime() !== run.finishedAt.getTime()) return refused("execution_settling", "Waiting for the previous run to stop. Your message will start automatically.");
run = lockedRun; run = lockedRun;
// A user's message written after their Stop request is new direction even // A user's message written after their Stop request is new direction even
// when the stopped native run settled later. // when the stopped native run settled later.
@@ -421,12 +424,16 @@ export async function admitExplicitNativeContinuation(input: {
// a conversation. Those adapters retain their reconciliation contract. // a conversation. Those adapters retain their reconciliation contract.
if (historicalAdapter && !isConversationAdapter(historicalAdapter)) { if (historicalAdapter && !isConversationAdapter(historicalAdapter)) {
if (!(taskRetry && run.id !== taskRetry.sourceRunId && freshIntent && agentIsNative && if (!(taskRetry && run.id !== taskRetry.sourceRunId && freshIntent && agentIsNative &&
historicalAdapter === "paperclip_runner" && run.status === "failed") || historicalAdapter === "paperclip_runner" && run.status === "failed"))
run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date() || return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
await hasPendingRunCleanup(db, companyId, run.id) || if (run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date())
!await isFailedNativePreparationRetryAncestor(db, run, coordinator)) return null; return refused("execution_settling", "Waiting for the previous run to stop. Your message will start automatically.");
if (await hasPendingRunCleanup(db, companyId, run.id))
return refused("cleanup_pending", "Waiting for the previous run to finish cleanup. Your message is saved.");
if (!await isFailedNativePreparationRetryAncestor(db, run, coordinator))
return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
} }
if (!historicalAdapter && agent.adapterType === "paperclip_runner") return null; if (!historicalAdapter && agent.adapterType === "paperclip_runner") return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
} }
const interruptedByThisClick = queuedInterrupt && const interruptedByThisClick = queuedInterrupt &&
interruptIntent?.targetRunId === run.id && run.resultJson?.operatorInterrupted === true && interruptIntent?.targetRunId === run.id && run.resultJson?.operatorInterrupted === true &&
@@ -511,7 +518,7 @@ export async function admitExplicitNativeContinuation(input: {
if (retry && run.status === "cancelled" && !canContinueCancelledRun(run) && !cancelledStartup) if (retry && run.status === "cancelled" && !canContinueCancelledRun(run) && !cancelledStartup)
return blocked("cancelled_by_operator", "Inspect the stopped run and send a new message to continue."); return blocked("cancelled_by_operator", "Inspect the stopped run and send a new message to continue.");
if (cancelledStartup) cancelledStartupIds.add(run.id); if (cancelledStartup) cancelledStartupIds.add(run.id);
if (run.runtimeMode !== "native" && !unusedAdmission && !legacyUserTurn && !cancelledStartup) return null; if (run.runtimeMode !== "native" && !unusedAdmission && !legacyUserTurn && !cancelledStartup) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
// A provider failure can finish the normal result/assessment commit path. // A provider failure can finish the normal result/assessment commit path.
// Its accepted failed result is immutable history, not a live controller. // Its accepted failed result is immutable history, not a live controller.
// Only a new user turn may pass this gate; the process/lease stop proofs // Only a new user turn may pass this gate; the process/lease stop proofs
@@ -559,7 +566,7 @@ export async function admitExplicitNativeContinuation(input: {
if (!leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease))) return blocked("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically."); if (!leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease))) return blocked("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically.");
if (run.runtimeMode === "native" && !input.dryRun && !leases.every(lease => completeTerminatedRemoteNativeSessionCleanup({ if (run.runtimeMode === "native" && !input.dryRun && !leases.every(lease => completeTerminatedRemoteNativeSessionCleanup({
companyId, runId: run.id, remoteCleanupScope: remoteLeaseCleanupScope(lease)!, companyId, runId: run.id, remoteCleanupScope: remoteLeaseCleanupScope(lease)!,
}))) return null; }))) return refused("remote_cleanup", "Waiting for the previous environment to stop. Your message will start automatically.");
} else { } else {
if (leases.some(lease => !lease.releasedAt || lease.cleanupStatus === "failed")) return blocked("local_cleanup", "Waiting for the previous environment to finish cleanup. Your message will start automatically."); if (leases.some(lease => !lease.releasedAt || lease.cleanupStatus === "failed")) return blocked("local_cleanup", "Waiting for the previous environment to finish cleanup. Your message will start automatically.");
if (!unusedAdmission && !cancelledStartup && !failedPreparation) { if (!unusedAdmission && !cancelledStartup && !failedPreparation) {
@@ -652,7 +659,7 @@ export async function admitExplicitNativeContinuation(input: {
const nativeSources = sources.filter(run => run.runtimeMode === "native"); const nativeSources = sources.filter(run => run.runtimeMode === "native");
const executedSources = sources.filter(run => run.runtimeMode === "native" || run.errorCode !== "execution_reconciliation_required" || run.startedAt); const executedSources = sources.filter(run => run.runtimeMode === "native" || run.errorCode !== "execution_reconciliation_required" || run.startedAt);
if (!executedSources.length || (retry && !sources.some(run => run.id === input.failedRunId)) || if (!executedSources.length || (retry && !sources.some(run => run.id === input.failedRunId)) ||
(taskRetry && !sources.some(run => run.id === taskRetry.sourceRunId))) return null; (taskRetry && !sources.some(run => run.id === taskRetry.sourceRunId))) return refused("previous_execution_unverified", PREVIOUS_EXECUTION_UNVERIFIED);
const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( const [active] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.companyId, companyId),
or(eq(heartbeatRuns.nativeIssueId, issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`), or(eq(heartbeatRuns.nativeIssueId, issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`),
@@ -1,5 +1,5 @@
import { and, eq, isNull, sql } from "drizzle-orm"; import { and, eq, inArray, isNull, sql } from "drizzle-orm";
import { executionWorkspaceSnapshots, workspaceOperations, type Db } from "@paperclipai/db"; import { executionWorkspaceSnapshots, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db";
import type { RemoteWorkspaceRunReference } from "@paperclipai/shared"; import type { RemoteWorkspaceRunReference } from "@paperclipai/shared";
type Snapshot = typeof executionWorkspaceSnapshots.$inferSelect; type Snapshot = typeof executionWorkspaceSnapshots.$inferSelect;
@@ -48,3 +48,30 @@ export async function verifiedWorkspaceSnapshotReuse(db: Pick<Db, "select">, inp
|| typeof settled.quiescedAt !== "string" || !Number.isFinite(Date.parse(settled.quiescedAt))) return null; || typeof settled.quiescedAt !== "string" || !Number.isFinite(Date.parse(settled.quiescedAt))) return null;
return receipt.id; return receipt.id;
} }
export async function verifiedUnchangedRestoredSnapshot(db: Pick<Db, "select">, input: {
companyId: string; issueId: string; runId: string; reference: RemoteWorkspaceRunReference; snapshot: Snapshot;
}): Promise<string | null> {
const { reference, snapshot } = input;
if (reference.state !== "saved" || reference.snapshotId !== snapshot.id || snapshot.companyId !== input.companyId
|| snapshot.executionWorkspaceId !== reference.workspaceId || !["committed", "superseded"].includes(snapshot.status)
|| !snapshot.sha256) return null;
const [run] = await db.select({ resultJson: heartbeatRuns.resultJson, executionStage: heartbeatRuns.executionStage })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
const recovery = run?.resultJson?.executionRecovery as Record<string, unknown> | undefined;
if (run?.executionStage !== "preparing" || recovery?.kind !== "bootstrap" || recovery.providerWorkStarted !== false) return null;
const operations = await db.select({ id: workspaceOperations.id, phase: workspaceOperations.phase,
status: workspaceOperations.status, metadata: workspaceOperations.metadata, issueId: workspaceOperations.issueId,
workspaceId: workspaceOperations.executionWorkspaceId }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, input.companyId), eq(workspaceOperations.heartbeatRunId, input.runId),
inArray(workspaceOperations.phase, ["remote_workspace_restore", "remote_workspace_snapshot"])));
if (operations.some(operation => operation.phase === "remote_workspace_snapshot")) return null;
const restores = operations.filter(operation => {
const cleanup = operation.metadata?.exportCleanup as Record<string, unknown> | undefined;
return operation.status === "succeeded" && operation.issueId === input.issueId
&& operation.workspaceId === reference.workspaceId && operation.metadata?.epoch === reference.epoch
&& operation.metadata?.snapshotId === snapshot.id && cleanup?.leaseId === reference.leaseId
&& cleanup?.providerLeaseId === reference.providerLeaseId && cleanup?.remoteCwd === reference.remoteCwd;
});
return restores.length === 1 ? restores[0]!.id : null;
}
@@ -0,0 +1,329 @@
import { createHash, randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, companies, createDb, environmentLeases, executionWorkspaces,
executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues,
workspaceOperations } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { requestTaskRetry } from "./task-retry.js";
import { heartbeatService } from "./heartbeat.js";
import { getExecutionBlocker } from "./execution-blocker.js";
import { taskInterruptionPresentation } from "./task-interruption-presentation.js";
import { REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA } from "./remote-workspace-epoch-abandonment.js";
const logs = vi.hoisted(() => ({ content: new Map<string, string>() }));
vi.mock("./run-log-store.js", async importOriginal => {
const actual = await importOriginal<typeof import("./run-log-store.js")>();
return { ...actual, getRunLogStore: () => ({ ...actual.getRunLogStore(),
read: async (handle: { logRef: string }) => {
const content = logs.content.get(handle.logRef);
if (content === undefined) throw new Error("Object log not found");
return { content };
} }) };
});
describe("Retry of a task whose saved work recovery settled", () => {
let postgres: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
postgres = await startEmbeddedPostgresTestDatabase("saved-work-retry-");
db = createDb(postgres.connectionString);
}, 30_000);
afterAll(async () => { await db?.$client.end(); await postgres?.cleanup(); });
beforeEach(() => { logs.content.clear(); });
const actorId = "board";
const terminated = (companyId: string, runId: string, leaseId: string, providerLeaseId: string, at: Date) => ({
pendingCleanupInFlight: false, remoteExecutionTermination: { schema: "paperclip.remote-termination.v1",
companyId, runId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed", confirmedAt: at.toISOString() } });
async function base(prefix: string) {
const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID(), workspaceId = randomUUID();
await db.insert(companies).values({ id: companyId, name: prefix, issuePrefix: `${prefix}${companyId.slice(0, 5)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Codie", adapterType: "paperclip_runner", status: "idle",
runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
await db.insert(issues).values({ id: issueId, companyId, title: "Continue the task", status: "blocked", assigneeAgentId: agentId });
return { companyId, issueId, agentId, workspaceId };
}
async function sourceUnavailable() {
const f = await base("SU");
const { companyId, issueId, agentId, workspaceId } = f;
const firstId = randomUUID(), preparedId = randomUUID(), heldId = randomUUID(), actionId = randomUUID();
const leaseId = randomUUID(), providerLeaseId = randomUUID(), snapshotId = randomUUID(), now = Date.now();
const remoteCwd = "/home/daytona/paperclip-workspace";
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
mode: "isolated_workspace", strategyType: "git_worktree", name: "Task workspace", providerType: "cloud_sandbox",
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 9, dirty: false, source: { kind: "empty" },
location: null, snapshotId, writerRunId: null },
remoteWorkspaceAbandonedEpochs: [{ schema: REMOTE_WORKSPACE_ABANDONED_EPOCH_SCHEMA, runId: preparedId, epoch: 8,
location: { leaseId, providerLeaseId, remoteCwd, environmentId: randomUUID() }, snapshotId,
abandonedAt: new Date(now - 500_000).toISOString(), providerRef: remoteCwd }] } });
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
await db.insert(heartbeatRuns).values({ id: firstId, companyId, agentId, issueId, status: "failed", runtimeMode: "legacy",
errorCode: "process_lost", executionStage: "preparing", invocationSource: "automation",
createdAt: new Date(now - 900_000), startedAt: new Date(now - 880_000), finishedAt: new Date(now - 800_000),
contextSnapshot: { issueId }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
resultJson: { stopReason: "process_lost", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
const scheduledRetryAt = new Date(now - 780_000), outputAt = new Date(now - 700_000);
const lines = ["[paperclip] Execution workspace reuse freshness action \"replace\" because execution workspace configuration changed: repo/base ref, environment.\n",
`[paperclip] Skipping saved session resume for task "${issueId}" because effective run configuration changed: environment.\n`];
const attemptId = randomUUID();
const content = lines.map((chunk, index) => JSON.stringify({ ts: (index ? outputAt : new Date(outputAt.getTime() - 5)).toISOString(),
attemptId, stream: "stdout", chunk, seq: index + 1 }) + "\n").join("");
const logRef = `${companyId}/${agentId}/${preparedId}`;
logs.content.set(logRef, content);
const error = "input.provider.profile does not match the qualified ACPX v1 profile";
await db.insert(heartbeatRuns).values({ id: preparedId, companyId, agentId, issueId, status: "failed", runtimeMode: "legacy",
errorCode: "adapter_failed", error, executionStage: "preparing", invocationSource: "automation", retryOfRunId: firstId,
createdAt: new Date(now - 790_000), startedAt: new Date(now - 760_000), finishedAt: new Date(now - 690_000),
scheduledRetryAt, scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure",
controllerBootId: randomUUID(), logStore: "s3_segments", logRef, logBytes: Buffer.byteLength(content),
logSha256: createHash("sha256").update(content).digest("hex"), logCompressed: false, stdoutExcerpt: lines.join(""),
stderrExcerpt: "", lastOutputAt: outputAt, lastOutputSeq: 2, lastOutputStream: "stdout", lastOutputBytes: Buffer.byteLength(content),
usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } },
contextSnapshot: { issueId, retryOfRunId: firstId, wakeReason: "transient_failure_retry", retryReason: "transient_failure",
scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryAttempt: 1 },
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId, providerLeaseId, remoteCwd,
epoch: 8, state: "prepared", snapshotId },
chatControlRecoveryAdmission: { phase: "required", runId: preparedId, agentId, issueId, version: 1, companyId } },
resultJson: { stopReason: "adapter_failed", failureCause: "native_execution_input_invalid", timeoutFired: false,
timeoutSource: "config", executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false,
effectiveTimeoutSec: 0, systemRecoveryAdoption: { actionId, policyVersion: 1 }, workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } } });
const event = { companyId, agentId, runId: preparedId, stream: "system" };
await db.insert(heartbeatRunEvents).values([
{ ...event, seq: 1, eventType: "lifecycle", level: "info", message: "Scheduled retry became due and was promoted to the queued run pool",
payload: { scheduledRetryAt: scheduledRetryAt.toISOString(), scheduledRetryReason: "transient_failure", scheduledRetryAttempt: 1 } },
{ ...event, seq: 2, eventType: "lifecycle", level: "info", message: "run started" },
{ ...event, seq: 3, eventType: "error", level: "error", message: error },
{ ...event, seq: 4, eventType: "lifecycle", level: "info", message: "task.run.measured",
payload: { phase: "task.run.measured", outcome: "failed", durationMs: 318_581, durableSettlement: true } },
]);
const releasedAt = new Date(now - 650_000);
await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: preparedId,
executionWorkspaceId: workspaceId, provider: "daytona", providerLeaseId, status: "expired",
leasePolicy: "reuse_by_environment", releasedAt, cleanupStatus: "success",
metadata: terminated(companyId, preparedId, leaseId, providerLeaseId, releasedAt) });
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId,
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: preparedId,
status: "resolved", outcome: "blocked",
nextAction: "Automatic recovery of saved work failed (last failure: source_unavailable). Retry to try again; the agent will not repeat its work.",
evidence: { runId: preparedId,
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, capturePolicyVersion: 2, phase: "verify_execution",
state: "exhausted", failure: { code: "source_unavailable", retryClass: "use_alternative_strategy", evidenceFingerprint: "a".repeat(64) },
attempts: 1, intentId: null, revision: 1, attemptId: randomUUID(), deadlineAt: now + 600_000, retriesUsed: 0,
sourceRunId: preparedId, generationId: randomUUID(), nextAttemptAt: null, firstFailureAt: now - 600_000,
freshAttemptRunId: null, providerCooldownAt: null, preservationOperationIds: [] },
automaticRecovery: { runId: preparedId, policy: "preserve_without_replay_v1", replay: "blocked",
recordedAt: new Date(now - 680_000).toISOString(), actionOutcome: "unknown" },
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: preparedId, stage: "verifying_stop", status: "blocked",
attempt: 1, attemptLimit: 6, retryable: true, requestIds: [],
message: "Automatic recovery of saved work failed (last failure: source_unavailable). Retry to try again; the agent will not repeat its work." },
originalFailureCode: "adapter_failed", continuationDelivery: "invalidated",
workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] },
historicalWorkspaceSource: { schema: "paperclip.historical-workspace-source.v1", runId: preparedId, leaseIds: [leaseId] },
previousWorkspaceRecovery: null } });
await db.insert(heartbeatRuns).values({ id: heldId, companyId, agentId, issueId, status: "cancelled", runtimeMode: "legacy",
errorCode: "execution_reconciliation_required", error: "Recovering saved work…", invocationSource: "automation",
triggerDetail: "system", retryOfRunId: preparedId, createdAt: new Date(now - 680_000), finishedAt: new Date(now - 600_000),
contextSnapshot: { issueId, taskId: issueId, source: "issue.continuation_recovery", wakeReason: "issue_continuation_needed",
retryOfRunId: preparedId },
resultJson: { stopReason: "execution_reconciliation_required", timeoutFired: false,
executionWait: { issueId, recoveryActionId: actionId }, timeoutSource: "stale_queued_run_gate",
timeoutConfigured: false, effectiveTimeoutSec: 0 } });
return { ...f, preparedId, heldId, actionId };
}
async function restoredSnapshot() {
const f = await base("RS");
const { companyId, issueId, agentId, workspaceId } = f;
const earlierId = randomUUID(), sourceId = randomUUID(), actionId = randomUUID(), leaseId = randomUUID();
const providerLeaseId = randomUUID(), snapshotId = randomUUID(), now = Date.now(), sha256 = "c".repeat(64);
const remoteCwd = `/home/daytona/paperclip-workspace/.paperclip-workspaces/${workspaceId}-1-${leaseId}`;
const location = { leaseId, providerLeaseId, remoteCwd, environmentId: randomUUID() };
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, sourceIssueId: issueId,
mode: "isolated_workspace", strategyType: "git_worktree", name: "Task workspace", providerType: "cloud_sandbox",
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 3, dirty: false, source: { kind: "empty" },
location, snapshotId, writerRunId: null } } });
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, issueId));
await db.insert(heartbeatRuns).values({ id: earlierId, companyId, agentId, issueId, status: "succeeded", runtimeMode: "native",
createdAt: new Date(now - 7_200_000), startedAt: new Date(now - 7_100_000), finishedAt: new Date(now - 7_000_000),
contextSnapshot: { issueId } });
const [earlierSave] = await db.insert(workspaceOperations).values({ companyId, executionWorkspaceId: workspaceId,
heartbeatRunId: earlierId, issueId, phase: "remote_workspace_snapshot", status: "succeeded",
metadata: { boundary: "final", epoch: 3, snapshotId } }).returning();
await db.insert(executionWorkspaceSnapshots).values({ id: snapshotId, companyId, executionWorkspaceId: workspaceId,
sourceOperationId: earlierSave.id, sourceRunId: earlierId, generation: 3, epoch: 3, status: "committed",
objectKey: `${companyId}/workspace-snapshots/${workspaceId}/${snapshotId}/archive`, sha256, byteSize: 48_507_938,
key: { kind: "fixture" }, manifestDigest: "d".repeat(64) });
const error = "Request failed with status code 502";
await db.insert(heartbeatRuns).values({ id: sourceId, companyId, agentId, issueId, status: "failed", runtimeMode: "legacy",
errorCode: "setup_failed", error, executionStage: "preparing", createdAt: new Date(now - 300_000),
startedAt: new Date(now - 280_000), finishedAt: new Date(now - 220_000), controllerBootId: randomUUID(),
usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } },
contextSnapshot: { issueId, wakeReason: "task_retry" },
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" },
remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId, providerLeaseId, remoteCwd,
epoch: 3, state: "saved", snapshotId } },
resultJson: { stopReason: "adapter_failed", failureCause: "JsonRpcCallError", timeoutFired: false, timeoutSource: "config",
executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, timeoutConfigured: false, effectiveTimeoutSec: 0,
systemRecoveryAdoption: { actionId, policyVersion: 1 }, workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] } } });
const event = { companyId, agentId, runId: sourceId, stream: "system" };
await db.insert(heartbeatRunEvents).values([
{ ...event, seq: 1, eventType: "error", level: "error", message: error },
{ ...event, seq: 2, eventType: "lifecycle", level: "info", message: "task.run.measured",
payload: { phase: "task.run.measured", outcome: "failed", durationMs: 61_906, durableSettlement: true } },
]);
const releasedAt = new Date(now - 200_000);
await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: sourceId,
executionWorkspaceId: workspaceId, provider: "daytona", providerLeaseId, status: "expired",
leasePolicy: "reuse_by_environment", releasedAt, cleanupStatus: "success",
metadata: terminated(companyId, sourceId, leaseId, providerLeaseId, releasedAt) });
await db.insert(workspaceOperations).values({ companyId, executionWorkspaceId: workspaceId, heartbeatRunId: sourceId, issueId,
phase: "remote_workspace_restore", status: "succeeded", metadata: { epoch: 3, snapshotId,
exportCleanup: { state: "cleaned", leaseId, providerLeaseId, remoteCwd, lastError: null,
stagingRoot: "/home/daytona/paperclip-workspace/.paperclip-workspaces", environmentId: location.environmentId } } });
const [recovered] = await db.insert(workspaceOperations).values({ companyId, executionWorkspaceId: workspaceId,
heartbeatRunId: sourceId, issueId, phase: "workspace_recovery", status: "succeeded",
metadata: { epoch: 3, sha256, leaseId, storage: "remote_workspace", generation: 3, snapshotId, providerLeaseId,
recoveryActionId: actionId } }).returning();
await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: sourceId, issueId, phase: "workspace_finalize",
status: "succeeded", metadata: { owningService: "workspace_recovery", preservationOperationIds: [recovered.id] } });
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId,
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: sourceId,
status: "resolved", outcome: "blocked", nextAction: "Saved work recovered.",
evidence: { runId: sourceId,
systemRecovery: { schema: "paperclip.system-recovery.v1", policyVersion: 1, capturePolicyVersion: 2, phase: "reconcile_effects",
state: "complete", failure: null, attempts: 1, intentId: null, revision: 3, attemptId: randomUUID(), deadlineAt: null,
retriesUsed: 0, sourceRunId: sourceId, generationId: randomUUID(), nextAttemptAt: null, firstFailureAt: null,
freshAttemptRunId: null, providerCooldownAt: null, preservationOperationIds: [recovered.id] },
automaticRecovery: { runId: sourceId, policy: "preserve_without_replay_v1", replay: "blocked",
recordedAt: new Date(now - 210_000).toISOString(), actionOutcome: "unknown" },
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: sourceId, stage: "saving_receipts", status: "recovered",
attempt: 1, attemptLimit: 6, retryable: true, requestIds: [] },
originalFailureCode: "setup_failed", continuationDelivery: "invalidated", workspaceRestoreFailure: "restore_failed",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [leaseId] },
historicalWorkspaceSource: { schema: "paperclip.historical-workspace-source.v1", runId: sourceId, leaseIds: [leaseId] },
previousWorkspaceRecovery: null } });
return { ...f, sourceId, actionId, snapshotId };
}
async function retry(f: { companyId: string; issueId: string }, observedRunId: string) {
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId,
observedRunId, idempotencyKey: randomUUID(), actorSource: "local_implicit" });
expect(receipt).toMatchObject({ accepted: true });
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
return wake;
}
async function issue(issueId: string) {
const [row] = await db.select().from(issues).where(eq(issues.id, issueId));
return row;
}
it("offers the same Retry in the task view and the blocker when saved work is no longer available", async () => {
const f = await sourceUnavailable();
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
expect(blocker?.workspaceRecovery).toMatchObject({ status: "blocked", canRetry: true });
expect(blocker?.canRetry).toBe(true);
expect(interruption?.canRetry).toBe(true);
});
it("offers no Retry in either view while saved-work recovery waits for an unconfirmed file copy to stop", async () => {
const f = await sourceUnavailable();
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
workspaceOwnerRecovery: { state: "operator_required", owner: { pid: 3, token: randomUUID(), hostname: "previous-host" } } })}::jsonb` })
.where(eq(issueRecoveryActions.id, f.actionId));
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
expect(blocker).toMatchObject({ canRetry: false, workspaceRecovery: { status: "blocked", canRetry: false } });
expect(interruption).toMatchObject({ canRetry: false, retryWait: { reason: "workspace_repair_required" } });
});
it("offers no Retry for a held Retry with the generic wait over a hold waiting for an unconfirmed file copy to stop", async () => {
const f = await sourceUnavailable();
await db.update(issueRecoveryActions).set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({
workspaceOwnerRecovery: { state: "operator_required", owner: { pid: 3, token: randomUUID(), hostname: "previous-host" } } })}::jsonb` })
.where(eq(issueRecoveryActions.id, f.actionId));
const [agent] = await db.select({ id: agents.id }).from(agents).where(eq(agents.companyId, f.companyId));
const requestId = randomUUID();
await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.companyId, agentId: agent.id, source: "on_demand",
triggerDetail: "manual", reason: "task_retry", status: "deferred_issue_execution", requestedByActorType: "user",
requestedByActorId: actorId, idempotencyKey: `task-retry:${randomUUID()}`,
payload: { issueId: f.issueId, executionWait: { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." },
taskRetry: { schema: "paperclip.task-retry.v1", generationId: requestId, sourceRunId: f.heldId, actorId,
actorSource: "local_implicit", strategy: "evaluate" } } });
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
expect(blocker?.canRetry).toBe(false);
expect(interruption).toMatchObject({ canRetry: false, retryWait: { reason: "execution_recovery" } });
});
it("continues in a fresh session from the recorded work when saved work is no longer available", async () => {
const f = await sourceUnavailable();
const wake = await retry(f, f.heldId);
expect(wake.payload?.executionWait).toBeUndefined();
expect(wake).toMatchObject({ status: "coalesced", runId: expect.any(String) });
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!));
expect(["queued", "running"]).toContain(successor.status);
expect(successor.contextSnapshot).toMatchObject({ taskRetryRequestId: wake.id });
});
it("verifies the repair of a restored snapshot that the failed preparation never changed", async () => {
const f = await restoredSnapshot();
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
expect(blocker).toMatchObject({ workspaceRepairRequired: false, canRetry: true,
workspaceRecovery: { status: "recovered" } });
expect(interruption?.canRetry).toBe(true);
const wake = await retry(f, f.sourceId);
expect(wake).toMatchObject({ status: "coalesced", runId: expect.any(String) });
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!));
expect(["queued", "running"]).toContain(successor.status);
});
it("keeps the repair unverified when the failed preparation may have changed the restored snapshot", async () => {
const f = await restoredSnapshot();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceId));
await db.update(heartbeatRuns).set({ resultJson: { ...run.resultJson,
executionRecovery: { kind: "bootstrap", providerWorkStarted: true } } }).where(eq(heartbeatRuns.id, f.sourceId));
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
expect(blocker).toMatchObject({ workspaceRepairRequired: true, canRetry: false });
expect(await taskInterruptionPresentation(db, await issue(f.issueId))).toMatchObject({ canRetry: false });
});
it("names a refusal that another click cannot change and offers no Retry for it in either view", async () => {
const f = await restoredSnapshot();
await db.update(workspaceOperations).set({ status: "failed" }).where(and(eq(workspaceOperations.heartbeatRunId, f.sourceId),
eq(workspaceOperations.phase, "remote_workspace_restore")));
await db.update(issueRecoveryActions).set({ evidence: sql`jsonb_set(${issueRecoveryActions.evidence},
'{workspaceRecovery,status}', '"blocked"')` }).where(eq(issueRecoveryActions.id, f.actionId));
const wake = await retry(f, f.sourceId);
expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null,
payload: { executionWait: { reason: "previous_execution_unverified" } } });
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
expect(interruption).toMatchObject({ canRetry: false, retryWait: { reason: "previous_execution_unverified" } });
expect(blocker?.canRetry).toBe(false);
});
it("names the admission refusal of a held Retry and offers Retry only when admission would take it", async () => {
const f = await restoredSnapshot();
await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 3_600_000) })
.where(eq(heartbeatRuns.id, f.sourceId));
const wake = await retry(f, f.sourceId);
expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null });
const reason = (wake.payload?.executionWait as { reason?: string } | undefined)?.reason;
expect(reason).toBeTruthy();
expect(reason).not.toBe("execution_recovery");
const interruption = await taskInterruptionPresentation(db, await issue(f.issueId));
expect(interruption?.retryWait?.reason).toBe(reason);
expect(interruption?.retryWait?.message).not.toMatch(/execution checks/);
});
});
@@ -1,7 +1,7 @@
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js"; import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
import { AGENT_DIRECTORY_WAIT_RETRY_REASON } from "./execution-recovery-attempt.js"; import { AGENT_DIRECTORY_WAIT_RETRY_REASON } from "./execution-recovery-attempt.js";
import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js"; import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js";
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm"; import { and, desc, eq, inArray, isNull, not, sql, type SQL } from "drizzle-orm";
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js"; import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js"; import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions, import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions,
@@ -23,6 +23,7 @@ import { canonicalExecutionUuid } from "./execution-identity.js";
import { chatActionDeadline } from "./chat-action-work.js"; import { chatActionDeadline } from "./chat-action-work.js";
import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js"; import { NATIVE_CLEANUP_SETTLING_MESSAGE, NATIVE_SESSION_CLEANUP_QUARANTINED } from "./native-runtime/completed-turn-cleanup.js";
import { isQueuedDependencyWait } from "./run-cancellation.js"; import { isQueuedDependencyWait } from "./run-cancellation.js";
import { heldRetryOffersRetry } from "./task-retry-eligibility.js";
import { exhaustedNativeFinalizationCondition, savedResultRetryPendingCondition } from "./native-runtime/exhausted-finalization-retry.js"; import { exhaustedNativeFinalizationCondition, savedResultRetryPendingCondition } from "./native-runtime/exhausted-finalization-retry.js";
import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON, import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON,
WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js"; WORKSPACE_BUSY_RETRY_REASON } from "../modules/run-dispatch/index.js";
@@ -161,6 +162,7 @@ const retryWaitMessages: Record<string, string> = {
// A new click replaces a Retry declined for its requester's access with a // A new click replaces a Retry declined for its requester's access with a
// fresh attempt under the clicker's access, so Retry stays available. // fresh attempt under the clicker's access, so Retry stays available.
retry_authority_denied: "Retry could not start with the requesting user's access. Press Retry to start it with yours.", retry_authority_denied: "Retry could not start with the requesting user's access. Press Retry to start it with yours.",
previous_execution_unverified: "Retry can't continue because Paperclip can't confirm that the previous attempt stopped before it started work. Your message is saved.",
// An edited saved message can never be replayed; only new input continues. // An edited saved message can never be replayed; only new input continues.
retry_source_changed: "Retry cannot continue because its saved message changed. Send a new message to continue from saved work.", retry_source_changed: "Retry cannot continue because its saved message changed. Send a new message to continue from saved work.",
}; };
@@ -339,8 +341,7 @@ function projectTaskInterruptionState(issue: ProjectionIssue, input: ProjectionS
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery"; const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
// Keep the latest input receipt so message submission can settle independently. // Keep the latest input receipt so message submission can settle independently.
return { ...base, state: "failed", return { ...base, state: "failed",
canRetry: !["cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required"].includes(reason) canRetry: heldRetryOffersRetry(reason, input.cleanupSettling), retryWait: { reason,
&& (reason !== "cleanup_quarantined" || !input.cleanupSettling), retryWait: { reason,
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } }; message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
} }
const queuedDuringRun = Boolean(wait && run.status === "succeeded" && ["todo", "in_progress", "in_review"].includes(issue.status) && run.finishedAt && wait.requestedAt.getTime() <= run.finishedAt.getTime() const queuedDuringRun = Boolean(wait && run.status === "succeeded" && ["todo", "in_progress", "in_review"].includes(issue.status) && run.finishedAt && wait.requestedAt.getTime() <= run.finishedAt.getTime()
@@ -744,6 +745,22 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
agentAdapter: run.agentAdapter ?? "unknown" }); agentAdapter: run.agentAdapter ?? "unknown" });
} }
} }
const offered = [...result].flatMap(([issueId, presentation]) => presentation?.canRetry ? [issueId] : []);
if (offered.length) {
const { executionBlockerPredicate, getExecutionBlocker, restoreHoldPredicate, restoreHoldReleasedForCurrentOwnerPredicate } =
await import("./execution-blocker.js");
const held = await db.selectDistinct({ issueId: issueRecoveryActions.sourceIssueId }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, offered),
executionBlockerPredicate(), not(restoreHoldReleasedForCurrentOwnerPredicate()), restoreHoldPredicate()));
for (const { issueId } of held) {
const blocker = await getExecutionBlocker(db, companyId, issueId);
const presentation = result.get(issueId);
if (!presentation || !blocker || blocker.canRetry) continue;
presentation.canRetry = false;
if (!presentation.retryWait && blocker.workspaceRecovery && blocker.workspaceRecovery.status !== "recovered")
presentation.retryWait = { reason: "workspace_repair_required", message: retryWaitMessages.workspace_repair_required! };
}
}
// Redact each explanation with the registry of the run that recorded it: the // Redact each explanation with the registry of the run that recorded it: the
// workspace failure is the observed run's, while a scheduled retry's provider // workspace failure is the observed run's, while a scheduled retry's provider
// failure may be its predecessor's. // failure may be its predecessor's.
@@ -0,0 +1,30 @@
import { and, desc, eq, inArray, isNull, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
const REFUSALS_A_CLICK_CANNOT_CHANGE: ReadonlySet<string> = new Set([
"cancelled_by_operator", "retry_source_changed", "workspace_owner_stop_required", "previous_execution_unverified",
]);
export function heldRetryOffersRetry(reason: string | null | undefined, cleanupSettling = false): boolean {
if (!reason) return true;
return !REFUSALS_A_CLICK_CANNOT_CHANGE.has(reason) && (reason !== "cleanup_quarantined" || !cleanupSettling);
}
export async function heldTaskRetryRefusals(db: Pick<Db, "selectDistinctOn">, companyId: string, issueIds: string[]): Promise<Map<string, string>> {
if (!issueIds.length) return new Map();
const issueId = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
const rows = await db.selectDistinctOn([issueId], { issueId, sourceRunId: sql<string | null>`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`,
reason: sql<string | null>`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`,
latestRunId: sql<string | null>`(select latest.id::text from ${heartbeatRuns} latest
where latest.company_id = ${companyId} and (latest.issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
or latest.native_issue_id::text = ${agentWakeupRequests.payload}->>'issueId'
or latest.context_snapshot->>'issueId' = ${agentWakeupRequests.payload}->>'issueId')
order by latest.created_at desc, latest.id desc limit 1)` })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"),
isNull(agentWakeupRequests.runId), inArray(issueId, issueIds),
sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}' is not null`))
.orderBy(issueId, desc(agentWakeupRequests.requestedAt));
return new Map(rows.flatMap(row => row.reason && row.sourceRunId && row.sourceRunId === row.latestRunId
? [[row.issueId, row.reason] as const] : []));
}
@@ -65,6 +65,50 @@ describe("verified host preparation output", () => {
expect(await hasVerifiedPreparationLog(f.run)).toBe(false); expect(await hasVerifiedPreparationLog(f.run)).toBe(false);
}); });
it("verifies a log made only of the host's workspace notices written before the adapter starts", async () => {
const f = source();
const issueId = randomUUID(), ts = f.run.lastOutputAt!;
const chunks = [f.record.chunk,
"[paperclip] Execution workspace reuse freshness action \"replace\" because execution workspace configuration changed: repo/base ref, environment.\n",
`[paperclip] Skipping saved session resume for task "${issueId}" because effective run configuration changed: environment.\n`];
const content = chunks.map((chunk, index) => JSON.stringify({ ...f.record, chunk, seq: index + 1, ts: ts.toISOString() }) + "\n").join("");
Object.assign(f.run, { lastOutputSeq: 3, stdoutExcerpt: chunks.join(""), logBytes: Buffer.byteLength(content),
lastOutputBytes: Buffer.byteLength(content), logSha256: createHash("sha256").update(content).digest("hex") });
fixture.read.mockResolvedValue({ content });
expect(await hasVerifiedPreparationLog(f.run)).toBe(true);
const notices = chunks.slice(1);
const withoutSkills = notices.map((chunk, index) => JSON.stringify({ ...f.record, chunk, seq: index + 1, ts: ts.toISOString() }) + "\n").join("");
Object.assign(f.run, { contextSnapshot: null, lastOutputSeq: 2, stdoutExcerpt: notices.join(""), logBytes: Buffer.byteLength(withoutSkills),
lastOutputBytes: Buffer.byteLength(withoutSkills), logSha256: createHash("sha256").update(withoutSkills).digest("hex") });
fixture.read.mockResolvedValue({ content: withoutSkills });
expect(await hasVerifiedPreparationLog(f.run)).toBe(true);
});
it("accepts two notices of the same attempt", async () => {
const f = source();
const notice = "[paperclip] Skipping saved session resume because effective run configuration changed: environment.\n";
const content = [1, 2].map(seq => JSON.stringify({ ...f.record, chunk: notice, seq }) + "\n").join("");
Object.assign(f.run, { lastOutputSeq: 2, stdoutExcerpt: notice + notice, logBytes: Buffer.byteLength(content),
lastOutputBytes: Buffer.byteLength(content), logSha256: createHash("sha256").update(content).digest("hex") });
fixture.read.mockResolvedValue({ content });
expect(await hasVerifiedPreparationLog(f.run)).toBe(true);
});
it.each(["provider_line", "other_attempt", "skills_after_notice"] as const)("keeps a log with other output held: %s", async state => {
const f = source();
const notice = "[paperclip] Skipping saved session resume because effective run configuration changed: environment.\n";
const records = [{ ...f.record, chunk: notice, seq: 1 }, { ...f.record, seq: 2 }];
if (state !== "skills_after_notice") records[1]!.chunk = notice;
if (state === "provider_line") records[1]!.chunk = "[paperclip] Provider wrote a file\n";
if (state === "other_attempt") records[1]!.attemptId = randomUUID();
const content = records.map(record => JSON.stringify(record) + "\n").join("");
Object.assign(f.run, { lastOutputSeq: 2, stdoutExcerpt: records.map(record => record.chunk).join(""),
logBytes: Buffer.byteLength(content), lastOutputBytes: Buffer.byteLength(content),
logSha256: createHash("sha256").update(content).digest("hex") });
fixture.read.mockResolvedValue({ content });
expect(await hasVerifiedPreparationLog(f.run)).toBe(false);
});
it("cancels a slow read, fails closed, and leaves an unrelated recovery responsive", async () => { it("cancels a slow read, fails closed, and leaves an unrelated recovery responsive", async () => {
const slow = source(), fast = source(); const slow = source(), fast = source();
let signal: AbortSignal | undefined; let signal: AbortSignal | undefined;
+42 -21
View File
@@ -3,7 +3,13 @@ import { isUuidLike } from "@paperclipai/shared";
import { getRunLogStore } from "./run-log-store.js"; import { getRunLogStore } from "./run-log-store.js";
const MAX_BYTES = 2048; const MAX_BYTES = 2048;
const MAX_RECORDS = 4;
const READ_TIMEOUT_MS = 2000; const READ_TIMEOUT_MS = 2000;
const SKILLS_PREFIX = "[paperclip] Enabled run-scoped skills from issue mentions: ";
const HOST_WORKSPACE_NOTICES = [
/^\[paperclip\] Execution workspace reuse freshness action "[a-z_]+" because [^\r\n]+\.\n$/,
/^\[paperclip\] Skipping saved session resume (?:for task "[^"\r\n]+" )?because [^\r\n]+\.\n$/,
];
type PreparationLog = { type PreparationLog = {
companyId: string; agentId: string; id: string; companyId: string; agentId: string; id: string;
contextSnapshot: Record<string, unknown> | null; contextSnapshot: Record<string, unknown> | null;
@@ -12,14 +18,26 @@ type PreparationLog = {
lastOutputAt: Date | null; lastOutputSeq: number; lastOutputStream: string | null; lastOutputBytes: number | null; lastOutputAt: Date | null; lastOutputSeq: number; lastOutputStream: string | null; lastOutputBytes: number | null;
}; };
export async function hasVerifiedPreparationLog(run: PreparationLog): Promise<boolean> { function isMentionedSkillsNotice(chunk: string, context: Record<string, unknown> | null) {
const versions = run.contextSnapshot?.skillVersions; const versions = context?.skillVersions;
if (run.contextSnapshot?.skillSelectionSnapshot !== true || !versions || typeof versions !== "object" if (context?.skillSelectionSnapshot !== true || !versions || typeof versions !== "object"
|| Array.isArray(versions) || Object.keys(versions).length === 0 || Array.isArray(versions) || Object.keys(versions).length === 0
|| run.logStore !== "s3_segments" || run.logRef !== `${run.companyId}/${run.agentId}/${run.id}` || !chunk.startsWith(SKILLS_PREFIX) || !chunk.endsWith("\n")) return false;
const keys = chunk.slice(SKILLS_PREFIX.length, -1).split(", ");
return keys.length > 0 && new Set(keys).size === keys.length
&& keys.every(key => {
const version = (versions as Record<string, unknown>)[key];
return key.length > 0 && Object.hasOwn(versions, key) && !/[\r\n]/.test(key)
&& typeof version === "string" && isUuidLike(version);
});
}
export async function hasVerifiedPreparationLog(run: PreparationLog): Promise<boolean> {
if (run.logStore !== "s3_segments" || run.logRef !== `${run.companyId}/${run.agentId}/${run.id}`
|| !Number.isSafeInteger(run.logBytes) || run.logBytes! <= 0 || run.logBytes! > MAX_BYTES || !Number.isSafeInteger(run.logBytes) || run.logBytes! <= 0 || run.logBytes! > MAX_BYTES
|| !run.logSha256 || !/^[a-f0-9]{64}$/.test(run.logSha256) || run.logCompressed || !run.logSha256 || !/^[a-f0-9]{64}$/.test(run.logSha256) || run.logCompressed
|| run.lastOutputSeq !== 1 || run.lastOutputStream !== "stdout" || !run.lastOutputAt || !Number.isSafeInteger(run.lastOutputSeq) || run.lastOutputSeq < 1 || run.lastOutputSeq > MAX_RECORDS
|| run.lastOutputStream !== "stdout" || !run.lastOutputAt
|| run.lastOutputBytes !== run.logBytes || run.stderrExcerpt !== "") return false; || run.lastOutputBytes !== run.logBytes || run.stderrExcerpt !== "") return false;
const controller = new AbortController(); const controller = new AbortController();
let timer: ReturnType<typeof setTimeout> | undefined; let timer: ReturnType<typeof setTimeout> | undefined;
@@ -31,22 +49,25 @@ export async function hasVerifiedPreparationLog(run: PreparationLog): Promise<bo
timer.unref(); timer.unref();
})]); })]);
if (!result || result.nextOffset !== undefined || Buffer.byteLength(result.content) !== run.logBytes if (!result || result.nextOffset !== undefined || Buffer.byteLength(result.content) !== run.logBytes
|| createHash("sha256").update(result.content).digest("hex") !== run.logSha256) return false; || createHash("sha256").update(result.content).digest("hex") !== run.logSha256
const record = JSON.parse(result.content) as Record<string, unknown> | null; || !result.content.endsWith("\n")) return false;
if (!record || Array.isArray(record) || typeof record !== "object" const records = result.content.slice(0, -1).split("\n").map(line => JSON.parse(line) as Record<string, unknown> | null);
|| Object.keys(record).some(key => !["ts", "attemptId", "stream", "chunk", "seq"].includes(key)) if (records.length !== run.lastOutputSeq) return false;
|| record.ts !== run.lastOutputAt.toISOString() || typeof record.attemptId !== "string" || !isUuidLike(record.attemptId) const attemptId = records[0]?.attemptId;
|| record.seq !== 1 || record.stream !== "stdout" || typeof record.chunk !== "string" let excerpt = "";
|| record.chunk !== run.stdoutExcerpt) return false; for (const [index, record] of records.entries()) {
const prefix = "[paperclip] Enabled run-scoped skills from issue mentions: "; if (!record || Array.isArray(record) || typeof record !== "object"
if (!record.chunk.startsWith(prefix) || !record.chunk.endsWith("\n")) return false; || Object.keys(record).some(key => !["ts", "attemptId", "stream", "chunk", "seq"].includes(key))
const keys = record.chunk.slice(prefix.length, -1).split(", "); || typeof record.ts !== "string" || !Number.isFinite(Date.parse(record.ts))
return keys.length > 0 && new Set(keys).size === keys.length || typeof record.attemptId !== "string" || !isUuidLike(record.attemptId) || record.attemptId !== attemptId
&& keys.every(key => { || record.seq !== index + 1 || record.stream !== "stdout" || typeof record.chunk !== "string") return false;
const version = (versions as Record<string, unknown>)[key]; const chunk = record.chunk;
return key.length > 0 && Object.hasOwn(versions, key) && !/[\r\n]/.test(key) const notice = index === 0 && isMentionedSkillsNotice(chunk, run.contextSnapshot)
&& typeof version === "string" && isUuidLike(version); || HOST_WORKSPACE_NOTICES.some(pattern => pattern.test(chunk));
}); if (!notice) return false;
excerpt += chunk;
}
return records.at(-1)!.ts === run.lastOutputAt.toISOString() && excerpt === run.stdoutExcerpt;
} catch { return false; } } catch { return false; }
finally { if (timer) clearTimeout(timer); controller.abort(); } finally { if (timer) clearTimeout(timer); controller.abort(); }
} }
@@ -969,8 +969,9 @@ describe("automatic retained workspace recovery", () => {
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)); const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
expect(wake.payload?.executionWait).toMatchObject({ reason: "remote_cleanup" }); expect(wake.payload?.executionWait).toMatchObject({ reason: "remote_cleanup" });
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true, expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: false,
retryWait: { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." } }); retryWait: { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." } });
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.canRetry).toBe(false);
const commentId = randomUUID(), messageWakeId = randomUUID(); const commentId = randomUUID(), messageWakeId = randomUUID();
await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId, await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId,
authorType: "user", authorUserId: "board", body: "Keep my saved message while the previous environment stops." }); authorType: "user", authorUserId: "board", body: "Keep my saved message while the previous environment stops." });
@@ -5,8 +5,8 @@ import { z } from "zod";
import { readNativeWorkspaceSyncReference, type NativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js"; import { readNativeWorkspaceSyncReference, type NativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-migration.js"; import { readMigratedLegacyWorkspaceSource } from "./legacy-workspace-recovery-migration.js";
import { readSnapshot } from "./workspace-recovery-snapshot.js"; import { readSnapshot } from "./workspace-recovery-snapshot.js";
import { exhaustedSystemRecoveryAction, readSystemRecovery } from "./system-recovery-state.js"; import { exhaustedSystemRecoveryAction, isCapturePolicyRearmable, readSystemRecovery } from "./system-recovery-state.js";
import { verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js"; import { verifiedUnchangedRestoredSnapshot, verifiedWorkspaceSnapshotReuse } from "./remote-workspace-snapshot-reuse-receipt.js";
import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js"; import { isAbandonedPreparedRemoteWorkspace } from "./remote-workspace-abandoned-preparation.js";
export const workspaceRecoverySchema = z.object({ export const workspaceRecoverySchema = z.object({
@@ -62,6 +62,17 @@ export function recoveredWorkMessage(runStatus: string | null | undefined, retry
: "Saved work recovered. Send a message to continue."; : "Saved work recovered. Send a message to continue.";
} }
export function savedWorkOffersRetry(state: { status: string; canRetry: boolean } | undefined,
runStatus: string | null | undefined, workspaceRepairRequired: boolean, evidence: Record<string, unknown>) {
if (!state || (runStatus !== "failed" && runStatus !== "timed_out")) return false;
if (state.status === "recovered") return !workspaceRepairRequired;
const ownerRecovery = evidence.workspaceOwnerRecovery as { state?: unknown } | undefined;
const recovery = readSystemRecovery(evidence);
return state.canRetry && !(ownerRecovery?.state === "operator_required" && evidence.workspaceOwnerStop == null)
&& recovery?.state === "exhausted" && !recovery.freshAttemptRunId && recovery.failure?.retryClass !== "recheck_evidence"
&& !isCapturePolicyRearmable(recovery);
}
export function workspaceRecoveryDisplay(evidence: Record<string, unknown>, runId: string | null, export function workspaceRecoveryDisplay(evidence: Record<string, unknown>, runId: string | null,
run?: { status: string | null; retryOffered: boolean }): ExecutionBlocker["workspaceRecovery"] { run?: { status: string | null; retryOffered: boolean }): ExecutionBlocker["workspaceRecovery"] {
const state = readWorkspaceRecovery(evidence); const state = readWorkspaceRecovery(evidence);
@@ -175,7 +186,8 @@ export async function verifiedWorkspaceRepairReceipt(db: Db, run: Pick<typeof he
)); ));
if (!snapshot || snapshot.sha256 !== proof.sha256 || snapshot.generation !== proof.generation) return null; if (!snapshot || snapshot.sha256 !== proof.sha256 || snapshot.generation !== proof.generation) return null;
if ((snapshot.epoch !== remote.epoch || snapshot.sourceRunId !== run.id) if ((snapshot.epoch !== remote.epoch || snapshot.sourceRunId !== run.id)
&& !await verifiedWorkspaceSnapshotReuse(db, { companyId: run.companyId, issueId, runId: run.id, reference: remote, snapshot })) return null; && !await verifiedWorkspaceSnapshotReuse(db, { companyId: run.companyId, issueId, runId: run.id, reference: remote, snapshot })
&& !await verifiedUnchangedRestoredSnapshot(db, { companyId: run.companyId, issueId, runId: run.id, reference: remote, snapshot })) return null;
} }
const [action] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and( const [action] = await db.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),