Files
PaperClipAI/server/src/services/remote-workspace-unchanged-proof.ts
T
DottaandClaude Opus 5.5 c1ea9304f7 fix(recovery): continue Runner preparations whose remote workspace was handed on unchanged (#21481)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.

**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.

**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.

**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.

## What Changed

- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.

## Verification

- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:05 -06:00

117 lines
8.7 KiB
TypeScript

import { and, eq, inArray, sql } from "drizzle-orm";
import {
executionWorkspaceRepositories, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, nativeRunnerCheckpoints,
nativeRunResults, workspaceOperations, type Db,
} from "@paperclipai/db";
import { remoteWorkspaceRunReferenceSchema, type RemoteWorkspaceAuthority } from "@paperclipai/shared";
import { executionStartedEventPredicate } from "./pre-dispatch-execution-evidence.js";
import { runOwnerFenceHeldSql } from "./remote-workspace-writer-hold.js";
import { carriesInheritedRunnerIdentity } from "./native-runtime/native-inherited-runner-retry.js";
import { readFailedBeforeProviderTurnRecord } from "./native-runtime/failed-before-provider-turn.js";
import { readUnchangedBaseline, UNCHANGED_BASELINE_KEY, UNCHANGED_BASELINE_SCHEMA,
type UnchangedBaseline } from "./remote-workspace-unchanged-baseline.js";
const PROVIDER_TURN_EVENT_TYPES = ["turn.started", "turn.completed", "turn.failed", "provider.event", "provider.rpc_result",
"tool.execution.started", "item.started", "item.completed"];
export { readUnchangedBaseline, UNCHANGED_BASELINE_KEY, UNCHANGED_BASELINE_SCHEMA, type UnchangedBaseline };
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
type Repository = { id: string; relativePath: string; repoUrl: string; ref: string | null; pinnedCommit?: string };
const object = (value: unknown): Record<string, unknown> =>
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
/** Operations that cannot change the saved files: a restore writes exactly the
* saved version, and finalize and recovery rows are receipts that wrap a save
* (whose own snapshot operation is still counted). */
const FILE_NEUTRAL_OPERATION_PHASES = ["remote_workspace_restore", "workspace_finalize", "workspace_recovery"];
/** Terminal states of a run that ended without a result. */
const ENDED_WITHOUT_RESULT = ["failed", "cancelled", "interrupted", "timed_out"];
export type UnchangedProofRefusal =
| "baseline_missing" | "authority_mismatch" | "writer_not_ended" | "writer_owner_live" | "process_recorded"
| "reference_mismatch" | "provider_session_started" | "execution_recorded" | "result_recorded"
| "runner_commanded" | "workspace_operation_recorded" | "repository_unsaved" | "predecessor_save_unsettled";
/**
* Whether the ended writer of `authority` provably left its files exactly as
* the saved version it started from. The caller holds the workspace row lock.
* Any missing, ambiguous or older-release evidence refuses; a refusal keeps
* today's full save.
*/
export async function proveRemoteWorkspaceUnchanged(tx: Pick<Db, "select">, input: {
workspace: Workspace; authority: RemoteWorkspaceAuthority;
/** The caller owns the writer's finalization, so its own fence is expected. */
callerOwnsWriter?: boolean;
isSavedRepository: (repository: Repository) => boolean;
}): Promise<{ proven: true; baseline: UnchangedBaseline } | { proven: false; reason: UnchangedProofRefusal }> {
const { workspace, authority } = input;
const runId = authority.writerRunId;
const baseline = readUnchangedBaseline(workspace.metadata);
if (!baseline || baseline.releasedAt || !runId || baseline.runId !== runId) return { proven: false, reason: "baseline_missing" };
if (!authority.dirty || !authority.location || authority.epoch !== baseline.epoch || authority.snapshotId !== baseline.snapshotId)
return { proven: false, reason: "authority_mismatch" };
const [run] = await tx.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
nativeSessionId: heartbeatRuns.nativeSessionId, createdAt: heartbeatRuns.createdAt,
status: heartbeatRuns.status, processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
sessionIdAfter: heartbeatRuns.sessionIdAfter, runnerProfileJson: heartbeatRuns.runnerProfileJson,
ownerFenceHeld: runOwnerFenceHeldSql() }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, runId)));
if (!run || !ENDED_WITHOUT_RESULT.includes(run.status)) return { proven: false, reason: "writer_not_ended" };
if (!input.callerOwnsWriter && run.ownerFenceHeld) return { proven: false, reason: "writer_owner_live" };
const beforeProviderTurn = readFailedBeforeProviderTurnRecord(run.runnerProfileJson) !== null;
if (run.sessionIdAfter !== null) return { proven: false, reason: "process_recorded" };
if (!beforeProviderTurn && (run.processPid !== null || run.processGroupId !== null || run.processStartedAt !== null)
&& !await carriesInheritedRunnerIdentity(tx, run)) return { proven: false, reason: "process_recorded" };
const profile = object(run.runnerProfileJson);
const reference = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
if (profile.nativeWorkspaceSync != null || !reference.success || reference.data.state !== "prepared"
|| reference.data.workspaceId !== workspace.id || reference.data.epoch !== authority.epoch
|| reference.data.snapshotId !== authority.snapshotId || reference.data.leaseId !== authority.location.leaseId
|| reference.data.remoteCwd !== authority.location.remoteCwd) return { proven: false, reason: "reference_mismatch" };
// A rebound admission checkpoint may name this run; it proves nothing ran
// only while it holds no turn, result or terminal state of its own.
const checkpoint = object(profile.sessionCheckpoint);
if (object(checkpoint.identity).runId === runId && (checkpoint.activeTurnId != null || checkpoint.semanticResult != null
|| checkpoint.terminal != null || (checkpoint.terminalTurns != null
&& (!Array.isArray(checkpoint.terminalTurns) || checkpoint.terminalTurns.length > 0))))
return { proven: false, reason: "provider_session_started" };
const [event] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, workspace.companyId), eq(heartbeatRunEvents.runId, runId),
beforeProviderTurn ? inArray(heartbeatRunEvents.eventType, PROVIDER_TURN_EVENT_TYPES) : executionStartedEventPredicate())).limit(1);
if (event) return { proven: false, reason: "execution_recorded" };
const [result] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, workspace.companyId), eq(nativeRunResults.runId, runId))).limit(1);
if (result) return { proven: false, reason: "result_recorded" };
// The controller writes this checkpoint before every runner command.
const [commanded] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, workspace.companyId), eq(nativeRunnerCheckpoints.runId, runId))).limit(1);
if (commanded && !beforeProviderTurn) return { proven: false, reason: "runner_commanded" };
// Provisioning and saves record their operation before they touch files.
const [operation] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.heartbeatRunId, runId),
sql`(${workspaceOperations.executionWorkspaceId} is null or ${workspaceOperations.executionWorkspaceId} = ${workspace.id})`,
sql`${workspaceOperations.phase} not in (${sql.join(FILE_NEUTRAL_OPERATION_PHASES.map(phase => sql`${phase}`), sql`, `)})`)).limit(1);
if (operation) return { proven: false, reason: "workspace_operation_recorded" };
if (beforeProviderTurn) {
const [unsettled] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
eq(workspaceOperations.status, "running"), sql`${workspaceOperations.heartbeatRunId} is distinct from ${runId}`)).limit(1);
if (unsettled) return { proven: false, reason: "predecessor_save_unsettled" };
}
// A requested repository is cloned before its save is recorded. Only
// repositories already in the saved version are proven untouched.
const repositories = await tx.select().from(executionWorkspaceRepositories).where(and(
eq(executionWorkspaceRepositories.companyId, workspace.companyId),
eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id)));
if (repositories.some(row => !row.pinnedCommit || !input.isSavedRepository({ id: row.id, relativePath: row.relativePath,
repoUrl: row.repoUrl, ref: row.requestedRef === "HEAD" ? null : row.requestedRef, pinnedCommit: row.pinnedCommit })))
return { proven: false, reason: "repository_unsaved" };
return { proven: true, baseline };
}