mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately
## Linked Issues or Issue Description
**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.
**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.
**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.
**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.
**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.
## What Changed
- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.
## Verification
- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.
## Risks
- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
117 lines
8.7 KiB
TypeScript
117 lines
8.7 KiB
TypeScript
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import {
|
|
executionWorkspaceRepositories, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, nativeRunnerCheckpoints,
|
|
nativeRunResults, workspaceOperations, type Db,
|
|
} from "@paperclipai/db";
|
|
import { remoteWorkspaceRunReferenceSchema, type RemoteWorkspaceAuthority } from "@paperclipai/shared";
|
|
import { executionStartedEventPredicate } from "./pre-dispatch-execution-evidence.js";
|
|
import { runOwnerFenceHeldSql } from "./remote-workspace-writer-hold.js";
|
|
import { carriesInheritedRunnerIdentity } from "./native-runtime/native-inherited-runner-retry.js";
|
|
import { readFailedBeforeProviderTurnRecord } from "./native-runtime/failed-before-provider-turn.js";
|
|
import { readUnchangedBaseline, UNCHANGED_BASELINE_KEY, UNCHANGED_BASELINE_SCHEMA,
|
|
type UnchangedBaseline } from "./remote-workspace-unchanged-baseline.js";
|
|
|
|
const PROVIDER_TURN_EVENT_TYPES = ["turn.started", "turn.completed", "turn.failed", "provider.event", "provider.rpc_result",
|
|
"tool.execution.started", "item.started", "item.completed"];
|
|
|
|
export { readUnchangedBaseline, UNCHANGED_BASELINE_KEY, UNCHANGED_BASELINE_SCHEMA, type UnchangedBaseline };
|
|
|
|
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
|
|
type Repository = { id: string; relativePath: string; repoUrl: string; ref: string | null; pinnedCommit?: string };
|
|
|
|
const object = (value: unknown): Record<string, unknown> =>
|
|
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
|
|
|
|
/** Operations that cannot change the saved files: a restore writes exactly the
|
|
* saved version, and finalize and recovery rows are receipts that wrap a save
|
|
* (whose own snapshot operation is still counted). */
|
|
const FILE_NEUTRAL_OPERATION_PHASES = ["remote_workspace_restore", "workspace_finalize", "workspace_recovery"];
|
|
/** Terminal states of a run that ended without a result. */
|
|
const ENDED_WITHOUT_RESULT = ["failed", "cancelled", "interrupted", "timed_out"];
|
|
|
|
export type UnchangedProofRefusal =
|
|
| "baseline_missing" | "authority_mismatch" | "writer_not_ended" | "writer_owner_live" | "process_recorded"
|
|
| "reference_mismatch" | "provider_session_started" | "execution_recorded" | "result_recorded"
|
|
| "runner_commanded" | "workspace_operation_recorded" | "repository_unsaved" | "predecessor_save_unsettled";
|
|
|
|
/**
|
|
* Whether the ended writer of `authority` provably left its files exactly as
|
|
* the saved version it started from. The caller holds the workspace row lock.
|
|
* Any missing, ambiguous or older-release evidence refuses; a refusal keeps
|
|
* today's full save.
|
|
*/
|
|
export async function proveRemoteWorkspaceUnchanged(tx: Pick<Db, "select">, input: {
|
|
workspace: Workspace; authority: RemoteWorkspaceAuthority;
|
|
/** The caller owns the writer's finalization, so its own fence is expected. */
|
|
callerOwnsWriter?: boolean;
|
|
isSavedRepository: (repository: Repository) => boolean;
|
|
}): Promise<{ proven: true; baseline: UnchangedBaseline } | { proven: false; reason: UnchangedProofRefusal }> {
|
|
const { workspace, authority } = input;
|
|
const runId = authority.writerRunId;
|
|
const baseline = readUnchangedBaseline(workspace.metadata);
|
|
if (!baseline || baseline.releasedAt || !runId || baseline.runId !== runId) return { proven: false, reason: "baseline_missing" };
|
|
if (!authority.dirty || !authority.location || authority.epoch !== baseline.epoch || authority.snapshotId !== baseline.snapshotId)
|
|
return { proven: false, reason: "authority_mismatch" };
|
|
|
|
const [run] = await tx.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId, createdAt: heartbeatRuns.createdAt,
|
|
status: heartbeatRuns.status, processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
|
|
sessionIdAfter: heartbeatRuns.sessionIdAfter, runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
|
ownerFenceHeld: runOwnerFenceHeldSql() }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, runId)));
|
|
if (!run || !ENDED_WITHOUT_RESULT.includes(run.status)) return { proven: false, reason: "writer_not_ended" };
|
|
if (!input.callerOwnsWriter && run.ownerFenceHeld) return { proven: false, reason: "writer_owner_live" };
|
|
const beforeProviderTurn = readFailedBeforeProviderTurnRecord(run.runnerProfileJson) !== null;
|
|
if (run.sessionIdAfter !== null) return { proven: false, reason: "process_recorded" };
|
|
if (!beforeProviderTurn && (run.processPid !== null || run.processGroupId !== null || run.processStartedAt !== null)
|
|
&& !await carriesInheritedRunnerIdentity(tx, run)) return { proven: false, reason: "process_recorded" };
|
|
const profile = object(run.runnerProfileJson);
|
|
const reference = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
|
|
if (profile.nativeWorkspaceSync != null || !reference.success || reference.data.state !== "prepared"
|
|
|| reference.data.workspaceId !== workspace.id || reference.data.epoch !== authority.epoch
|
|
|| reference.data.snapshotId !== authority.snapshotId || reference.data.leaseId !== authority.location.leaseId
|
|
|| reference.data.remoteCwd !== authority.location.remoteCwd) return { proven: false, reason: "reference_mismatch" };
|
|
// A rebound admission checkpoint may name this run; it proves nothing ran
|
|
// only while it holds no turn, result or terminal state of its own.
|
|
const checkpoint = object(profile.sessionCheckpoint);
|
|
if (object(checkpoint.identity).runId === runId && (checkpoint.activeTurnId != null || checkpoint.semanticResult != null
|
|
|| checkpoint.terminal != null || (checkpoint.terminalTurns != null
|
|
&& (!Array.isArray(checkpoint.terminalTurns) || checkpoint.terminalTurns.length > 0))))
|
|
return { proven: false, reason: "provider_session_started" };
|
|
|
|
const [event] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
|
|
eq(heartbeatRunEvents.companyId, workspace.companyId), eq(heartbeatRunEvents.runId, runId),
|
|
beforeProviderTurn ? inArray(heartbeatRunEvents.eventType, PROVIDER_TURN_EVENT_TYPES) : executionStartedEventPredicate())).limit(1);
|
|
if (event) return { proven: false, reason: "execution_recorded" };
|
|
const [result] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
|
|
eq(nativeRunResults.companyId, workspace.companyId), eq(nativeRunResults.runId, runId))).limit(1);
|
|
if (result) return { proven: false, reason: "result_recorded" };
|
|
// The controller writes this checkpoint before every runner command.
|
|
const [commanded] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
|
|
eq(nativeRunnerCheckpoints.companyId, workspace.companyId), eq(nativeRunnerCheckpoints.runId, runId))).limit(1);
|
|
if (commanded && !beforeProviderTurn) return { proven: false, reason: "runner_commanded" };
|
|
// Provisioning and saves record their operation before they touch files.
|
|
const [operation] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.heartbeatRunId, runId),
|
|
sql`(${workspaceOperations.executionWorkspaceId} is null or ${workspaceOperations.executionWorkspaceId} = ${workspace.id})`,
|
|
sql`${workspaceOperations.phase} not in (${sql.join(FILE_NEUTRAL_OPERATION_PHASES.map(phase => sql`${phase}`), sql`, `)})`)).limit(1);
|
|
if (operation) return { proven: false, reason: "workspace_operation_recorded" };
|
|
if (beforeProviderTurn) {
|
|
const [unsettled] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
|
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id),
|
|
eq(workspaceOperations.status, "running"), sql`${workspaceOperations.heartbeatRunId} is distinct from ${runId}`)).limit(1);
|
|
if (unsettled) return { proven: false, reason: "predecessor_save_unsettled" };
|
|
}
|
|
// A requested repository is cloned before its save is recorded. Only
|
|
// repositories already in the saved version are proven untouched.
|
|
const repositories = await tx.select().from(executionWorkspaceRepositories).where(and(
|
|
eq(executionWorkspaceRepositories.companyId, workspace.companyId),
|
|
eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id)));
|
|
if (repositories.some(row => !row.pinnedCommit || !input.isSavedRepository({ id: row.id, relativePath: row.relativePath,
|
|
repoUrl: row.repoUrl, ref: row.requestedRef === "HEAD" ? null : row.requestedRef, pinnedCommit: row.pinnedCommit })))
|
|
return { proven: false, reason: "repository_unsaved" };
|
|
return { proven: true, baseline };
|
|
}
|