Files
PaperClipAI/server/src/services/remote-workspace-unchanged-proof.ts
T
DottaandClaude Opus 5.5 098f7d370b fix(workspaces): release an unchanged remote writer without a snapshot and wake its waiters (#20171)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote (sandbox) task workspace has one writer run at a time. The
writer holds the workspace until its files are saved as a snapshot
> - A run that a user comment interrupts right after its claim did no
work. No provider turn started and no command ran in the sandbox
> - Paperclip still kept its sandbox, then took a full snapshot of the
unchanged workspace. On a 2 GB repository this took 3 minutes
> - The next runs failed setup with `workspace_busy` ("ended, but its
workspace save has not settled"). The retry then waited for the
one-minute settling re-check after the save. The user's message waited
more than 4 minutes
> - This pull request releases such a writer at once, at the saved
version it started from, when durable evidence proves that it changed
nothing. It also wakes parked `workspace_busy` waiters when a hold
settles
> - The benefit is that an interrupted run no longer delays the next
message by a full workspace save

## Linked Issues or Issue Description

**What happened?**
A remote-workspace run was claimed (`native.coordinator.claim`). One
second later a board comment interrupted it (`operator_interrupted`). It
made no provider turn and no tool calls. Cancelled-workspace
preservation still retained its sandbox, and workspace recovery took a
full `remote_workspace_snapshot` of the unchanged workspace (3 minutes
for a 2 GB repository). Two follow-up runs were cancelled with
`workspace_busy`. Their retry was due during the save, and it parked for
the one-minute settling re-check after the save finished.

**Expected behavior**
A writer that provably changed nothing releases the workspace at once
and keeps its existing `snapshotId`. The next run is admitted without
`workspace_busy`. When a save does run, waiters start as soon as it
commits.

**Steps to reproduce**
1. Run a task on a remote sandbox workspace that already has a saved
version.
2. Interrupt the run with a board comment right after the claim, before
the provider turn.
3. See the full snapshot operation on the interrupted run and
`workspace_busy` on the follow-up runs.

**Paperclip version or commit**
Staging served revision b43cccbb0. Reproduced on `master` by the new
regression tests.

## What Changed

- `prepareRemoteWorkspace` records `remoteWorkspaceUnchangedBaseline`
(epoch, writer run, saved version) when preparation started from a clean
saved version and added nothing to it: no recovered attachments, no
repository outside the saved version and no workspace migration.
- New `proveRemoteWorkspaceUnchanged`
(`remote-workspace-unchanged-proof.ts`). It uses only durable evidence.
The writer run must have ended without a result, and its owner fence
must be clear. It must have no process fields, no execution, provider or
tool events (the shared `executionStartedEventPredicate`), no native
result, no runner checkpoint rows (the controller writes one before
every runner command), and no provider turn in a checkpoint that names
this run. It must have no workspace operation other than a restore or a
`workspace_finalize`/`workspace_recovery` receipt, so provisioning and
saves count as writes. Every task repository must be in the saved
version. Any missing or ambiguous evidence refuses.
- New `releaseUnchangedRemoteWorkspaceWriter`. It sets the authority to
clean with no writer, keeps `snapshotId`, marks the run reference
`saved`, logs `workspace.snapshot_skipped_unchanged` and sends the same
notifications as a final save. It never waits on a writer run that
another transaction holds (`skip locked`).
- The release applies in three places: the next run's
`reserveRemoteWorkspace`, cancelled/failed-workspace preservation
(`preserveNativeFailedWorkspace`), and `saveRemoteWorkspace` with the
final boundary. The final save returns the same version on a repeated
call, so a recovery that raced the release finishes without a capture.
- The final-save commit and the unchanged release call
`wakeWorkspaceBusyWaitersInTx`. A parked waiter whose holder was that
writer is due at once, not at the one-minute settling re-check.
- `doc/execution-semantics.md` describes the release and the wake.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-unchanged-release.test.ts` (13 tests,
embedded Postgres, real archive capture):
- Staging shape (claim, then interrupt with no provider work): the next
reservation is admitted. No snapshot operation is recorded. The run
reference is `saved` at the old `snapshotId`. Red on `master`: "The
workspace still has an unfinished writer or save operation."
- A later final save, behind the `workspace_finalize` receipt that
production finalizers publish first, returns the same version with no
capture, a repeat returns it again, and a parked waiter becomes due now.
Red on `master`: a new snapshot is captured.
- Negatives that still do a full save: `turn.started`, `provider.event`,
`tool.execution.started`, `native.process_start_requested`, a
provisioning command that wrote into the workspace, a recorded process,
a live owner fence, and a writer that is still running.
- Upgrade safety: no baseline (a writer prepared by the previous
release), a baseline without its saved version, and a baseline with
another schema each keep the full save.
  - A full final save wakes the parked waiter. Red on `master`.
- `native-cancelled-workspace-preservation.test.ts`: a new case shows
that a cancelled writer with a baseline and no work is released, with no
retained lease and no recovery action. Red on `master` (the
restore-recovery hold was recorded). The other 44 cases pass unchanged.
- I also ran the related remote-workspace, workspace-recovery, native
preservation and run-dispatch suites locally.

## Risks

- A false "unchanged" verdict would drop unsaved files. The proof
requires a positive baseline from preparation and refuses on any write
record. Only `.paperclip-runtime` (excluded from snapshots) is written
by runner staging before a process starts. A deploy that lands mid-run
keeps the full save, because the proof reads only durable rows.
- Previous-release state: a writer prepared without the baseline (or
with an unknown or partial one) keeps today's full save. An older server
ignores the new metadata key.
- Waiters now start sooner after a final save. Arrival order among
waiters is unchanged (the park check still admits the earliest first).
- Other ways a hold can settle (for example a stop receipt that turns an
unused reservation free) still rely on the one-minute re-check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits and a read-only browser for staging
evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:37:52 -06:00

121 lines
8.3 KiB
TypeScript

import { and, eq, sql } from "drizzle-orm";
import {
executionWorkspaceRepositories, executionWorkspaces, heartbeatRunEvents, heartbeatRuns, nativeRunnerCheckpoints,
nativeRunResults, workspaceOperations, type Db,
} from "@paperclipai/db";
import { remoteWorkspaceRunReferenceSchema, type RemoteWorkspaceAuthority } from "@paperclipai/shared";
import { executionStartedEventPredicate } from "./pre-dispatch-execution-evidence.js";
import { runOwnerFenceHeldSql } from "./remote-workspace-writer-hold.js";
/**
* A writer's prepared files equal its saved version only while nothing ran in
* them. Preparation records that starting point; everything after it leaves a
* durable record (an operation, a runner checkpoint, a run event or a
* process) before it can write. A run with none of those changed nothing.
*/
export const UNCHANGED_BASELINE_KEY = "remoteWorkspaceUnchangedBaseline";
export const UNCHANGED_BASELINE_SCHEMA = "paperclip.remote-workspace-unchanged-baseline.v1";
export type UnchangedBaseline = {
schema: typeof UNCHANGED_BASELINE_SCHEMA; epoch: number; runId: string; snapshotId: string;
/** Set once the writer was released without a new save. */
releasedAt?: string;
};
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
type Repository = { id: string; relativePath: string; repoUrl: string; ref: string | null; pinnedCommit?: string };
const object = (value: unknown): Record<string, unknown> =>
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
/** Missing or malformed fields (an older or partial record) never prove anything. */
export function readUnchangedBaseline(metadata: Record<string, unknown> | null | undefined): UnchangedBaseline | null {
const value = object(metadata?.[UNCHANGED_BASELINE_KEY]);
if (value.schema !== UNCHANGED_BASELINE_SCHEMA || !Number.isSafeInteger(value.epoch) || (value.epoch as number) < 1
|| typeof value.runId !== "string" || !value.runId || typeof value.snapshotId !== "string" || !value.snapshotId
|| (value.releasedAt !== undefined && typeof value.releasedAt !== "string")) return null;
return value as UnchangedBaseline;
}
/** Operations that cannot change the saved files: a restore writes exactly the
* saved version, and finalize and recovery rows are receipts that wrap a save
* (whose own snapshot operation is still counted). */
const FILE_NEUTRAL_OPERATION_PHASES = ["remote_workspace_restore", "workspace_finalize", "workspace_recovery"];
/** Terminal states of a run that ended without a result. */
const ENDED_WITHOUT_RESULT = ["failed", "cancelled", "interrupted", "timed_out"];
export type UnchangedProofRefusal =
| "baseline_missing" | "authority_mismatch" | "writer_not_ended" | "writer_owner_live" | "process_recorded"
| "reference_mismatch" | "provider_session_started" | "execution_recorded" | "result_recorded"
| "runner_commanded" | "workspace_operation_recorded" | "repository_unsaved";
/**
* Whether the ended writer of `authority` provably left its files exactly as
* the saved version it started from. The caller holds the workspace row lock.
* Any missing, ambiguous or older-release evidence refuses; a refusal keeps
* today's full save.
*/
export async function proveRemoteWorkspaceUnchanged(tx: Pick<Db, "select">, input: {
workspace: Workspace; authority: RemoteWorkspaceAuthority;
/** The caller owns the writer's finalization, so its own fence is expected. */
callerOwnsWriter?: boolean;
isSavedRepository: (repository: Repository) => boolean;
}): Promise<{ proven: true; baseline: UnchangedBaseline } | { proven: false; reason: UnchangedProofRefusal }> {
const { workspace, authority } = input;
const runId = authority.writerRunId;
const baseline = readUnchangedBaseline(workspace.metadata);
if (!baseline || baseline.releasedAt || !runId || baseline.runId !== runId) return { proven: false, reason: "baseline_missing" };
if (!authority.dirty || !authority.location || authority.epoch !== baseline.epoch || authority.snapshotId !== baseline.snapshotId)
return { proven: false, reason: "authority_mismatch" };
const [run] = await tx.select({ status: heartbeatRuns.status, processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
sessionIdAfter: heartbeatRuns.sessionIdAfter, runnerProfileJson: heartbeatRuns.runnerProfileJson,
ownerFenceHeld: runOwnerFenceHeldSql() }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, runId)));
if (!run || !ENDED_WITHOUT_RESULT.includes(run.status)) return { proven: false, reason: "writer_not_ended" };
if (!input.callerOwnsWriter && run.ownerFenceHeld) return { proven: false, reason: "writer_owner_live" };
if (run.processPid !== null || run.processGroupId !== null || run.processStartedAt !== null || run.sessionIdAfter !== null)
return { proven: false, reason: "process_recorded" };
const profile = object(run.runnerProfileJson);
const reference = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
if (profile.nativeWorkspaceSync != null || !reference.success || reference.data.state !== "prepared"
|| reference.data.workspaceId !== workspace.id || reference.data.epoch !== authority.epoch
|| reference.data.snapshotId !== authority.snapshotId || reference.data.leaseId !== authority.location.leaseId
|| reference.data.remoteCwd !== authority.location.remoteCwd) return { proven: false, reason: "reference_mismatch" };
// A rebound admission checkpoint may name this run; it proves nothing ran
// only while it holds no turn, result or terminal state of its own.
const checkpoint = object(profile.sessionCheckpoint);
if (object(checkpoint.identity).runId === runId && (checkpoint.activeTurnId != null || checkpoint.semanticResult != null
|| checkpoint.terminal != null || (checkpoint.terminalTurns != null
&& (!Array.isArray(checkpoint.terminalTurns) || checkpoint.terminalTurns.length > 0))))
return { proven: false, reason: "provider_session_started" };
const [event] = await tx.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, workspace.companyId), eq(heartbeatRunEvents.runId, runId),
executionStartedEventPredicate())).limit(1);
if (event) return { proven: false, reason: "execution_recorded" };
const [result] = await tx.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, workspace.companyId), eq(nativeRunResults.runId, runId))).limit(1);
if (result) return { proven: false, reason: "result_recorded" };
// The controller writes this checkpoint before every runner command.
const [commanded] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints).where(and(
eq(nativeRunnerCheckpoints.companyId, workspace.companyId), eq(nativeRunnerCheckpoints.runId, runId))).limit(1);
if (commanded) return { proven: false, reason: "runner_commanded" };
// Provisioning and saves record their operation before they touch files.
const [operation] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.heartbeatRunId, runId),
sql`(${workspaceOperations.executionWorkspaceId} is null or ${workspaceOperations.executionWorkspaceId} = ${workspace.id})`,
sql`${workspaceOperations.phase} not in (${sql.join(FILE_NEUTRAL_OPERATION_PHASES.map(phase => sql`${phase}`), sql`, `)})`)).limit(1);
if (operation) return { proven: false, reason: "workspace_operation_recorded" };
// A requested repository is cloned before its save is recorded. Only
// repositories already in the saved version are proven untouched.
const repositories = await tx.select().from(executionWorkspaceRepositories).where(and(
eq(executionWorkspaceRepositories.companyId, workspace.companyId),
eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id)));
if (repositories.some(row => !row.pinnedCommit || !input.isSavedRepository({ id: row.id, relativePath: row.relativePath,
repoUrl: row.repoUrl, ref: row.requestedRef === "HEAD" ? null : row.requestedRef, pinnedCommit: row.pinnedCommit })))
return { proven: false, reason: "repository_unsaved" };
return { proven: true, baseline };
}