fix(workspaces): release an unchanged remote writer without a snapshot and wake its waiters (#20171)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote (sandbox) task workspace has one writer run at a time. The
writer holds the workspace until its files are saved as a snapshot
> - A run that a user comment interrupts right after its claim did no
work. No provider turn started and no command ran in the sandbox
> - Paperclip still kept its sandbox, then took a full snapshot of the
unchanged workspace. On a 2 GB repository this took 3 minutes
> - The next runs failed setup with `workspace_busy` ("ended, but its
workspace save has not settled"). The retry then waited for the
one-minute settling re-check after the save. The user's message waited
more than 4 minutes
> - This pull request releases such a writer at once, at the saved
version it started from, when durable evidence proves that it changed
nothing. It also wakes parked `workspace_busy` waiters when a hold
settles
> - The benefit is that an interrupted run no longer delays the next
message by a full workspace save

## Linked Issues or Issue Description

**What happened?**
A remote-workspace run was claimed (`native.coordinator.claim`). One
second later a board comment interrupted it (`operator_interrupted`). It
made no provider turn and no tool calls. Cancelled-workspace
preservation still retained its sandbox, and workspace recovery took a
full `remote_workspace_snapshot` of the unchanged workspace (3 minutes
for a 2 GB repository). Two follow-up runs were cancelled with
`workspace_busy`. Their retry was due during the save, and it parked for
the one-minute settling re-check after the save finished.

**Expected behavior**
A writer that provably changed nothing releases the workspace at once
and keeps its existing `snapshotId`. The next run is admitted without
`workspace_busy`. When a save does run, waiters start as soon as it
commits.

**Steps to reproduce**
1. Run a task on a remote sandbox workspace that already has a saved
version.
2. Interrupt the run with a board comment right after the claim, before
the provider turn.
3. See the full snapshot operation on the interrupted run and
`workspace_busy` on the follow-up runs.

**Paperclip version or commit**
Staging served revision b43cccbb0. Reproduced on `master` by the new
regression tests.

## What Changed

- `prepareRemoteWorkspace` records `remoteWorkspaceUnchangedBaseline`
(epoch, writer run, saved version) when preparation started from a clean
saved version and added nothing to it: no recovered attachments, no
repository outside the saved version and no workspace migration.
- New `proveRemoteWorkspaceUnchanged`
(`remote-workspace-unchanged-proof.ts`). It uses only durable evidence.
The writer run must have ended without a result, and its owner fence
must be clear. It must have no process fields, no execution, provider or
tool events (the shared `executionStartedEventPredicate`), no native
result, no runner checkpoint rows (the controller writes one before
every runner command), and no provider turn in a checkpoint that names
this run. It must have no workspace operation other than a restore or a
`workspace_finalize`/`workspace_recovery` receipt, so provisioning and
saves count as writes. Every task repository must be in the saved
version. Any missing or ambiguous evidence refuses.
- New `releaseUnchangedRemoteWorkspaceWriter`. It sets the authority to
clean with no writer, keeps `snapshotId`, marks the run reference
`saved`, logs `workspace.snapshot_skipped_unchanged` and sends the same
notifications as a final save. It never waits on a writer run that
another transaction holds (`skip locked`).
- The release applies in three places: the next run's
`reserveRemoteWorkspace`, cancelled/failed-workspace preservation
(`preserveNativeFailedWorkspace`), and `saveRemoteWorkspace` with the
final boundary. The final save returns the same version on a repeated
call, so a recovery that raced the release finishes without a capture.
- The final-save commit and the unchanged release call
`wakeWorkspaceBusyWaitersInTx`. A parked waiter whose holder was that
writer is due at once, not at the one-minute settling re-check.
- `doc/execution-semantics.md` describes the release and the wake.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-unchanged-release.test.ts` (13 tests,
embedded Postgres, real archive capture):
- Staging shape (claim, then interrupt with no provider work): the next
reservation is admitted. No snapshot operation is recorded. The run
reference is `saved` at the old `snapshotId`. Red on `master`: "The
workspace still has an unfinished writer or save operation."
- A later final save, behind the `workspace_finalize` receipt that
production finalizers publish first, returns the same version with no
capture, a repeat returns it again, and a parked waiter becomes due now.
Red on `master`: a new snapshot is captured.
- Negatives that still do a full save: `turn.started`, `provider.event`,
`tool.execution.started`, `native.process_start_requested`, a
provisioning command that wrote into the workspace, a recorded process,
a live owner fence, and a writer that is still running.
- Upgrade safety: no baseline (a writer prepared by the previous
release), a baseline without its saved version, and a baseline with
another schema each keep the full save.
  - A full final save wakes the parked waiter. Red on `master`.
- `native-cancelled-workspace-preservation.test.ts`: a new case shows
that a cancelled writer with a baseline and no work is released, with no
retained lease and no recovery action. Red on `master` (the
restore-recovery hold was recorded). The other 44 cases pass unchanged.
- I also ran the related remote-workspace, workspace-recovery, native
preservation and run-dispatch suites locally.

## Risks

- A false "unchanged" verdict would drop unsaved files. The proof
requires a positive baseline from preparation and refuses on any write
record. Only `.paperclip-runtime` (excluded from snapshots) is written
by runner staging before a process starts. A deploy that lands mid-run
keeps the full save, because the proof reads only durable rows.
- Previous-release state: a writer prepared without the baseline (or
with an unknown or partial one) keeps today's full save. An older server
ignores the new metadata key.
- Waiters now start sooner after a final save. Arrival order among
waiters is unchanged (the park check still admits the earliest first).
- Other ways a hold can settle (for example a stop receipt that turns an
unused reservation free) still rely on the one-minute re-check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits and a read-only browser for staging
evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 22:37:52 -06:00
1 parent 8e5959c759
commit 098f7d370b
6 files changed
+488 -7

No files matched your search

@@ -460,6 +460,31 @@ import { readRemoteWorkspaceAuthority } from "@paperclipai/shared";
} finally { await worker.stop(); }
});
it("hands the workspace on at its saved version when the stopped writer never started provider work", async () => {
const f = await seed();
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceUnchangedBaseline: {
schema: "paperclip.remote-workspace-unchanged-baseline.v1", epoch: 2, runId: f.runId, snapshotId: f.reference.snapshotId,
} } }).where(eq(executionWorkspaces.id, f.workspaceId));
const worker = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
try {
await registerCancelledWorkspacePreservationWork(db, worker, () => true, async () => true).ready;
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
expect(run.resultJson).not.toHaveProperty("workspaceRestoreRecovery");
expect(run.runnerProfileJson?.remoteWorkspace).toMatchObject({ state: "saved", snapshotId: f.reference.snapshotId });
const [released] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
const authority = readRemoteWorkspaceAuthority(released.metadata)!;
expect(authority).toMatchObject({ epoch: 2, snapshotId: f.reference.snapshotId, writerRunId: null, dirty: false });
expect(await readRemoteWorkspaceWriterHold(db, { companyId: f.companyId, authority })).toEqual({ kind: "free" });
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.leaseId)))[0])
.toMatchObject({ status: "active", leasePolicy: "ephemeral" });
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId))).toHaveLength(0);
// A later pass finds nothing left to keep.
await preserveCancelledNativeWorkspaces(db);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, f.companyId))).toHaveLength(0);
} finally { await worker.stop(); }
});
it("rolls preservation back with its outer transaction", async () => {
const f = await seed();
await expect(db.transaction(async tx => {
@@ -18,6 +18,7 @@ import { preserveRemoteWorkspacePreparationFailure } from "./native-workspace-fi
import { pinFailedObservedWorkspace } from "./native-observed-workspace-preservation.js";
import { hasConfirmedSandboxStopAndRetain } from "../sandbox-stop-and-retain.js";
import { preparedRemoteWorkspaceSource } from "../remote-workspace-lifecycle.js";
import { releaseUnchangedRemoteWorkspaceWriterById } from "../remote-workspaces.js";
const reusableLeasePolicies = ["reuse_by_environment", "reuse_by_execution_workspace"];
@@ -137,6 +138,12 @@ export async function preserveNativeFailedWorkspace(db: Db, selected: Pick<typeo
)).limit(1);
if (!accepted) return;
}
// A run that ended without changing its files has nothing to keep: its
// workspace is handed on at the saved version it started from.
if (!exhaustedAcceptedCopyback && !coordinator.resultId && coordinator.phase === "terminal_failure"
&& run.runnerProfileJson?.nativeWorkspaceSync == null
&& await releaseUnchangedRemoteWorkspaceWriterById(tx, { companyId: run.companyId,
workspaceId: currentReference.workspaceId, runId: run.id })) return;
// A stopped reusable sandbox is retained only with its exact stop receipt
// while the workspace is still bound to this writer. Otherwise record nothing.
if (!exhaustedAcceptedCopyback && source.status !== "active" && reusableLeasePolicies.includes(source.leasePolicy) && (!hasConfirmedSandboxStopAndRetain(source)