mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer
## Linked Issues or Issue Description
**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.
**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.
**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.
**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.
## What Changed
- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.
Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
- After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.
## Risks
- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
31 lines
1.9 KiB
TypeScript
31 lines
1.9 KiB
TypeScript
import { and, eq } from "drizzle-orm";
|
|
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
|
import { isUuidLike, type IssueRecoveryAction } from "@paperclipai/shared";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
|
|
export const NATIVE_WORKSPACE_OWNER_UNVERIFIED = "native_workspace_finalization_owner_unverified";
|
|
/** Describes the wait. It never asks a person to inspect a server or sandbox. */
|
|
export const NATIVE_WORKSPACE_OWNER_WAIT_NEXT_ACTION = "Paperclip is holding this saved result until it can confirm that the previous server stopped, so the work is not saved twice. Saving then resumes without repeating the agent's work. If this platform cannot confirm it, an owner or administrator can resume the save from this task (Resume saved result).";
|
|
|
|
export function hasNativeWorkspaceFinalizationOwner(profile: Record<string, unknown> | null | undefined): boolean {
|
|
return profile?.nativeWorkspaceFinalizationOwner != null;
|
|
}
|
|
|
|
/** A terminal task does not prove that an older controller's copyback stopped. */
|
|
export async function hasUnverifiedNativeWorkspaceFinalizationOwner(
|
|
db: Db,
|
|
action: Pick<IssueRecoveryAction, "companyId" | "sourceIssueId" | "cause" | "fingerprint" | "evidence">,
|
|
): Promise<boolean> {
|
|
if (action.cause !== NATIVE_WORKSPACE_OWNER_UNVERIFIED) return false;
|
|
const runId = action.evidence?.runId;
|
|
if (typeof runId !== "string" || !isUuidLike(runId)) return false;
|
|
const [run] = await db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, action.companyId),
|
|
eq(heartbeatRuns.id, runId),
|
|
eq(heartbeatRuns.nativeIssueId, action.sourceIssueId),
|
|
eq(heartbeatRuns.runtimeMode, "native"),
|
|
)).limit(1);
|
|
const owner = run?.profile?.nativeWorkspaceFinalizationOwner;
|
|
return hasNativeWorkspaceFinalizationOwner(run?.profile) && action.fingerprint === nativeSha256({ runId, owner });
|
|
}
|