feat(recovery): let the board confirm that an unrecorded owner stopped (#20067)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, recovery releases an in-progress record only once
its owner is proven gone, never by age
> - A run lease saved before owners were stamped has no owner Paperclip
can judge on an ephemeral (multi-host) server, so it stays stuck and
holds its run's sandbox forever
> - Only a person can tell that the previous server stopped (for
example, its container was replaced)
> - This pull request asks the board on the stuck task and lets an owner
confirm the stop, with an activity-log entry; recovery then releases the
record on its normal path
> - The benefit is that legacy stuck records have a way out that never
guesses

## Linked Issues or Issue Description

Follow-up to the Phase 2 durable-ownership work: #20054 (controller
instance lease) and #19883 (process-space recording). It generalizes the
existing "confirm the previous server stopped" action for native
controllers (`native_controller_host_unverified`). No duplicate PR was
found.

**What happened?**

On an ephemeral server, an active run lease of a finished run that was
saved before holders were stamped has no owner stamp. The ownership
reconciler reports it `unprovable (no_owner_stamp)` and never settles
it. No operator action exists, so the lease and its sandbox stay
reserved.

**Expected behavior**

The task shows the stuck resource, and a board owner can confirm that
the previous server stopped. Recovery then releases the lease. Agents
and operators without the owner-stop permission cannot.

**Steps to reproduce**

1. Run with `PAPERCLIP_EPHEMERAL_SERVER=true`.
2. Have an active environment lease with no `leaseController` stamp,
whose run is terminal.
3. Run an ownership reconciliation pass. The lease stays `unprovable`
with no action on the task.

**Paperclip version or commit**

`master` before this PR.

## What Changed

- `packages/shared`: `confirmOwnerGoneSchema` (`actionId`, `recordId`,
`stopEvidence` of 20 to 12000 characters).
- `recovery/owner-gone-attestation.ts`:
- `requestOwnerGoneConfirmation(kind)` is an `onUnprovable` hook. For a
legacy unprovable owner (`no_owner_stamp`, `process_space_unrecorded`,
`missing_identity`) on a record that has a task, it opens one idempotent
board recovery action on that task (cause `ownership_owner_unprovable`).
- `confirmOwnerGone(...)` verifies the exact action and record. It asks
the kind to write the attestation (compare-and-swap), resolves the
action, and writes an `issue.ownership_owner_gone_confirmed`
activity-log entry with the actor and the stated evidence.
- Ownership registry:
  - `OwnableKind.attestOwnerGone` is a new optional hook.
- `environment_run_lease_active` implements it: only an unstamped active
lease of a terminal run, with the notification for the cleanup path.
- The lease's owner then reads `operator_attested`, which proves gone,
so the existing settle releases it. A holder stamped since the board was
asked owns the lease, and its stamp decides.
- Route: `POST /api/issues/<built-in function
id>/recovery-actions/confirm-owner-gone`. It uses `assertBoard` plus
`runtime:confirm_owner_stop`, the same board-only permission as
confirm-controller-stopped. It is registered in OpenAPI.
- UI: `issuesApi.confirmOwnerGone`. The task's recovery card shows the
existing `PreviousServerStopConfirmation` for this cause, with
resource-specific wording; owners confirm and operators see the wait.
`isPreviousServerStopCause` includes the cause, so the generic card does
not offer other resolutions.

## Verification

- `owner-gone-attestation.test.ts`:
  - the stuck lease opens one board action on the task;
- confirmation resolves the action, writes the activity log, and the
next pass settles the lease (`operator_attested`) exactly once;
  - a mismatched record is refused with 409;
- a holder stamped after the ask is refused and the lease stays active;
  - no action for a judgeable owner or for a record with no task.
- Red, with the `onUnprovable` hook removed: 2 of 3 fail.
- UI: `WorkspaceExportRecovery.control.test.tsx` checks that the owner
confirms with the right payload and that an operator sees the wait
without the control (47/47).
- Server suites: openapi-routes, ownership and recovery suites, plus the
existing native-controller-host recovery. Low-memory server typecheck is
clean; `check:token-gates` is clean.

## Risks

- The board can attest a server stopped that is in fact running. The
action and the UI copy ask for the evidence, and the activity log
records who attested what. This is the same trust model as
confirm-controller-stopped.
- Only run leases opt in so far. Records without a task (login/probe
leases) keep the legacy idle sweep.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-08 09:46:36 -06:00
1 parent 66a2cbab98
commit 0746d1203d
13 files changed
+313 -4

No files matched your search

+1
View File
@@ -1996,6 +1996,7 @@ export {
reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
confirmNativeControllerStoppedSchema,
confirmOwnerGoneSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
checkoutIssueSchema,
+1
View File
@@ -452,6 +452,7 @@ export {
reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
confirmNativeControllerStoppedSchema,
confirmOwnerGoneSchema,
issueReviewRequestSchema,
issueExecutionWorkspaceSettingsSchema,
checkoutIssueSchema,
+8
View File
@@ -599,6 +599,14 @@ export const confirmNativeControllerStoppedSchema = z.object({
stopEvidence: z.string().trim().min(20).max(12000),
}).strict();
/** Board attestation that the owner of a record Paperclip cannot judge (saved
* before owners were recorded) has stopped, so recovery may settle the record. */
export const confirmOwnerGoneSchema = z.object({
actionId: z.string().guid(),
recordId: z.string().guid(),
stopEvidence: z.string().trim().min(20).max(12000),
}).strict();
export const retryWorkspaceExportSchema = z.object({
actionId: z.string().guid(),
runId: z.string().guid(),
@@ -0,0 +1,115 @@
import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
activityLog, agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, heartbeatRuns,
issueRecoveryActions, issues,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { ENVIRONMENT_LEASE_CONTROLLER_KEY } from "../services/environment-lease-controller.js";
import { confirmOwnerGone, OWNER_UNPROVABLE_CAUSE } from "../services/recovery/owner-gone-attestation.js";
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
import { recoveryService } from "../services/recovery/service.js";
const KIND = "environment_run_lease_active";
const stopEvidence = "The old server container was replaced and is no longer running.";
describe("board confirmation that an unrecorded owner stopped", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-owner-gone-attestation-");
db = createDb(temporary.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => temporary?.cleanup());
afterEach(() => vi.unstubAllEnvs());
/** A run lease saved before holders were stamped, on an ephemeral server, after its run finished. */
async function seed(holder: Record<string, unknown> | null = null) {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), leaseId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Owner", issuePrefix: `O${companyId.slice(0, 6)}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "codex_local", status: "idle" });
await db.insert(issues).values({ id: issueId, companyId, title: "Task", status: "in_progress", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, status: "failed", invocationSource: "on_demand" });
await db.insert(environmentLeases).values({ id: leaseId, companyId, issueId, heartbeatRunId: runId, status: "active",
leasePolicy: "ephemeral", metadata: holder ? { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: holder } : {} });
return { companyId, issueId, runId, leaseId };
}
const pass = async () => (await runOwnershipReconciliationPass(db, { state: createOwnershipReportState() }))
.kinds.find(kind => kind.kind === KIND)!;
const actionFor = async (issueId: string) => (await db.select().from(issueRecoveryActions)
.where(and(eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.cause, OWNER_UNPROVABLE_CAUSE))))[0];
const lease = async (leaseId: string) => (await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)))[0]!;
it("asks the board on the task, and releases the lease once the board confirms the stop", async () => {
const { companyId, issueId, runId, leaseId } = await seed();
expect(await pass()).toMatchObject({ settled: 0, reasons: { no_owner_stamp: 1 } });
const action = await actionFor(issueId);
expect(action).toMatchObject({ ownerType: "board", status: "active", evidence: { kind: KIND, recordId: leaseId, runId } });
await pass();
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))).toHaveLength(1);
await expect(confirmOwnerGone({ db, kinds: OWNABLE_KINDS, companyId, issueId, actionId: action!.id, recordId: leaseId,
stopEvidence, actorId: "board-user" })).resolves.toEqual({ recordId: leaseId, status: "queued" });
expect(await actionFor(issueId)).toMatchObject({ status: "resolved", evidence: { ownerGone: { actorId: "board-user", stopEvidence } } });
const [entry] = await db.select().from(activityLog).where(and(eq(activityLog.entityId, issueId),
eq(activityLog.action, "issue.ownership_owner_gone_confirmed")));
expect(entry).toMatchObject({ actorType: "user", actorId: "board-user", runId,
details: expect.objectContaining({ kind: KIND, recordId: leaseId, stopEvidence }) });
expect(await pass()).toMatchObject({ settled: 1, reasons: { operator_attested: 1 } });
expect((await lease(leaseId)).status).toBe("pending_cleanup");
expect(await pass()).toMatchObject({ settled: 0 });
});
it("refuses a stale or mismatched confirmation, and never attests a stamped holder", async () => {
const { companyId, issueId, leaseId } = await seed();
await pass();
const action = (await actionFor(issueId))!;
await expect(confirmOwnerGone({ db, kinds: OWNABLE_KINDS, companyId, issueId, actionId: action.id, recordId: randomUUID(),
stopEvidence, actorId: "board-user" })).rejects.toMatchObject({ status: 409 });
// A holder stamped since the board was asked owns the lease now: its stamp decides.
await db.update(environmentLeases).set({ metadata: { [ENVIRONMENT_LEASE_CONTROLLER_KEY]: {
bootId: "new-boot", pid: 7, hostHash: "another-host", processStartedAt: null } } }).where(eq(environmentLeases.id, leaseId));
await expect(confirmOwnerGone({ db, kinds: OWNABLE_KINDS, companyId, issueId, actionId: action.id, recordId: leaseId,
stopEvidence, actorId: "board-user" })).rejects.toMatchObject({ status: 409 });
expect(await actionFor(issueId)).toMatchObject({ status: "active" });
expect((await lease(leaseId)).status).toBe("active");
});
it("is never closed by ordinary recovery, even once the task is done", async () => {
const { companyId, issueId } = await seed();
await pass();
await db.update(issues).set({ status: "done" }).where(eq(issues.id, issueId));
await recoveryService(db, { enqueueWakeup: vi.fn() } as never).reconcileActiveRecoveryActions({ companyId });
expect(await actionFor(issueId)).toMatchObject({ status: "active" });
});
it("closes the request without attesting when the lease was released another way", async () => {
const { companyId, issueId, leaseId } = await seed();
await pass();
const action = (await actionFor(issueId))!;
await db.update(environmentLeases).set({ status: "released" }).where(eq(environmentLeases.id, leaseId));
await expect(confirmOwnerGone({ db, kinds: OWNABLE_KINDS, companyId, issueId, actionId: action.id, recordId: leaseId,
stopEvidence, actorId: "board-user" })).resolves.toEqual({ recordId: leaseId, status: "released" });
const resolved = await actionFor(issueId);
expect(resolved).toMatchObject({ status: "resolved", evidence: { ownerGoneNotNeeded: { actorId: "board-user" } } });
expect(resolved?.evidence).not.toHaveProperty("ownerGone");
expect((await lease(leaseId)).metadata).not.toHaveProperty("ownerGoneAttestation");
});
it("does not ask about an owner Paperclip can judge, or a record with no task", async () => {
const stamped = await seed({ bootId: "boot", pid: 7, hostHash: "another-host", processStartedAt: null });
await pass();
expect(await actionFor(stamped.issueId)).toBeUndefined();
const orphan = await seed();
await db.update(environmentLeases).set({ issueId: null }).where(eq(environmentLeases.id, orphan.leaseId));
await db.update(heartbeatRuns).set({ issueId: null }).where(eq(heartbeatRuns.id, orphan.runId));
await pass();
expect(await actionFor(orphan.issueId)).toBeUndefined();
});
});
+18
View File
@@ -40,6 +40,8 @@ import { createIssueReadTiming } from "../services/issue-read-timing.js";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { resumeNativeWorkspaceAfterOwnerStop } from "../services/native-runtime/native-workspace-owner-recovery.js";
import { confirmNativeControllerStopped, NATIVE_CONTROLLER_HOST_UNVERIFIED } from "../services/native-runtime/native-controller-host-recovery.js";
import { confirmOwnerGone, OWNER_UNPROVABLE_CAUSE } from "../services/recovery/owner-gone-attestation.js";
import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js";
import { retryNativeWorkspaceExport } from "../services/native-runtime/native-workspace-export-retry.js";
import { reconstructNativeWorkspace, unreconciledNativeExportPredicate } from "../services/native-runtime/native-workspace-reconstruction.js";
import { reconstructWorkspaceSchema } from "@paperclipai/shared";
@@ -124,6 +126,7 @@ import {
taskRetrySchema,
resumeWorkspaceFinalizationSchema,
confirmNativeControllerStoppedSchema,
confirmOwnerGoneSchema,
runnerGoalActionRequestSchema,
feedbackTargetTypeSchema,
feedbackTraceStatusSchema,
@@ -10112,6 +10115,17 @@ export function issueRoutes(
res.status(202).json(receipt);
});
router.post("/issues/:id/recovery-actions/confirm-owner-gone", validate(confirmOwnerGoneSchema), async (req, res) => {
assertBoard(req);
const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found");
if (!issue || !(await assertIssueReadAllowed(req, res, issue))) return;
const decision = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", resource: { type: "company", companyId: issue.companyId } });
if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
const receipt = await confirmOwnerGone({ db, kinds: OWNABLE_KINDS, companyId: issue.companyId, issueId: issue.id,
...req.body, actorId: getActorInfo(req).actorId });
res.status(202).json(receipt);
});
router.post("/issues/:id/recovery-actions/retry-workspace-export", validate(retryWorkspaceExportSchema), async (req, res) => {
assertBoard(req);
const issue = await getAccessibleResource(req, res, svc.getById(req.params.id as string), "Issue not found");
@@ -10376,6 +10390,10 @@ export function issueRoutes(
throw conflict("Unsafe workspace export recovers automatically without another provider turn.", { code: "workspace_export_automatic_recovery" });
}
if ((outcome === "restored" || outcome === "false_positive") && activeRecoveryAction.cause === OWNER_UNPROVABLE_CAUSE) {
throw conflict("Confirm that the previous server stopped to release this resource.", { code: "ownership_owner_stop_required" });
}
if ((outcome === "restored" || outcome === "false_positive") && activeRecoveryAction.cause === NATIVE_CONTROLLER_HOST_UNVERIFIED) {
throw conflict("Confirm that the previous server stopped to resume this run.", { code: "native_controller_stop_required" });
}
+10
View File
@@ -261,6 +261,7 @@ import {
reconstructWorkspaceSchema,
resumeWorkspaceFinalizationSchema,
confirmNativeControllerStoppedSchema,
confirmOwnerGoneSchema,
cancelIssueThreadInteractionSchema,
// Secret provider configs and remote import
createSecretProviderConfigSchema,
@@ -11064,6 +11065,15 @@ registerCurrentRoute({
responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict },
});
registerCurrentRoute({
method: "post",
path: "/api/issues/{id}/recovery-actions/confirm-owner-gone",
tags: ["issues"],
summary: "Confirm that the unrecorded server holding a stuck recovery record stopped, so recovery can release it",
body: confirmOwnerGoneSchema,
responses: { 202: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound, 409: r.conflict },
});
registerCurrentRoute({
method: "post",
path: "/api/issues/{id}/recovery-actions/retry-workspace-export",
@@ -6,7 +6,8 @@ import { pendingCleanupMetadataObjectSql } from "./environment-cleanup-state.js"
import { readEnvironmentLeaseController } from "./environment-lease-controller.js";
import { allowLegacyShutdownWorkspaceCleanup } from "./legacy-shutdown-workspace-settlement.js";
import { nativeWorkspaceExportPendingForLease } from "./native-runtime/native-workspace-finalization-state.js";
import type { OwnerProof } from "./owner-proof.js";
import { OPERATOR_ATTESTED_PROCESS_SPACE, type OwnerProof } from "./owner-proof.js";
import { OWNER_GONE_ATTESTATION_KEY } from "./recovery/owner-gone-attestation.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
import { isBuiltinSandboxProvider } from "./sandbox-provider-runtime.js";
import { prepareSandboxStopAndRetain } from "./sandbox-stop-and-retain.js";
@@ -96,6 +97,8 @@ export function holderUnchanged(lease: LeaseRow, record: InProgressRecord): bool
if (record.owner.kind === "run") return holder === null;
if (record.owner.kind !== "process") return false;
const stamp = record.owner.stamp;
// The board attested that the unrecorded holder stopped; still no holder stamped since.
if (stamp.processSpace === OPERATOR_ATTESTED_PROCESS_SPACE) return holder === null && Boolean(lease.metadata?.[OWNER_GONE_ATTESTATION_KEY]);
return holder !== null && holder.bootId === stamp.bootId && holder.pid === stamp.pid
&& holder.hostHash === stamp.processSpace && holder.processStartedAt === (stamp.processStartedAt ?? null);
}
@@ -0,0 +1,72 @@
import { and, eq } from "drizzle-orm";
import { issueRecoveryActions, type Db } from "@paperclipai/db";
import { confirmOwnerGoneSchema } from "@paperclipai/shared";
import { conflict } from "../../errors.js";
import { persistActivity, publishActivity } from "../activity-log.js";
import type { OwnerProof } from "../owner-proof.js";
import type { InProgressRecord, OwnableKind } from "./ownership-registry.js";
/** Recovery action cause: a stuck record whose owner Paperclip cannot judge. */
export const OWNER_UNPROVABLE_CAUSE = "ownership_owner_unprovable";
/** Lease metadata key the board's confirmation writes; the owner proof reads it as gone. */
export const OWNER_GONE_ATTESTATION_KEY = "ownerGoneAttestation";
/** Unprovable because the record names no usable owner: saved before owners were recorded. */
const LEGACY_REASONS = new Set(["no_owner_stamp", "process_space_unrecorded", "missing_identity"]);
const fingerprint = (kind: string, recordId: string) => `ownership-owner:${kind}:${recordId}`;
const changed = () => conflict("This record changed since recovery asked. Refresh the task and inspect its run.",
{ code: "ownership_owner_attestation_stale" });
/** Idempotent. Asks the board on the record's task; a record with no task keeps its other recovery path. */
export function requestOwnerGoneConfirmation(kindId: string) {
return async (db: Db, record: InProgressRecord, context: { now: Date; proof: OwnerProof }): Promise<void> => {
if (!LEGACY_REASONS.has(context.proof.reason) || !record.issueId || !record.companyId) return;
await db.insert(issueRecoveryActions).values({
companyId: record.companyId, sourceIssueId: record.issueId, kind: "active_run_watchdog",
status: "active", ownerType: "board", cause: OWNER_UNPROVABLE_CAUSE,
fingerprint: fingerprint(kindId, record.recordId), attemptCount: 1,
evidence: { kind: kindId, recordId: record.recordId, runId: record.runId, proof: context.proof.reason },
nextAction: "A resource from this task's earlier run was recorded before Paperclip stored which server held it, "
+ "so Paperclip cannot verify that the server stopped. Confirm that the previous server process has stopped "
+ "(for example, its container was replaced), and recovery releases the resource.",
}).onConflictDoNothing();
};
}
/** Board attestation for exactly the record recovery asked about. Recovery then settles it on its normal path. */
export async function confirmOwnerGone(input: {
db: Db; kinds: readonly OwnableKind[]; companyId: string; issueId: string;
actionId: string; recordId: string; stopEvidence: string; actorId: string;
}) {
const confirmation = confirmOwnerGoneSchema.parse({
actionId: input.actionId, recordId: input.recordId, stopEvidence: input.stopEvidence,
});
const committed = await input.db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
eq(issueRecoveryActions.id, input.actionId))).for("update");
const kind = input.kinds.find(candidate => candidate.id === action?.evidence?.kind);
if (!action || !kind?.attestOwnerGone || action.cause !== OWNER_UNPROVABLE_CAUSE || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || action.evidence?.recordId !== input.recordId) throw changed();
const now = new Date();
const proof = { actorId: input.actorId, stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() };
const attestation = await kind.attestOwnerGone(tx, { companyId: input.companyId, recordId: input.recordId }, proof);
if (attestation === "changed") throw changed();
// Released by another path: close the request; nothing remains to attest.
const released = attestation === "released";
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
evidence: { ...action.evidence, ...(released ? { ownerGoneNotNeeded: proof } : { ownerGone: proof }) },
nextAction: released ? "The resource was already released. No confirmation was needed."
: "The previous server was confirmed stopped. Recovery releases the resource.",
}).where(eq(issueRecoveryActions.id, action.id));
const publication = await persistActivity(tx, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
action: "issue.ownership_owner_gone_confirmed", entityType: "issue", entityId: input.issueId,
runId: typeof action.evidence?.runId === "string" ? action.evidence.runId : undefined,
details: { recoveryActionId: input.actionId, kind: kind.id, recordId: input.recordId, outcome: attestation, ...proof } });
return { publication, released };
});
publishActivity(committed.publication.publication);
return { recordId: input.recordId, status: committed.released ? "released" as const : "queued" as const };
}
@@ -28,6 +28,9 @@ import {
NATIVE_OWNERSHIP_UNVERIFIED_ERROR_CODE,
} from "../native-runtime/native-runner-ownership.js";
import type { LocalOwnerStamp, OwnerProof, RemoteFenceState, RemoteOwnerStamp } from "../owner-proof.js";
import { OPERATOR_ATTESTED_PROCESS_SPACE } from "../owner-proof.js";
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
import { OWNER_GONE_ATTESTATION_KEY, requestOwnerGoneConfirmation } from "./owner-gone-attestation.js";
import {
requestNativeAcceptedResultLeaseOperatorAction,
settleNativeAcceptedResultLease,
@@ -148,6 +151,10 @@ export type OwnableKind = {
settle: SettleRecord | null;
/** Optional operator action for an unprovable owner of a settling kind. */
onUnprovable?: UnprovableAction;
/** Records the board's confirmation that this record's unrecorded owner stopped, so its
* owner proves gone (`operator_attested`). Returns false when the record no longer qualifies. */
attestOwnerGone?: (tx: Db, target: { companyId: string; recordId: string }, proof: Record<string, unknown>)
=> Promise<"attested" | "released" | "changed">;
/** When a record of this kind next becomes settleable without a signal
* (a retention window ending). The reconciler wakes then. */
nextDueAt?: (db: Db, now: Date) => Promise<Date | null>;
@@ -320,6 +327,10 @@ function runLeaseOwner(row: RunLeaseRow): RecordOwner {
if (row.runStatus !== null && !TERMINAL_RUN_STATUSES.includes(row.runStatus)) return runOwner(row.heartbeatRunId);
const holder = controllerStamp(json(row.metadata)[ENVIRONMENT_LEASE_CONTROLLER_KEY], "environment_run_lease_active");
if (holder.kind !== "none") return holder;
// The board confirmed that the server holding this unstamped lease stopped.
if (json(row.metadata)[OWNER_GONE_ATTESTATION_KEY]) return { kind: "process", stamp: { kind: "local",
scope: "environment_run_lease_active", generation: null, host: null, processSpace: OPERATOR_ATTESTED_PROCESS_SPACE,
bootId: null, pid: null, processStartedAt: null, processGroupId: null } };
return ephemeralServerEnabled() ? { kind: "none" } : runOwner(row.heartbeatRunId);
}
@@ -336,6 +347,23 @@ function runLeaseKind(): OwnableKind {
return {
id: "environment_run_lease_active", title: "Run lease (active)", obligation: "ownership", storage: "database",
invariantChecks: [1, 2], settle: settleEnvironmentRunLease,
onUnprovable: requestOwnerGoneConfirmation("environment_run_lease_active"),
// Only an unstamped lease of a finished run: a stamped holder is judged by its stamp.
attestOwnerGone: async (tx, target, proof) => {
const attested = (await writeEnvironmentCleanup(tx, write => write.update(environmentLeases)
.set({ metadata: sql`coalesce(${environmentLeases.metadata}, '{}'::jsonb) || ${JSON.stringify({ [OWNER_GONE_ATTESTATION_KEY]: proof })}::jsonb` })
.where(and(eq(environmentLeases.id, target.recordId), eq(environmentLeases.companyId, target.companyId),
eq(environmentLeases.status, "active"), isNotNull(environmentLeases.heartbeatRunId),
sql`not (coalesce(${environmentLeases.metadata}, '{}'::jsonb) ? ${ENVIRONMENT_LEASE_CONTROLLER_KEY})`,
sql`exists (select 1 from heartbeat_runs run where run.id = ${environmentLeases.heartbeatRunId}
and run.status in ('succeeded', 'failed', 'cancelled', 'timed_out', 'interrupted'))`))
.returning({ id: environmentLeases.id }), { companyId: target.companyId, leaseId: target.recordId })).length > 0;
if (attested) return "attested";
// Released by another path since the board was asked: nothing is left to attest.
const [lease] = await tx.select({ status: environmentLeases.status }).from(environmentLeases)
.where(and(eq(environmentLeases.id, target.recordId), eq(environmentLeases.companyId, target.companyId)));
return !lease || lease.status !== "active" ? "released" : "changed";
},
listInProgress: async (db, limit, offset) => {
const rows = await select(db).where(where).orderBy(asc(environmentLeases.id)).limit(limit + 1).offset(offset);
const result = page(rows, limit);
+7
View File
@@ -1,4 +1,5 @@
import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js";
import { OWNER_UNPROVABLE_CAUSE } from "./owner-gone-attestation.js";
import { promoteDeferredWorkIntents } from "../work-intent-outbox.js";
import { isContextCompactionRun } from "../context-compaction.js";
import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js";
@@ -3490,6 +3491,12 @@ export function recoveryService(
issueIds: [] as string[],
};
for (const { action, issue } of rows) {
// Only the board's stop confirmation, or the resource's own release, settles this request:
// the reserved resource stays reserved however the task ends.
if (action.cause === OWNER_UNPROVABLE_CAUSE) {
result.skipped += 1;
continue;
}
if (await hasUnverifiedNativeWorkspaceFinalizationOwner(db, action)) {
result.skipped += 1;
continue;
+2
View File
@@ -422,6 +422,8 @@ export const issuesApi = {
),
confirmControllerStopped: (id: string, data: { actionId: string; runId: string; stopEvidence: string }) =>
api.post<{ runId: string; status: "queued" }>(`/issues/${id}/recovery-actions/confirm-controller-stopped`, data),
confirmOwnerGone: (id: string, data: { actionId: string; recordId: string; stopEvidence: string }) =>
api.post<{ recordId: string; status: "queued" | "released" }>(`/issues/${id}/recovery-actions/confirm-owner-gone`, data),
resumeWorkspaceFinalization: (id: string, data: { actionId: string; runId: string; ownerToken: string; controllerAndCopybackStopped: true; stopEvidence: string }) =>
api.post<{ runId: string; resultId: string | null; status: "queued" }>(`/issues/${id}/recovery-actions/resume-workspace-finalization`, data),
retryWorkspaceExport: (id: string, data: { actionId: string; runId: string; repairNote: string }) =>
@@ -9,9 +9,9 @@ import { queryKeys } from "../lib/queryKeys";
import { writePendingSend } from "../lib/pending-send";
import { WorkspaceExportRecovery } from "./WorkspaceExportRecovery";
const api = vi.hoisted(() => ({ inspect: vi.fn(), recover: vi.fn(), resume: vi.fn(), confirmController: vi.fn() }));
const api = vi.hoisted(() => ({ inspect: vi.fn(), recover: vi.fn(), resume: vi.fn(), confirmController: vi.fn(), confirmOwnerGone: vi.fn() }));
vi.mock("../api/issues", () => ({ issuesApi: { recoveryActions: api.inspect, recoverWorkspace: api.recover,
resumeWorkspaceFinalization: api.resume, confirmControllerStopped: api.confirmController } }));
resumeWorkspaceFinalization: api.resume, confirmControllerStopped: api.confirmController, confirmOwnerGone: api.confirmOwnerGone } }));
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
const action = {
@@ -495,6 +495,30 @@ describe("governed retained-file recovery", () => {
expect(container.textContent).toContain("Resuming the run…");
});
it("confirms the unrecorded server holding a stuck resource, for board owners only", async () => {
api.confirmOwnerGone.mockResolvedValue({ recordId: "lease-1", status: "queued" });
const ownerGoneAction = { ...action, cause: "ownership_owner_unprovable", ownerType: "board",
evidence: { kind: "environment_run_lease_active", recordId: "lease-1", runId: "run-1" } } as unknown as IssueRecoveryAction;
// An operator sees the wait but cannot attest the stop.
await mount(ownerGoneAction);
expect(container.textContent).toContain("before Paperclip stored which server held it");
expect(button("Confirm the previous server stopped")).toBeUndefined();
client.setQueryData(queryKeys.access.currentBoardAccess, { userId: "board-1", source: "session", isInstanceAdmin: false,
companyIds: ["company-1"], memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }] });
await mount(ownerGoneAction);
await click("Confirm the previous server stopped");
const textarea = container.querySelector("textarea")!;
await act(async () => {
const setter = Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!;
setter.call(textarea, "The old server container was replaced and is not running.");
textarea.dispatchEvent(new Event("input", { bubbles: true }));
});
await click("Confirm and resume");
expect(api.confirmOwnerGone).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", recordId: "lease-1",
stopEvidence: "The old server container was replaced and is not running." });
expect(container.textContent).toContain("Releasing the resource…");
});
it("shows status only, without the action, once automatic recovery is resuming or for non-administrators", async () => {
// The default fixture is an operator: it may resolve other recovery but not attest a controller stop.
await mount({ ...ownerAction, evidence: { ...ownerAction.evidence, workspaceOwnerRecovery: { state: "resuming" } } } as IssueRecoveryAction);
+21 -1
View File
@@ -26,11 +26,13 @@ function ReportRecoveryControl({ owns, onChange }: { owns: boolean; onChange?: (
}
const CONTROLLER_HOST_UNVERIFIED = "native_controller_host_unverified";
const OWNER_UNPROVABLE = "ownership_owner_unprovable";
/** Causes whose one action is the board's previous-server stop confirmation;
* the generic recovery card must not offer other resolutions for them. */
export function isPreviousServerStopCause(cause: string | null | undefined) {
return cause === "native_workspace_finalization_owner_unverified" || cause === CONTROLLER_HOST_UNVERIFIED;
return cause === "native_workspace_finalization_owner_unverified" || cause === CONTROLLER_HOST_UNVERIFIED
|| cause === OWNER_UNPROVABLE;
}
/** Requests preservation; the existing worker owns stop, copyback and verification. */
@@ -65,6 +67,18 @@ export function WorkspaceExportRecovery({ action, className, issueId, companyId,
placeholder: "For example: the old server container was replaced and is no longer running.",
resuming: "Resuming the run…" }} /></>;
}
// A resource held by a server recorded before Paperclip stored which server held it.
if (action?.cause === OWNER_UNPROVABLE && issueId && companyId && isOwnerGoneRequired(action, issueId, companyId)) {
return <><ReportRecoveryControl owns onChange={onRecoveryControlChange} />
<PreviousServerStopConfirmation key={`${action.id}:${String(action.evidence.recordId)}`}
action={action} issueId={issueId} companyId={companyId} className={className}
confirmStop={stopEvidence => issuesApi.confirmOwnerGone(issueId, {
actionId: action.id, recordId: action.evidence.recordId as string, stopEvidence })}
copy={{ waiting: "A resource from this task’s earlier run was recorded before Paperclip stored which server held it, so Paperclip can’t verify that the server stopped. The resource stays reserved until then.",
question: "How did you confirm that the previous server stopped?",
placeholder: "For example: the old server container was replaced and is no longer running.",
resuming: "Releasing the resource…" }} /></>;
}
const ownerRecovery = action?.cause === "native_workspace_finalization_owner_unverified";
// The task's interruption names this wait; the confirmation is its one action.
if (ownerRecovery && issueId && companyId && isOwnerStopRequired(action, issueId, companyId)) {
@@ -99,6 +113,12 @@ function isControllerStopRequired(action: IssueRecoveryAction, issueId: string,
&& typeof action.evidence.runId === "string" && !action.evidence.controllerStop;
}
function isOwnerGoneRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)
&& typeof action.evidence.recordId === "string" && !action.evidence.ownerGone;
}
function isOwnerStopRequired(action: IssueRecoveryAction, issueId: string, companyId: string) {
return action.companyId === companyId && action.sourceIssueId === issueId
&& action.ownerType === "board" && ["active", "escalated"].includes(action.status)