Guard Native recovery with process-space proof and explicit containment (#19813)

## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-11-09 12:06:11 -06:00
1 parent 365c771030
commit a3e7e63330
40 files changed
+2674 -147

No files matched your search

+48 -6
View File
@@ -1899,8 +1899,13 @@ A native run can accept its result and then lose its server to a hard kill
before finalization. Its result stays leased to the controller of the earlier
boot, which cannot release the lease the way a graceful drain does. Startup
native recovery releases that lease at once when the lease belongs to a
different boot and its recorded controller process has exited or its PID now
names another process. Only the executor's lease carries that controller
different boot in the same known kernel and PID namespace, and its recorded
controller process has exited or its PID now names another process. The
executor saves that namespace with its claim. Linux uses its boot UUID and PID
namespace; macOS uses its kernel boot-session UUID. An unreadable or unsupported
identity, a different namespace, or an older claim without this proof keeps the
original lease expiry and does not probe the recorded PID. Only the executor's
lease carries that controller
identity; a finalizer or cleanup lease is never released this way and expires
on its own short lease. The stored result is then
finalized without another
@@ -1908,22 +1913,59 @@ provider turn, and saved follow-up messages are reconsidered. A live owner, an
owner without a recorded process, a lease of the current boot, and a remote
workspace keep the lease until it expires. Lease age alone is never proof.
Before claiming an in-flight local native run, restart recovery also requires
this same known process space. Expiry alone does not prove a local controller
stopped. A missing, foreign or unreadable namespace creates one board-owned
recovery action for the exact saved authority, without probing the recorded PID,
spending retry attempts, or arming a polling loop. Renewed authority gets a new
opaque fingerprint and preserves the previous action. Remote and plugin tasks
retain their ordinary Stop/provider-cleanup path and do not offer the local
containment confirmation.
When automatic recovery cannot prove a local owner stopped, an operator can
record containment through the task's recovery action after the original lease
expires. The operator separately confirms the controller and provider/tools
stopped on their original host, and records reviewed action outcomes and evidence.
The server rechecks board `runtime:confirm_owner_stop` permission for a company
owner, company administrator or instance administrator, plus company, assignee, source binding,
opaque authority, exact expiry (including microseconds), pending Stop, accepted
results and cleanup obligations under the normal ownership locks. A renewed or
replaced owner invalidates the confirmation. This is explicit operator evidence,
not a local process-stop observation.
Containment uses normal failed-owner settlement and preserves usage uncertainty,
workspaces and outcome evidence. Containment records the canonical `restored`
outcome while keeping the task status unchanged, including when it is already
`todo`. A separate ordinary Retry then consumes that committed evidence. It does not accept a provider result or queue a
replacement. Only exact unadmitted Retry receipts for the contained source are
closed with an inspectable reason; accepted or newer work stays intact. After
cleanup and outcome review, the user makes a new ordinary Retry request under
current authorization. Unknown outcomes keep the hold; reviewed completed or
mixed effects remain in the continuation context and never authorize automatic
replay. A later successful ordinary recovery claim resolves only its exact prior
namespace attention.
A recorded process is judged only in the PID space that recorded it. Native
controller rows store `controller_host_hash` and run processes store
`process_host_hash` (`"remote"` for a sandbox process). A dead PID on this host
never proves that a process on another host, in another PID namespace, or in a
sandbox is gone; such a record keeps its lease, fence or staleness TTL.
Rows saved before those columns existed have no recorded host. For one release
only, a single-host server keeps the previous local proof for them, and each use
Rows saved before those columns existed have no recorded host. The legacy
compatibility helper and run-process sweep retain one-release single-host local
proof where their existing owners permit it. Each use
writes a `legacy_host_unknown_dead_pid_proof` activity-log entry, so the release that
removes this fallback can confirm that no rows still depend on it. An ephemeral
server (`PAPERCLIP_EPHEMERAL_SERVER=true`) never uses the fallback. Its native
controller rows with no recorded host wait as `controller_host_unrecorded`, with
a board recovery action to confirm that the previous server stopped. Its run
processes with no recorded host are left to the run's Stop control. The next
release removes the fallback and treats every row with no recorded host as
unprovable.
release removes that compatibility fallback. The strict Native in-flight claim,
early accepted-result release, and failed-owner guards above do not use it.
A previous-server-only confirmation is not same-namespace proof and does not
confirm that a provider or its tools stopped. It cannot override those guards.
Its notification requests recovery inspection; unfinished ownership can still
require the separate containment and ordinary Retry described above.
A server boot that holds work keeps a controller instance lease
(`controller_instances`): a row named by its boot id, with an epoch and a lease
+11
View File
@@ -100,6 +100,17 @@ Gaps: Component host test doesn't prove actual filesystem/provider repair.
## `bounded-continuation`
Original-host Native controller containment requires board-only
`runtime:confirm_owner_stop` authority on the existing recovery resolution route:
a company owner, company administrator, or instance administrator. Agents cannot submit it.
It requires separate controller/provider stop confirmations, reviewed outcomes,
a server-bound exact authority digest, elapsed lease and unchanged locked
source/assignee/cleanup fences. It records containment without queuing work; a
separate ordinary task Retry is required. Unknown process space never authorizes
a local PID probe or automatic physical takeover, even after expiry.
Evidence: `source-only` — candidate authority, route, UI and real-owner controls;
no deployed/original-host stop verification is claimed.
Capability ID: `recovery/bounded-continuation`.
Ordinary task Retry uses `POST /api/issues/{id}/retry`. This is authenticated
+8
View File
@@ -64,6 +64,14 @@ and the [disconnect recovery journey](../../tests/e2e/disconnect-write-recovery.
### `source-task`
A Native run held for unknown controller process space remains blocked even
after its lease expires. A company owner, company administrator, or instance
administrator can record separate original-host controller and provider/tool stop confirmations with reviewed
action outcomes through the existing task recovery card. Recording containment
does not replay work. After normal settlement/cleanup, use a separate ordinary
Retry; stale authority or unresolved cleanup must keep it blocked. This candidate
path is source-only pending its focused real-owner qualification.
Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link.
Observe one working, backoff, failed or waiting-for-user state. Use Retry after
an infrastructure failure. No process or file certification is required.
@@ -85,6 +85,9 @@ export function requiresExecutionReconciliation(
export interface ExecutionReconciliation {
runId: string;
providerStopped: true;
/** Only the exact namespace-unverified Native ownership hold accepts these. */
controllerStopped?: true;
controllerAuthorityDigest?: string;
actionOutcome: "completed" | "not_performed" | "mixed";
outcomeEvidence: string;
/** Operator evidence bound to this failed run; required after workspace restore failure. */
@@ -369,6 +369,37 @@ describe("issue validators", () => {
} }).success).toBe(true);
});
it.each(["backlog", "in_progress", "blocked"] as const)("allows a complete controller containment request to retain %s status for locked route validation", sourceIssueStatus => {
const input = { actionId: "22222222-2222-4222-8222-222222222222", outcome: "restored", sourceIssueStatus,
executionReconciliation: { runId: "11111111-1111-4111-8111-111111111111", providerStopped: true,
controllerStopped: true, controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed",
outcomeEvidence: "The original-host controller and provider stopped; action outcomes were reviewed." } };
// The real historical HTTP request failed here before its route could
// enforce same-status/current-authority checks. Syntax grants no authority.
expect(resolveIssueRecoveryActionSchema.parse(input)).toEqual(input);
});
it.each(["controller-only", "digest-only", "missing-action", "wrong-outcome", "partial-with-workspace-evidence"] as const)(
"rejects incomplete controller containment even when ordinary status validation would pass: %s", mode => {
const complete = { runId: "11111111-1111-4111-8111-111111111111", providerStopped: true,
controllerStopped: true, controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed",
outcomeEvidence: "The original-host controller and provider stopped; action outcomes were reviewed." };
const ordinary = { runId: complete.runId, providerStopped: complete.providerStopped,
actionOutcome: complete.actionOutcome, outcomeEvidence: complete.outcomeEvidence };
const input = { actionId: mode === "missing-action" ? undefined : "22222222-2222-4222-8222-222222222222",
outcome: mode === "wrong-outcome" ? "blocked" : "restored",
sourceIssueStatus: mode === "wrong-outcome" ? "blocked" : "todo",
executionReconciliation: mode === "controller-only" ? { ...ordinary, controllerStopped: true }
: mode === "digest-only" ? { ...ordinary, controllerAuthorityDigest: complete.controllerAuthorityDigest }
: mode === "partial-with-workspace-evidence" ? { ...ordinary, controllerStopped: true,
workspaceRepairEvidence: "Retained workspace files were repaired and inspected separately." }
: complete };
const result = resolveIssueRecoveryActionSchema.safeParse(input);
expect(result.success).toBe(false);
if (!result.success) expect(result.error.issues).toContainEqual(expect.objectContaining({ path: ["executionReconciliation"] }));
},
);
it("allows cancelled recovery resolutions to atomically restore the source issue status", () => {
expect(
resolveIssueRecoveryActionSchema.parse({
+15 -3
View File
@@ -652,6 +652,8 @@ export const resolveIssueRecoveryActionSchema = z
.object({
runId: z.string().guid(),
providerStopped: z.literal(true),
controllerStopped: z.literal(true).optional(),
controllerAuthorityDigest: z.string().regex(/^[0-9a-f]{64}$/).optional(),
actionOutcome: z.enum(["completed", "not_performed", "mixed"]),
outcomeEvidence: z.string().trim().min(20).max(12000),
workspaceRepairEvidence: z.string().trim().min(20).max(12000).optional(),
@@ -668,10 +670,20 @@ export const resolveIssueRecoveryActionSchema = z
if (value.workspaceBaseRef && (!value.actionId || value.outcome !== "restored" || value.sourceIssueStatus !== "todo" || value.executionReconciliation)) {
ctx.addIssue({ code: z.ZodIssueCode.custom, path: ["workspaceBaseRef"], message: "Branch repair must retry the exact recovery action" });
}
const reconciliation = value.executionReconciliation;
const controllerContainment = reconciliation?.controllerStopped !== undefined
|| reconciliation?.controllerAuthorityDigest !== undefined;
if (controllerContainment && (!value.actionId || value.outcome !== "restored"
|| reconciliation?.controllerStopped !== true || !reconciliation?.controllerAuthorityDigest)) {
ctx.addIssue({ code: z.ZodIssueCode.custom, path: ["executionReconciliation"],
message: "Controller containment requires the exact recovery action, both stop confirmations, and its authority digest" });
return;
}
if (value.outcome === "restored") {
// A retained-source repair records evidence without changing task state.
// The route verifies the exact server-owned source and unchanged status.
if (value.executionReconciliation?.workspaceRepairEvidence) return;
// Source repair and complete controller containment record evidence
// without changing task state. The route checks the exact locked source,
// unchanged status, board authority, lease expiry, and cleanup evidence.
if (value.executionReconciliation?.workspaceRepairEvidence || controllerContainment) return;
if (
value.sourceIssueStatus !== "todo" &&
value.sourceIssueStatus !== "done" &&
@@ -1,3 +1,4 @@
import { releaseAbandonedNativeOwners } from "../services/native-runtime/native-failed-run-owner-release.js";
import { guardSlackFixtureNetwork } from "./helpers/slack-cloud/transport.js";
import { chatHistoryActionBlocks } from "./helpers/chat-history-proof.js";
import type { NativeExecutionInput, NativeSession, NativeSessionBackend, PrpEvent } from "../vendor/paperclip-runner/index.js";
@@ -67914,6 +67915,250 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
},
);
it.each(["blocked_owner_queued", "todo_owner_useful", "todo_admin_useful", "todo_operator_denied"] as const)(
"admits a separate connected-conversation Retry only after committed original-controller containment: %s", async mode => {
const f = await failedChatRetryFixture("telegram", undefined, true);
const companyId = f.fixture.companyId, issueId = f.issue.id, agentId = f.fixture.assignedAgentId;
let heartbeat: ReturnType<typeof heartbeatService> | undefined;
const useful = mode.endsWith("useful"), denied = mode === "todo_operator_denied";
const role = mode === "todo_admin_useful" ? "admin" : denied ? "operator" : "owner";
// Initial task state and real membership: no status/receipt rewrite occurs
// during containment or the new Retry journey.
await db.update(companyMemberships).set({ membershipRole: role }).where(and(
eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, "owner-user")));
if (mode.startsWith("todo")) await db.update(issues).set({ status: "todo" }).where(eq(issues.id, issueId));
const settings = instanceSettingsService(db), previousSettings = await settings.getExperimental();
const workingDirectory = useful ? mkdtempSync(path.join(os.tmpdir(), "chat-contained-todo-")) : null;
let providerTurns = 0;
const answer = "The separately retried todo request completed once after original-host containment.";
// Historical initial facts: the original connected request was admitted,
// failed after a recorded launch on a foreign host, and retained an expired
// result-less owner. No stop/result or successful provider outcome is seeded.
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, nativePhase: "observed",
processPid: 2_000_000_001, processGroupId: 2_000_000_001, processStartedAt: new Date(0),
runnerProfileJson: {} }).where(eq(heartbeatRuns.id, f.runId));
await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: f.runId, phase: "observed",
leaseOwner: "native-exec:historical-connected-source", leaseExpiresAt: new Date(Date.now() - 1_000),
controllerBootId: randomUUID(), controllerHostHash: "historical-foreign-process-space",
controllerPid: 2_000_000_001, controllerProcessStartedAt: new Date(0), controllerGeneration: 1, attempt: 1 });
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId,
eventType: "native.process_identity_recorded", payload: { targetKind: "local", hostname: "original-foreign-host",
processPid: 2_000_000_001, processGroupId: 2_000_000_001, processStartedAt: new Date(0).toISOString() } });
try {
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
const [action] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
eq(issueRecoveryActions.cause, "native_restart_recovery_blocked"), eq(issueRecoveryActions.status, "active")));
expect(action).toBeDefined();
const authority = action.evidence.controllerAuthority;
if (!authority || typeof authority !== "object" || Array.isArray(authority)
|| !("digest" in authority) || typeof authority.digest !== "string") throw new Error("Normal exact controller attention required");
const app = express(); app.use(express.json());
app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); });
app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); app.use(errorHandler);
const [task] = await readIssuesById(issueId);
const evidence = "Verified original controller and provider stopped on the original host; retained completed external actions.";
expect(task.status).toBe(mode.startsWith("todo") ? "todo" : "blocked");
const beforeRuns = await readHeartbeatRunsByCompanyId(companyId);
const beforeWakes = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
const beforeOwner = (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0];
const containment = await request(app).post(`/api/issues/${issueId}/recovery-actions/resolve`).send({ actionId: action.id,
outcome: "restored", sourceIssueStatus: task.status, executionReconciliation: { runId: f.runId,
controllerStopped: true, providerStopped: true, controllerAuthorityDigest: authority.digest,
actionOutcome: "mixed", outcomeEvidence: evidence } });
if (denied) {
expect(containment.status).toBe(403);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(beforeOwner);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]).toEqual(action);
expect(await readHeartbeatRunsByCompanyId(companyId)).toEqual(beforeRuns);
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toEqual(beforeWakes);
// A deliberate click may persist an intent; only the REAL normal
// queue may decide admission. The transport fixture's receipt stub is
// not sufficient evidence that an uncontained source cannot execute.
const unexpectedBackend = vi.fn((): NativeSessionBackend => {
throw new Error("An uncontained foreign owner must never create a new provider backend");
});
await settings.updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true });
heartbeat = heartbeatService(db, { nativeSessionBackendFactory: unexpectedBackend });
f.wakeup.mockImplementation(heartbeat.wakeup);
const idempotencyKey = randomUUID();
const retryInput = { observedRunId: f.runId, idempotencyKey };
const click = await request(app).post(`/api/issues/${issueId}/retry`).send(retryInput);
expect(click.status).toBe(202);
expect(click.body.requestId).toEqual(expect.any(String));
expect(click.body.chatActionId).toEqual(expect.any(String));
const deniedRetry = await f.service.processFailedChatRunRetry(click.body.chatActionId);
expect(deniedRetry).toMatchObject({ status: "failed", runId: null });
const [deniedAction] = await db.select().from(chatActions).where(eq(chatActions.id, click.body.chatActionId));
expect(deniedAction).toMatchObject({ status: "failed", result: { code: "failed_chat_retry_not_authorized" } });
await reconcileTaskChatRetryReceipts(db, { companyId, requestId: click.body.requestId });
const [intent] = await readAgentWakeupRequestsById(click.body.requestId);
expect(intent).toMatchObject({ status: "failed", runId: null, payload: { taskRetry: {
sourceRunId: f.runId, actorId: "owner-user", actorSource: "session", chatActionId: click.body.chatActionId,
} } });
// A lost HTTP response may be retried with the same key. It replays
// this exact denied intent, never creates another execution authority.
const replay = await request(app).post(`/api/issues/${issueId}/retry`).send(retryInput);
expect(replay.status).toBe(202);
expect(replay.body).toMatchObject({ requestId: click.body.requestId, chatActionId: click.body.chatActionId });
expect(await f.service.processFailedChatRunRetry(replay.body.chatActionId)).toMatchObject({ status: "failed", runId: null });
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
eq(agentWakeupRequests.idempotencyKey, `task-retry:${idempotencyKey}`)))).toHaveLength(1);
await heartbeat.resumeTaskRetry(companyId, click.body.requestId);
expect(unexpectedBackend).not.toHaveBeenCalled();
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(beforeOwner);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]).toEqual(action);
expect(await readHeartbeatRunsByCompanyId(companyId)).toEqual(beforeRuns);
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(0);
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
return;
}
expect(containment.status).toBe(200);
const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
expect(settled).toMatchObject({ phase: "terminal_failure", leaseOwner: null, resultId: null,
failureCode: "native_restart_recovery_blocked", failureDetail: { releasedOwner: { reason: "operator_containment" } } });
const [resolved] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
expect(resolved).toMatchObject({ status: "resolved", evidence: { continuationDelivery: "not_requested" } });
if (!useful) expect(resolved.outcome).toBe("restored");
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
expect(await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).toHaveLength(0);
const direct = { companyId, issueId, agentId, failedRunId: f.runId, initiatedByUserId: "owner-user" };
// These are corrupt/stale control receipts, not provider outcomes. Restore
// their exact committed beforeimages before the real new Retry below.
for (const invalid of ["wrong_digest", "changed_generation", "not_resolved"] as const) {
try {
if (invalid === "wrong_digest") await db.update(issueRecoveryActions).set({ evidence: {
...resolved.evidence, controllerAuthority: { schema: "paperclip.native-controller-authority.v1", digest: "0".repeat(64) },
} }).where(eq(issueRecoveryActions.id, action.id));
if (invalid === "changed_generation") await db.update(nativeRunFinalizations)
.set({ controllerGeneration: settled.controllerGeneration + 1 }).where(eq(nativeRunFinalizations.runId, f.runId));
if (invalid === "not_resolved") await db.update(issueRecoveryActions).set({ status: "cancelled" })
.where(eq(issueRecoveryActions.id, action.id));
await expect(db.transaction(tx => f.service.prepareFailedChatRunRetry(tx, direct)))
.rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } });
} finally {
await db.update(issueRecoveryActions).set({ evidence: resolved.evidence, status: resolved.status })
.where(eq(issueRecoveryActions.id, action.id));
await db.update(nativeRunFinalizations).set({ controllerGeneration: settled.controllerGeneration })
.where(eq(nativeRunFinalizations.runId, f.runId));
}
}
// Existing real-queue pattern: another task occupies capacity, so this
// assertion proves admission and exact provenance, not provider execution.
const backend = (execution: NativeExecutionInput): NativeSessionBackend => {
const turnId = randomUUID(), evidenceId = randomUUID(), evidenceRef = `work_product:${evidenceId}`;
const identity = { companyId, agentId, issueId, runId: execution.binding.runId,
sessionId: execution.session.normalizedSessionId! };
const result = structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT);
result.summary = answer;
result.completionClaim.contractRevision = execution.completionContract.contract.revision;
result.completionClaim.criteria = execution.completionContract.contract.criteria.map(criterion => ({
criterionId: criterion.id, status: "satisfied", evidenceRefs: [evidenceRef] }));
result.evidence = [{ kind: "work_product", ref: evidenceRef }]; result.verification[0]!.artifactRef = evidenceRef;
let started = () => {}; const turnStarted = new Promise<void>(resolve => { started = resolve; });
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
const session: NativeSession = {
identity: () => identity, async capabilities() { return capabilities; },
async *events() {
await turnStarted; const [run] = await readHeartbeatRunsById(identity.runId);
yield { schema: "paperclip.prp.event.v1", sourceEventId: `${identity.runId}:terminal`, sourceSeq: 1,
sourceInstanceId: run.runnerInstanceId!, sourceKind: "runner", runId: identity.runId,
normalizedSessionId: identity.sessionId, turnId, eventType: "turn.completed", schemaVersion: 1,
priority: 0, emittedAt: new Date().toISOString(), payload: {} } satisfies PrpEvent;
},
async startTurn() { providerTurns += 1; started(); return { turnId }; },
async result() { await turnStarted; return { result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }; },
async snapshot() { return { backendKind: "mock", sessionId: "contained-todo-fixture", identity,
providerSessionId: `provider-${identity.runId}`, activeTurnId: null, pendingRuntimeRequests: [], lineage: [], cursor: "1" }; },
async close() {},
};
return {
async descriptor() { return { kind: "mock", name: "contained-todo-retry", version: "1", capabilities,
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; },
async openSession() {
await db.insert(issueWorkProducts).values({ id: evidenceId, companyId, issueId,
type: "artifact", provider: "paperclip", title: "Contained todo request result", status: "ready_for_review", reviewState: "approved" });
return session;
},
async recoverSession() { throw new Error("Containment Retry must create a fresh authorized session, never replay the old provider"); },
};
};
if (useful) await settings.updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true });
await db.update(agents).set({ adapterType: "paperclip_runner", status: useful ? "idle" : "running",
...(useful ? { adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017",
cwd: workingDirectory, workspaceStrategy: { type: "project_primary" } } } : {}),
runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
if (!useful) await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", startedAt: new Date() });
heartbeat = heartbeatService(db, useful ? { nativeSessionBackendFactory: backend } : {});
f.wakeup.mockImplementation(heartbeat.wakeup);
const classifyDenial = (value: unknown) => value === "chat_failed_run_retry_not_authorized"
|| value === "failed_chat_retry_not_authorized" ? value : "withheld_unknown";
const click = await request(app).post(`/api/issues/${issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(click.status, JSON.stringify({ phase: "ordinary_retry_response", status: click.status,
code: classifyDenial(click.body?.code ?? click.body?.details?.code) })).toBe(202);
expect(resolved.outcome).toBe("restored");
expect(click.body.chatActionId).toBeTruthy();
const outcome = await f.service.processFailedChatRunRetry(click.body.chatActionId);
const [retryAction] = await db.select().from(chatActions).where(eq(chatActions.id, click.body.chatActionId));
const outcomeDiagnostic = JSON.stringify({ phase: "durable_chat_retry", status: outcome.status,
code: classifyDenial(retryAction?.result?.code) });
if (useful) expect(["queued", "running", "succeeded"], outcomeDiagnostic).toContain(outcome.status);
else expect(outcome.status, outcomeDiagnostic).toBe("queued");
const [wake] = await readAgentWakeupRequestsById(click.body.chatActionId);
expect(wake).toMatchObject({ ...(!useful ? { status: "queued" } : {}), runId: expect.any(String) });
const [successor] = await readHeartbeatRunsById(wake.runId!);
expect(successor).toMatchObject({ ...(!useful ? { status: "queued" } : {}), retryOfRunId: f.runId, contextSnapshot: {
source: "chat:telegram", wakeCommentIds: [f.action.payload.commentId], taskRetryRequestId: click.body.requestId,
} });
expect(successor.contextSnapshot?.forceFreshSession).toBe(true);
// Use the same canonical projection as heartbeat/task-context before
// execution. The chat payload carries a source reference, never a copy
// of operator authority or a fabricated provider outcome.
const { buildExecutionContinuation } = await import("../services/execution-continuation.js");
const continuation = await buildExecutionContinuation({ db, companyId, issueId, agentId,
runId: successor.id, context: successor.contextSnapshot ?? {}, summary: null, exposeLowTrustRaw: false });
expect(continuation.trigger.sourceRunId).toBe(f.runId);
expect(continuation.recoveryOutcomes).toContainEqual({ recoveryActionId: action.id,
decision: expect.objectContaining({ runId: f.runId, controllerStopped: true, providerStopped: true,
actionOutcome: "mixed", outcomeEvidence: evidence }) });
await f.service.processFailedChatRunRetry(click.body.chatActionId);
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
.toMatchObject({ resultId: null, failureDetail: { releasedOwner: { actionOutcome: "mixed", outcomeEvidence: evidence } } });
if (useful) {
await heartbeat.resumeQueuedRuns();
await vi.waitFor(async () => {
const [completed] = await readHeartbeatRunsById(successor.id);
expect(completed.status, JSON.stringify({ phase: "useful_successor", status: completed.status,
errorCode: completed.errorCode, nativePhase: completed.nativePhase })).toBe("succeeded");
expect(completed.resultJson?.nativeResult).toMatchObject({ summary: answer });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, successor.id)))[0].phase).toBe("committed");
expect(providerTurns).toBe(1);
await f.service.processPendingPublications(); await f.service.processPendingPublications();
expect((await readChatPublicationsByEndpointId(f.endpoint.id)).filter(publication => publication.state === "published"
&& typeof publication.payload.text === "string" && publication.payload.text.includes(answer))).toHaveLength(1);
}, { timeout: 10_000, interval: 25 });
const [completed] = await readHeartbeatRunsById(successor.id);
expect(completed.resultJson?.nativeResult).toMatchObject({ summary: answer });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, successor.id)))[0].phase).toBe("committed");
expect(providerTurns).toBe(1);
expect((await readChatPublicationsByEndpointId(f.endpoint.id)).filter(publication => publication.state === "published"
&& typeof publication.payload.text === "string" && publication.payload.text.includes(answer))).toHaveLength(1);
await f.service.processFailedChatRunRetry(click.body.chatActionId);
await heartbeat.resumeTaskRetry(companyId, click.body.requestId);
expect(providerTurns).toBe(1);
}
} finally {
await f.service.shutdown(); if (heartbeat) await heartbeat.drainActiveRunExecutions();
if (useful || denied) await settings.updateExperimental(previousSettings);
if (workingDirectory) rmSync(workingDirectory, { recursive: true, force: true });
}
});
it.each(["stopped", "real_queue", "unknown_executor", "source_edited", "cancelled_stopped", "cancelled_unknown"] as const)(
"admits ordinary chat task Retry with a missing checkpoint only after containment: %s", async kind => {
const context = await failedChatRetryFixture("telegram", undefined, kind === "real_queue");
File diff suppressed because it is too large. Load diff
@@ -1,3 +1,4 @@
import { currentControllerProcess } from "../services/controller-process-identity.js";
import { execFile } from "node:child_process";
import { randomUUID } from "node:crypto";
import { cp, mkdir, readFile, realpath, writeFile } from "node:fs/promises";
@@ -313,8 +314,15 @@ describeEmbeddedPostgres("stopped observed native workspace preservation", () =>
it.each(["before_pin", "after_pin"] as const)("serializes the real ephemeral checkpoint claim %s", async order => {
const f = await seedSource(`CLAIM-${order}`);
if (order === "before_pin") await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
else expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true);
if (order === "before_pin") {
// This branch exercises a same-host checkpoint claim, not a legacy or
// foreign namespace. Keep every other historical preservation fixture unchanged.
const controller = await currentControllerProcess();
expect(controller.hostHash).not.toBe("unknown");
await f.db.update(nativeRunFinalizations).set({ controllerHostHash: controller.hostHash })
.where(eq(nativeRunFinalizations.runId, f.runId));
await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
} else expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true);
const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], controller: fixture.current.successor });
if (order === "before_pin") {
expect(dispositions).toMatchObject([{ kind: "reattach_remote_runner", runId: f.runId }]);
@@ -157,8 +157,10 @@ export function nativeRestartFixture(options: { dedicatedDatabaseUrl?: string }
async function seedRun(
label: string,
existingDb?: ReturnType<typeof createDb>,
seedOptions: { agentId?: string } = {},
) {
const db = existingDb ?? createDb(temporary.connectionString);
const runAgentId = seedOptions.agentId ?? agentId;
const issueId = randomUUID();
const runId = randomUUID();
await db.insert(issues).values({
@@ -169,14 +171,14 @@ export function nativeRestartFixture(options: { dedicatedDatabaseUrl?: string }
status: "in_progress",
priority: "medium",
workMode: "standard",
assigneeAgentId: agentId,
assigneeAgentId: runAgentId,
});
const [run] = await db
.insert(heartbeatRuns)
.values({
id: runId,
companyId,
agentId,
agentId: runAgentId,
status: "running",
runtimeMode: "native",
nativeIssueId: issueId,
@@ -3747,4 +3747,20 @@ describe("agent issue mutation checkout ownership", () => {
expect(mockIssueService.update).not.toHaveBeenCalled();
});
});
it.each(["owner-agent", "board-without-runtime-access"] as const)("denies original-host containment for %s before settlement", async actorKind => {
mockIssueService.getById.mockResolvedValue(makeIssue({ status: "in_progress", assigneeAgentId: ownerAgentId }));
const actor = actorKind === "owner-agent" ? ownerActor()
: { type: "board", userId: "board-user", source: "session", companyIds: [companyId] };
const response = await request(await createApp(actor)).post(`/api/issues/${issueId}/recovery-actions/resolve`).send({
actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "in_progress",
executionReconciliation: { runId: ownerRunId, controllerStopped: true, providerStopped: true,
controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed",
outcomeEvidence: "Original-host containment and external action outcomes were reviewed." },
});
expect(response.status, JSON.stringify(response.body)).toBe(403);
if (actorKind === "owner-agent") expect(response.body.error).toBe("Board access required");
expect(mockIssueRecoveryActionService.resolveActiveForIssue).not.toHaveBeenCalled();
expect(mockIssueService.update).not.toHaveBeenCalled();
});
});
@@ -207,7 +207,7 @@ describe("report-only ownership reconciliation", () => {
});
});
it("proves a foreign-boot coordinator gone when its PID-space stamp matches and its PID is dead", async () => {
it("proves a recorded same-space Native controller gone while keeping a legacy NULL controller unprovable", async () => {
const issueId = randomUUID(); const runId = randomUUID(); const legacyRunId = randomUUID(); const legacyIssueId = randomUUID();
await db.insert(issues).values([{ id: issueId, companyId, title: "Dead owner" }, { id: legacyIssueId, companyId, title: "Legacy" }]);
await db.insert(heartbeatRuns).values([
@@ -220,13 +220,11 @@ describe("report-only ownership reconciliation", () => {
controllerProcessStartedAt: new Date("2026-11-07T09:00:00.000Z"), controllerGeneration: 1 };
await db.insert(nativeRunFinalizations).values([
{ ...claim, runId, issueId, controllerHostHash: currentProcessSpaceHash() },
// Written before the PID space was stamped: the one-release local proof
// on a single-host server (#19883), never on an ephemeral one.
{ ...claim, runId: legacyRunId, issueId: legacyIssueId, controllerHostHash: null },
]);
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
expect(kindReport(report, "native_executor_claim")).toMatchObject({ inProgress: 2,
verdicts: { gone: 2, unprovable: 0 }, reasons: { pid_dead: 2 } });
verdicts: { gone: 1, unprovable: 1 }, reasons: { pid_dead: 1, process_space_unrecorded: 1 } });
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
try {
const ephemeral = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
+29 -3
View File
@@ -10297,6 +10297,12 @@ export function issueRoutes(
const decision = await access.decide({ actor: req.actor, action: "runtime:manage", resource: { type: "company", companyId: existing.companyId } });
if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision));
}
if (executionReconciliation?.controllerStopped !== undefined || executionReconciliation?.controllerAuthorityDigest !== undefined) {
assertBoard(req);
const permission = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop",
resource: { type: "company", companyId: existing.companyId } });
if (!permission.allowed) throw forbidden(permission.explanation, authorizationDeniedDetails(permission));
}
const actor = getActorInfo(req);
const actionStatus = outcome === "cancelled" ? "cancelled" : "resolved";
const postCommitActivityPublications: ActivityPublication[] = [];
@@ -10419,6 +10425,11 @@ export function issueRoutes(
{ source: "recovery_action_resolution" },
);
if (activeRecoveryAction.kind === "active_run_watchdog" && activeRecoveryAction.ownerType === "board"
&& activeRecoveryAction.cause === "native_restart_recovery_blocked"
&& activeRecoveryAction.evidence.reason === "controller_process_space_unverified"
&& (!executionReconciliation?.controllerStopped || !executionReconciliation.controllerAuthorityDigest))
throw conflict("Confirm the original controller and provider stopped and review their action outcomes before resolving this hold.");
const workspaceRetry = outcome === "restored" && sourceIssueStatus === "todo"
? await prepareRestrictedWorkspaceRetry(tx as unknown as Db, lockedIssue, activeRecoveryAction)
: null;
@@ -10548,7 +10559,21 @@ export function issueRoutes(
}
}
if (
const controllerContainment = executionReconciliation?.controllerStopped === true
|| executionReconciliation?.controllerAuthorityDigest !== undefined;
if (controllerContainment) {
assertBoard(req);
const permission = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop",
resource: { type: "company", companyId: lockedIssue.companyId } });
if (!permission.allowed) throw forbidden(permission.explanation, authorizationDeniedDetails(permission));
if (outcome !== "restored" || sourceIssueStatus !== lockedIssue.status || !actionId
|| activeRecoveryAction.id !== actionId || workspaceBaseRef || workspaceRetry)
throw conflict("Containment only records the original execution. Keep the task unchanged, then use Retry separately.");
await validateExecutionReconciliation({ db: tx as unknown as Db, companyId: lockedIssue.companyId,
issueId: lockedIssue.id, agentId: lockedIssue.assigneeAgentId,
sourceRunId: activeRecoveryAction.evidence.runId, decision: executionReconciliation,
controllerContainment: { actionId, actorId: actor.actorId } });
} else if (
sourceIssueStatus === "todo" &&
requiresExecutionReconciliation(activeRecoveryAction.cause)
) {
@@ -10570,7 +10595,7 @@ export function issueRoutes(
}
let chatRetry: { actionId: string; issueId: string } | null = null;
if (outcome === "restored" && sourceIssueStatus === "todo") {
if (!controllerContainment && outcome === "restored" && sourceIssueStatus === "todo") {
const [chatBinding] = await tx
.select({ id: chatConversations.id })
.from(chatConversations)
@@ -10625,6 +10650,7 @@ export function issueRoutes(
chatRetry
? { kind: "chat_failed_run_retry", actionId: chatRetry.actionId }
: undefined,
controllerContainment ? { controllerContainmentOnly: true } : undefined,
);
}
let issue = lockedIssue;
@@ -10780,7 +10806,7 @@ export function issueRoutes(
issue = updatedIssue;
}
const recordedOutcome =
const recordedOutcome = controllerContainment ? "restored" :
outcome === "restored" &&
issue.status === "todo" &&
activeRecoveryAction.returnOwnerAgentId != null &&
@@ -1,3 +1,4 @@
import { hasCommittedNativeControllerContainment } from "../native-runtime/native-failed-run-owner-release.js";
import { notifyStrandedAgentStatus } from "../live-run-recovery-notifications.js";
import { notifyDeferredExecutionWork, notifyTaskRetrySettlement } from "../deferred-execution-notifications.js";
import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js";
@@ -382,6 +383,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
),
)
.for("share", { noWait: true });
if (coordinator && await hasCommittedNativeControllerContainment(tx, run, issueId)) {
return { coordinator, leases: [], retiredOwner: null, operatorContained: true };
}
// A queued request can fail before its first provider event because the
// predecessor's exact retained checkpoint has not yet been suspended.
// A later authenticated cleanup receipt can authorize retrying that
@@ -690,8 +694,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
if (run.runtimeMode !== "native") throw failedChatRetryDenied();
}
if (run.runtimeMode !== "native") return;
const { coordinator, leases, retiredOwner } =
await failedNativeRetryCoordinator(tx, run, source.issueId);
const state = await failedNativeRetryCoordinator(tx, run, source.issueId);
if ("operatorContained" in state && state.operatorContained) return;
const { coordinator, leases, retiredOwner } = state;
if (retiredOwner) {
if (
!nativePreProviderRetryAfterCleanupStateIsSafe({
@@ -1,3 +1,4 @@
import { hasCommittedNativeControllerContainment } from "../native-runtime/native-failed-run-owner-release.js";
import { and, eq, sql } from "drizzle-orm";
import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, issues, type Db } from "@paperclipai/db";
import { buildExecutionContinuation } from "../execution-continuation.js";
@@ -74,12 +75,11 @@ async function taskChatRetryExecutionContained(db: Db, run: typeof heartbeatRuns
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
)).for("share", { noWait: true });
// A verified stop counts, including one whose destroy is queued as stopped
// cleanup (a recovered or retained orphan sandbox awaiting teardown).
if (leases.some(lease => lease.provider !== "local")) {
return leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease));
}
return await recordedLocalExecutionStopped(db, run) ||
return await hasCommittedNativeControllerContainment(db, run, run.nativeIssueId ?? "") ||
await recordedLocalExecutionStopped(db, run) ||
await hasNativeLocalProcessStop(db, run.companyId, run.id) ||
await hasHistoricalSuspendedNativeSession(db, run);
}
@@ -1,6 +1,7 @@
import { createHash, randomUUID } from "node:crypto";
import { execFileSync } from "node:child_process";
import { readFileSync, readlinkSync } from "node:fs";
import { hostname, uptime } from "node:os";
import { hostname } from "node:os";
import { readProcessStartedAt } from "./hot-restart.js";
import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js";
@@ -63,10 +64,17 @@ function processSpaceHash(): string {
const parts = [hostname()];
if (process.platform === "linux") {
parts.push(readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim(), readlinkSync("/proc/self/ns/pid"));
} else if (process.platform === "darwin") {
// One bounded, read-only kernel identity observation per module. Rounded
// boot-time estimates can collide across hosts and cannot authorize PID
// comparisons. Old weak hashes simply stop matching this stronger proof.
const bootSession = execFileSync("/usr/sbin/sysctl", ["-n", "kern.bootsessionuuid"], {
encoding: "utf8", timeout: 1_500, maxBuffer: 4_096, windowsHide: true,
}).trim().toLowerCase();
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(bootSession)) return UNKNOWN_PROCESS_SPACE;
parts.push(bootSession);
} else {
// No PID namespaces here. The boot time keeps a rebooted host distinct;
// rounding can only make a record unmatched, which fails closed.
parts.push(String(Math.round((Date.now() - uptime() * 1000) / 600_000)));
return UNKNOWN_PROCESS_SPACE;
}
if (parts.some(part => !part)) return UNKNOWN_PROCESS_SPACE;
return createHash("sha256").update(parts.join("\0")).digest("hex").slice(0, 32);
@@ -0,0 +1,71 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const state = vi.hoisted(() => ({ read: vi.fn(), link: vi.fn(), sysctl: vi.fn(), selfBirth: vi.fn() }));
vi.mock("node:fs", async original => ({ ...await original<typeof import("node:fs")>(),
readFileSync: state.read, readlinkSync: state.link }));
vi.mock("node:os", async original => ({ ...await original<typeof import("node:os")>(), hostname: () => "same-host" }));
vi.mock("node:child_process", async original => ({ ...await original<typeof import("node:child_process")>(), execFileSync: state.sysctl }));
vi.mock("./hot-restart.js", () => ({ readProcessStartedAt: state.selfBirth }));
const descriptor = Object.getOwnPropertyDescriptor(process, "platform")!;
const firstBoot = "00000000-0000-4000-8000-000000000001";
const secondBoot = "00000000-0000-4000-8000-000000000002";
function platform(value: string) { Object.defineProperty(process, "platform", { ...descriptor, value }); }
async function fresh() { vi.resetModules(); return import("./controller-process-identity.js"); }
beforeEach(() => {
vi.resetAllMocks();
state.read.mockReturnValue(firstBoot + "\n"); state.link.mockReturnValue("pid:[42]");
state.sysctl.mockReturnValue(firstBoot.toUpperCase() + "\n");
state.selfBirth.mockResolvedValue("2026-11-07T00:00:00.000Z");
});
afterEach(() => { Object.defineProperty(process, "platform", descriptor); vi.restoreAllMocks(); });
describe("controller process-space identity", () => {
it("keeps Linux boot and PID-namespace isolation without launching a command", async () => {
platform("linux");
const original = await (await fresh()).currentControllerProcess();
expect(original.hostHash).toMatch(/^[0-9a-f]{32}$/);
expect(state.read).toHaveBeenCalledWith("/proc/sys/kernel/random/boot_id", "utf8");
expect(state.link).toHaveBeenCalledWith("/proc/self/ns/pid");
expect(state.sysctl).not.toHaveBeenCalled();
state.link.mockReturnValue("pid:[43]");
expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(original.hostHash);
state.link.mockReturnValue("pid:[42]"); state.read.mockReturnValue(secondBoot);
expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(original.hostHash);
});
it("uses Darwin kernel boot UUID once per module, never a rounded clock identity", async () => {
platform("darwin");
const owner = await fresh();
const first = await owner.currentControllerProcess();
expect(first.hostHash).toMatch(/^[0-9a-f]{32}$/);
expect(state.sysctl).toHaveBeenCalledExactlyOnceWith("/usr/sbin/sysctl", ["-n", "kern.bootsessionuuid"],
{ encoding: "utf8", timeout: 1_500, maxBuffer: 4_096, windowsHide: true });
expect(state.read).not.toHaveBeenCalled(); expect(state.link).not.toHaveBeenCalled();
state.sysctl.mockReturnValue(secondBoot);
expect((await owner.currentControllerProcess()).hostHash).toBe(first.hostHash);
expect(state.sysctl).toHaveBeenCalledTimes(1);
expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(first.hostHash);
expect(state.sysctl).toHaveBeenCalledTimes(2);
expect(JSON.stringify(first)).not.toContain(firstBoot);
// Old weak/foreign receipts retain their original deadline: no local probe.
const kill = vi.spyOn(process, "kill");
expect(await owner.controllerProcessExited({ bootId: "old", pid: 2_000_000_001,
hostHash: "old-rounded-clock-hash", processStartedAt: null })).toBe(false);
expect(kill).not.toHaveBeenCalled();
});
it.each(["unreadable", "malformed", "unsupported"] as const)("fails closed without an owner PID probe for %s identity", async mode => {
platform(mode === "unsupported" ? "win32" : "darwin");
if (mode === "unreadable") state.sysctl.mockImplementation(() => { throw new Error("sysctl unavailable"); });
if (mode === "malformed") state.sysctl.mockReturnValue("not-a-kernel-uuid");
const owner = await fresh();
expect((await owner.currentControllerProcess()).hostHash).toBe("unknown");
const kill = vi.spyOn(process, "kill");
expect(await owner.controllerProcessExited({ bootId: "old", pid: 2_000_000_001,
hostHash: "unknown", processStartedAt: null })).toBe(false);
expect(kill).not.toHaveBeenCalled();
if (mode === "unsupported") expect(state.sysctl).not.toHaveBeenCalled();
});
});
@@ -39,6 +39,24 @@ const project = (
) => projectExecution(r, c, pending, undefined, now);
describe("execution truth projection", () => {
it.each(["contained", "ordinary", "wrong-source", "continued"] as const)("projects controller containment without inventing queued work: %s", state => {
const stopped = run({ id: "old", status: "failed" });
const action = { status: "resolved" as const, cause: "native_restart_recovery_blocked", nextAction: "Inspect the original owner",
evidence: { reason: "controller_process_space_unverified", continuationDelivery: "not_requested",
executionReconciliation: { runId: state === "wrong-source" ? "another-run" : "old", providerStopped: true,
...(state === "ordinary" ? {} : { controllerStopped: true, controllerAuthorityDigest: "a".repeat(64) }) },
...(state === "continued" ? { explicitUserContinuation: { previousRunId: "old", runId: "new" } } : {}),
} };
const projected = projectExecution(stopped, coordinator({ phase: "terminal_failure" }), [], action, now);
expect(projected.phase).toBe(state === "contained" ? "failed" : state === "continued" ? "completed" : "queued");
if (state === "contained") {
expect(projected.label).toBe("Execution contained");
expect(projected.nextAction).toContain("Use Retry");
expect(projected.successorRunId).toBeNull();
}
if (state === "continued") expect(projected.successorRunId).toBe("new");
});
it("offers Stop for every live run, including a long-silent native turn", () => {
const silentSince = new Date(now.getTime() - 35 * 60_000);
const silent = run({ lastUsefulActionAt: silentSince, startedAt: silentSince });
@@ -334,6 +334,27 @@ export function projectExecution(
"The task was closed or reassigned before delivery. Inspect the current task owner and status.";
return set("failed", "Continuation cancelled");
}
const reconciliation = recoveryAction.evidence.executionReconciliation;
if (recoveryAction.cause === "native_restart_recovery_blocked"
&& recoveryAction.evidence.reason === "controller_process_space_unverified"
&& recoveryAction.evidence.continuationDelivery === "not_requested"
&& reconciliation && typeof reconciliation === "object" && !Array.isArray(reconciliation)
&& "runId" in reconciliation && reconciliation.runId === run.id
&& "controllerStopped" in reconciliation && reconciliation.controllerStopped === true
&& "providerStopped" in reconciliation && reconciliation.providerStopped === true
&& "controllerAuthorityDigest" in reconciliation && typeof reconciliation.controllerAuthorityDigest === "string" && /^[0-9a-f]{64}$/.test(reconciliation.controllerAuthorityDigest)) {
const continuation = recoveryAction.evidence.explicitUserContinuation as
{ previousRunId?: unknown; runId?: unknown } | undefined;
const explicitSuccessor = text(continuation?.runId);
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
projection.successorRunId = explicitSuccessor;
projection.nextAction = null;
return set("completed", "Continued in another run");
}
projection.cause = null;
projection.nextAction = "The original execution was contained and its action outcomes reviewed. Use Retry to request another attempt.";
return set("failed", "Execution contained");
}
const deliveredRunId = text(recoveryAction.evidence.continuationRunId);
projection.successorRunId = deliveredRunId;
projection.cause = null;
@@ -1,3 +1,4 @@
import { settleOperatorContainedNativeOwner } from "./native-runtime/native-failed-run-owner-release.js";
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
@@ -14,6 +15,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { logger } from "../middleware/logger.js";
import { and, eq, exists, inArray, isNull, not, or, sql } from "drizzle-orm";
import {
agentWakeupRequests,
chatActions,
environmentLeases,
heartbeatRuns,
@@ -53,7 +55,7 @@ export function executionRecoverySettlementNote(input: {
: "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated.";
}
export async function validateExecutionReconciliation(input: {
type ExecutionReconciliationInput = {
db: Db;
companyId: string;
issueId: string;
@@ -62,13 +64,37 @@ export async function validateExecutionReconciliation(input: {
decision: ExecutionReconciliation | undefined;
/** Board-only source repair: never authorize a continuation of this run. */
workspaceRepairOnly?: boolean;
}) {
controllerContainment?: { actionId: string; actorId: string };
};
export async function validateExecutionReconciliation(input: ExecutionReconciliationInput) {
if (input.controllerContainment) {
// Validation and contained-owner settlement must roll back together even
// when a caller is not already inside the board route's transaction.
return input.db.transaction(tx => validateExecutionReconciliationInScope({ ...input, db: tx as unknown as Db }));
}
return validateExecutionReconciliationInScope(input);
}
async function validateExecutionReconciliationInScope(input: ExecutionReconciliationInput) {
const { db, companyId, issueId, agentId, decision } = input;
if (!decision || decision.runId !== input.sourceRunId || (!agentId && !input.workspaceRepairOnly)) {
throw conflict(
"Reconcile the recorded execution and its action outcomes before continuing this task.",
);
}
let operatorContained = false;
if (decision.controllerStopped !== undefined || decision.controllerAuthorityDigest !== undefined) {
if (!decision.controllerStopped || !decision.controllerAuthorityDigest || !input.controllerContainment
|| input.workspaceRepairOnly) throw conflict("Original-host controller containment requires its exact board recovery action.");
operatorContained = await settleOperatorContainedNativeOwner(db, {
companyId, issueId, runId: decision.runId, actionId: input.controllerContainment.actionId,
actorId: input.controllerContainment.actorId, authorityDigest: decision.controllerAuthorityDigest,
actionOutcome: decision.actionOutcome, outcomeEvidence: decision.outcomeEvidence,
});
if (!operatorContained) throw conflict("The recorded controller authority changed, is still leased, or its cleanup is unsettled. Refresh before recording containment.",
{ code: "native_controller_containment_stale" });
}
const [run] = await db
.select()
.from(heartbeatRuns)
@@ -110,7 +136,7 @@ export async function validateExecutionReconciliation(input: {
(!decision.workspaceRepairEvidence || decision.workspaceRepairEvidence.trim().length < 20)) {
throw conflict("Verify safe workspace staging or repair and record workspaceRepairEvidence before continuing this run.");
}
let checkLocalProcesses = true;
let checkLocalProcesses = !operatorContained;
if (retainedWorkspace) {
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id),
@@ -128,7 +154,7 @@ export async function validateExecutionReconciliation(input: {
: !hasRemoteTerminationReceipt(lease))) {
throw conflict("The retained workspace environment has not confirmed that it stopped. Wait for cleanup before recording workspace repair.");
}
checkLocalProcesses = leases.some(lease => lease.provider === "local");
checkLocalProcesses = !operatorContained && leases.some(lease => lease.provider === "local");
}
// A retained remote source's PIDs belong to its sandbox, not this server.
// Its exact lease receipts above replace host-local process probes.
@@ -199,8 +225,10 @@ export async function markExecutionReconciliation(
decision: ExecutionReconciliation,
actorId: string,
deliveryOwner?: { kind: "chat_failed_run_retry"; actionId: string },
options?: { workspaceRepairOnly?: boolean },
options?: { workspaceRepairOnly?: boolean; controllerContainmentOnly?: boolean },
) {
if (options?.controllerContainmentOnly && (!decision.controllerStopped || !decision.controllerAuthorityDigest || deliveryOwner))
throw conflict("Controller containment records no automatic continuation.");
if (options?.workspaceRepairOnly && (deliveryOwner || !hasRequiredWorkspaceRecovery(action.evidence))) {
throw conflict("Workspace repair must target the retained source without a continuation owner.");
}
@@ -252,7 +280,7 @@ export async function markExecutionReconciliation(
actorId,
recordedAt: new Date().toISOString(),
},
continuationDelivery: options?.workspaceRepairOnly ? "not_requested" : deliveryOwner ? "delegated" : "pending",
continuationDelivery: options?.workspaceRepairOnly || options?.controllerContainmentOnly ? "not_requested" : deliveryOwner ? "delegated" : "pending",
...(deliveryOwner ? { continuationDeliveryOwner: deliveryOwner } : {}),
},
})
@@ -267,6 +295,20 @@ export async function markExecutionReconciliation(
kind: issueRecoveryActions.kind,
evidence: issueRecoveryActions.evidence,
});
if (options?.controllerContainmentOnly) {
// A Retry clicked before the operator reviewed the effects must not turn
// into an automatic replay after containment. Retain only that exact,
// unadmitted source receipt as skipped; newer/accepted intents are untouched.
await tx.update(agentWakeupRequests).set({ status: "skipped", finishedAt: new Date(), updatedAt: new Date(),
error: "controller_containment_requires_new_retry" }).where(and(
eq(agentWakeupRequests.companyId, action.companyId), isNull(agentWakeupRequests.runId),
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]),
sql`${agentWakeupRequests.payload}->>'issueId' = ${action.sourceIssueId}`,
sql`${agentWakeupRequests.payload}->'taskRetry'->>'schema' = 'paperclip.task-retry.v1'`,
sql`${agentWakeupRequests.payload}->'taskRetry'->>'sourceRunId' = ${decision.runId}`,
));
return;
}
if (
!source || options?.workspaceRepairOnly ||
source.cause !== "legacy_execution_requires_reconciliation" ||
+10 -1
View File
@@ -6,7 +6,7 @@ import { writeExecutionControl } from "./execution-control-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { and, desc, eq, inArray, isNull, isNotNull, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { heartbeatRuns, issueRecoveryActions, workspaceOperations } from "@paperclipai/db";
import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
import { isUuidLike } from "@paperclipai/shared";
import type {
IssueRecoveryAction,
@@ -623,6 +623,15 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
.where(and(...predicates)).limit(1);
if (!pending.length) return [];
// Resolving another attention releases this issue's preserved Native
// waiter to one normal inspection. No recurring monitor or new turn.
const [waiter] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.sourceIssueId),
eq(nativeRunFinalizations.phase, "observed"), isNull(nativeRunFinalizations.resultId),
isNotNull(nativeRunFinalizations.leaseOwner), inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted"]),
)).limit(1);
if (waiter) await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization);
await notifyIssueRecovery(tx, { companyId: input.companyId });
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
@@ -1,4 +1,4 @@
import { and, eq, gt, inArray, isNull, notLike, sql } from "drizzle-orm";
import { and, eq, gt, inArray, isNotNull, isNull, notInArray, notLike, sql } from "drizzle-orm";
import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
@@ -6,8 +6,37 @@ import { notifyNativeResultCommit } from "../native-result-commit-notifications.
import type { OwnerProof } from "../owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "../recovery/ownership-registry.js";
import { requestNativeControllerStopConfirmation } from "./native-controller-host-recovery.js";
import { nativeControllerOwnerStamp } from "./native-owner-stamps.js";
import { readNativeRemoteExecutionOwnershipForRuns } from "./native-runner-ownership.js";
import { recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
export async function classifyNativeAcceptedResultLeaseOwners(db: Db, records: readonly InProgressRecord[]): Promise<InProgressRecord[]> {
if (!records.length) return [];
const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations })
.from(nativeRunFinalizations).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
.where(and(inArray(nativeRunFinalizations.runId, records.map(record => record.recordId)),
isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, ["committed", "terminal_failure"]),
isNotNull(nativeRunFinalizations.leaseOwner), notLike(nativeRunFinalizations.leaseOwner, "native-finalizer:%"),
notLike(nativeRunFinalizations.leaseOwner, "native-cleanup:%"), sql`${nativeRunFinalizations.leaseExpiresAt} > now()`));
const ownership = await readNativeRemoteExecutionOwnershipForRuns(db,
rows.map(row => ({ run: row.run, issueId: row.coordinator.issueId })));
const byRun = new Map(rows.map((row, index) => [row.run.id, { ...row, remote: ownership[index]!.remoteProcess }]));
return records.flatMap<InProgressRecord>(record => {
const row = byRun.get(record.recordId);
if (!row || row.run.companyId !== record.companyId || row.coordinator.issueId !== record.issueId)
return [];
const current = { recordId: row.run.id, companyId: row.run.companyId, runId: row.run.id,
issueId: row.coordinator.issueId, updatedAt: row.coordinator.updatedAt };
if (!row.remote) return [{ ...current,
owner: { kind: "process", stamp: nativeControllerOwnerStamp(row.coordinator, "native_accepted_result_lease") } }];
return [{ ...current, owner: { kind: "fence", stamp: { kind: "remote", scope: "native_accepted_result_lease",
generation: row.coordinator.controllerGeneration, fenceToken: row.coordinator.leaseOwner },
fence: { currentFenceToken: row.coordinator.leaseOwner, currentGeneration: row.coordinator.controllerGeneration,
leaseExpiresAt: row.coordinator.leaseExpiresAt } } }];
});
}
/**
* Settle of the `native_accepted_result_lease` ownership kind (plan Phase 2).
*
@@ -36,6 +65,7 @@ export async function settleNativeAcceptedResultLease(db: Db, record: InProgress
issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId,
leaseOwner: nativeRunFinalizations.leaseOwner, controllerBootId: nativeRunFinalizations.controllerBootId,
controllerPid: nativeRunFinalizations.controllerPid, controllerHostHash: nativeRunFinalizations.controllerHostHash,
controllerGeneration: nativeRunFinalizations.controllerGeneration,
controllerProcessStartedAt: nativeRunFinalizations.controllerProcessStartedAt,
remote: sql<boolean>`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is not null
or ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' is not null)`,
@@ -55,7 +85,8 @@ export async function settleNativeAcceptedResultLease(db: Db, record: InProgress
// Remote copyback is fenced by its own controller lease, never by a local proof.
if (row.remote) return { settled: false, reason: "remote_lease_fenced" };
// The proof judged this exact controller; a different one holds it now.
if (row.controllerBootId !== stamp.bootId || row.controllerPid !== stamp.pid
if (row.leaseOwner !== stamp.fenceToken || row.controllerGeneration !== stamp.generation
|| row.controllerBootId !== stamp.bootId || row.controllerPid !== stamp.pid
|| (row.controllerHostHash ?? null) !== (stamp.processSpace ?? null)
|| (row.controllerProcessStartedAt?.toISOString() ?? null) !== (stamp.processStartedAt ?? null)) {
return { settled: false, reason: "owner_changed_since_proof" };
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { and, eq } from "drizzle-orm";
import {
activityLog, agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns,
activityLog, agents, companies, createDb, environmentLeases, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns,
issueRecoveryActions, issues, nativeRunFinalizations,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
@@ -29,6 +29,8 @@ describe("native controller rows saved before their host was recorded", () => {
await db.insert(issues).values({ id: issueId, companyId, title: "Native task", status: "in_progress", assigneeAgentId: agentId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, status: "running",
runtimeMode: "native", contextSnapshot: { issueId } });
// Native admission binds its running source before exposing recovery.
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
// The shape the previous release saved: a local controller and no host.
const [coordinator] = await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed",
attempt: 1, leaseOwner: "old-controller", leaseExpiresAt: new Date(0), controllerBootId: randomUUID(),
@@ -77,6 +79,66 @@ describe("native controller rows saved before their host was recorded", () => {
await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: actions[0]!.id, runId,
stopEvidence, actorId: "board-user" })).rejects.toMatchObject({ status: 409 });
});
it("keeps an observed no-checkpoint source recoverable after a controller-only confirmation", async () => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const { companyId, agentId, issueId, runId, coordinator } = await seed();
// Historical initial state: the previous release published its real
// controller-only action for an observed run with no checkpoint. There
// is no accepted provider result and no fabricated terminal outcome.
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
await db.update(nativeRunFinalizations).set({ recoveryState: "awaiting_evidence" })
.where(eq(nativeRunFinalizations.runId, runId));
await requestNativeControllerStopConfirmation(db, coordinator);
const [originalAction] = await db.select().from(issueRecoveryActions)
.where(eq(issueRecoveryActions.sourceIssueId, issueId));
await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: originalAction!.id, runId,
stopEvidence: "The original server container is stopped; provider effects are not yet verified.",
actorId: "board-user" })).resolves.toEqual({ runId, status: "queued" });
const [original] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, originalAction!.id));
expect(original).toMatchObject({ status: "resolved", evidence: { controllerStop: { actorId: "board-user" } } });
const [held] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
expect(held).toMatchObject({ phase: "observed", resultId: null, leaseOwner: coordinator.leaseOwner,
controllerGeneration: coordinator.controllerGeneration, attempt: coordinator.attempt });
const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(source).toMatchObject({ status: "running", resultJson: null });
const active = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.status, "active")));
expect(active).toEqual([expect.objectContaining({ cause: "native_restart_recovery_blocked", ownerType: "board",
returnOwnerAgentId: agentId, attemptCount: 0, wakePolicy: null, monitorPolicy: null,
evidence: expect.objectContaining({ runId, reason: "controller_process_space_unverified",
controllerAuthority: expect.objectContaining({ schema: "paperclip.native-controller-authority.v1",
digest: expect.stringMatching(/^[a-f0-9]{64}$/) }) }) })]);
expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId)))
.toEqual([]);
});
it.each(["newer-owner", "newer-source"] as const)(
"keeps a stale controller-only confirmation from changing %s", async kind => {
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true");
const { companyId, agentId, issueId, runId, coordinator } = await seed();
await requestNativeControllerStopConfirmation(db, coordinator);
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
if (kind === "newer-owner") {
await db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(),
controllerGeneration: coordinator.controllerGeneration + 1, leaseOwner: "new-controller" })
.where(eq(nativeRunFinalizations.runId, runId));
} else {
const nextRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issueId,
runtimeMode: "native", status: "queued", contextSnapshot: { issueId } });
await db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, issueId));
}
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: action!.id, runId,
stopEvidence: "The previous server stopped, but this dialog predates the current owner.", actorId: "board-user" }))
.rejects.toMatchObject({ status: 409 });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]).toEqual(before);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action!.id)))[0])
.toMatchObject({ status: "active" });
},
);
it("binds stop confirmation to an exact foreign-host controller and rejects changed ownership", async () => {
const { companyId, issueId, runId, coordinator } = await seed();
@@ -101,4 +163,28 @@ describe("native controller rows saved before their host was recorded", () => {
.toEqual({ allowed: true, reason: "operator_attested_controller_stopped" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status).toBe("running");
});
it.each(["same-task", "different-task"] as const)("binds controller follow-up to the %s runner lease", async scope => {
const { companyId, agentId, issueId, runId, coordinator } = await seed();
let leaseIssueId = issueId;
if (scope === "different-task") {
leaseIssueId = randomUUID();
await db.insert(issues).values({ id: leaseIssueId, companyId, title: "Another task", status: "in_progress", assigneeAgentId: agentId });
}
await db.insert(environmentLeases).values({ companyId, issueId: leaseIssueId, heartbeatRunId: runId,
provider: "plugin", status: "active", metadata: { runnerTaskLease: true } });
await requestNativeControllerStopConfirmation(db, coordinator);
const [original] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
await expect(confirmNativeControllerStopped({ db, companyId, issueId, runId, actionId: original.id,
actorId: "board-user", stopEvidence: "The original controller is stopped; provider containment remains unverified." }))
.resolves.toEqual({ runId, status: "queued" });
const [attention] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId),
eq(issueRecoveryActions.status, "active")));
expect(attention.evidence.reason).toBe(scope === "same-task"
? "remote_controller_process_space_unverified" : "controller_process_space_unverified");
if (scope === "same-task") expect(attention.evidence.controllerAuthority).toBeUndefined();
else expect(attention.evidence.controllerAuthority).toMatchObject({ schema: "paperclip.native-controller-authority.v1" });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0])
.toMatchObject({ phase: "observed", resultId: null, leaseOwner: coordinator.leaseOwner });
});
});
@@ -1,9 +1,12 @@
import { and, eq, isNull } from "drizzle-orm";
import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { readNativeRemoteExecutionOwnership } from "./native-runner-ownership.js";
import { and, eq, isNull, ne, or, sql } from "drizzle-orm";
import { heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { confirmNativeControllerStoppedSchema } from "@paperclipai/shared";
import { conflict } from "../../errors.js";
import { persistActivity, publishActivity } from "../activity-log.js";
import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js";
import { surfaceNativeControllerNamespaceAttention } from "./native-restart-recovery.js";
import { readPendingNativeRunStop } from "./native-cancellation-request.js";
export const NATIVE_CONTROLLER_HOST_UNVERIFIED = "native_controller_host_unverified";
/** controller_host_hash value written once the board confirms the stop. */
@@ -27,7 +30,7 @@ export async function requestNativeControllerStopConfirmation(tx: Db, coordinato
processStartedAt: coordinator.controllerProcessStartedAt?.toISOString() ?? null } },
nextAction: "Paperclip cannot verify that this run's previous server stopped from the current host. "
+ "Confirm that the previous server process has stopped (for example, its "
+ "container was replaced), and the run resumes from its saved state.",
+ "container was replaced), then recovery checks the saved run. Controller confirmation does not confirm that its provider or tools stopped.",
}).onConflictDoNothing();
}
@@ -41,35 +44,78 @@ export async function confirmNativeControllerStopped(input: {
});
const committed = await input.db.transaction(async transaction => {
const tx = transaction as unknown as Db;
// Match Native admission and containment lock order. This records only
// the previous controller stop, not provider containment or a new turn.
const [task] = await tx.select().from(issues).where(and(
eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))).for("update");
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId),
eq(issueRecoveryActions.id, input.actionId))).for("update");
const [run] = await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
const recorded = (action?.evidence?.controller ?? {}) as Record<string, unknown>;
if (!coordinator || !action || !run || action.cause !== NATIVE_CONTROLLER_HOST_UNVERIFIED || action.ownerType !== "board"
if (!task || !coordinator || !action || !run || action.cause !== NATIVE_CONTROLLER_HOST_UNVERIFIED || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || action.evidence?.runId !== input.runId
|| coordinator.controllerHostHash !== (recorded.hostHash ?? null) || coordinator.controllerPid !== recorded.pid
|| coordinator.controllerBootId !== recorded.bootId
|| (coordinator.controllerProcessStartedAt?.toISOString() ?? null) !== recorded.processStartedAt) throw changed();
|| (coordinator.controllerProcessStartedAt?.toISOString() ?? null) !== recorded.processStartedAt
|| run.runtimeMode !== "native"
|| !(coordinator.resultId === null ? run.status === "running" : ["running", "cancelled"].includes(run.status))
|| (coordinator.resultId === null ? !["observed", "retryable_failure"].includes(coordinator.phase)
: ["committed", "terminal_failure"].includes(coordinator.phase))
|| task.assigneeAgentId !== run.agentId || task.executionRunId !== run.id
|| (task.checkoutRunId !== null && task.checkoutRunId !== run.id)
|| (run.nativeIssueId ?? run.contextSnapshot?.issueId) !== task.id
|| ["done", "cancelled"].includes(task.status)
|| !coordinator.leaseOwner || !coordinator.leaseExpiresAt
|| coordinator.leaseOwner.startsWith("native-finalizer:") || coordinator.leaseOwner.startsWith("native-cleanup:")
|| await readPendingNativeRunStop(tx, run)) throw changed();
const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), ne(heartbeatRuns.id, run.id),
or(eq(heartbeatRuns.nativeIssueId, task.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${task.id}`,
sql`exists (select 1 from ${nativeRunFinalizations} newer_owner
where newer_owner.run_id = ${heartbeatRuns.id} and newer_owner.company_id = ${run.companyId}
and newer_owner.issue_id = ${task.id})`),
sql`${heartbeatRuns.createdAt} >= (select source.created_at from ${heartbeatRuns} source
where source.id = ${run.id} and source.company_id = ${run.companyId})`,
)).limit(1);
if (newer) throw changed();
// Use the DB's full precision, not a Date or session-dependent rendering.
const [expiry] = await tx.select({ epoch: sql<string | null>`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
if (!expiry?.epoch) throw changed();
// Work intent before the write it announces (sleep-aware background work).
await notifyLiveRunRecovery(tx, { companyId: input.companyId, runId: input.runId, agentId: run.agentId });
const now = new Date();
const updated = await tx.update(nativeRunFinalizations).set({ controllerHostHash: OPERATOR_ATTESTED_CONTROLLER_HOST, updatedAt: now })
const [updated] = await tx.update(nativeRunFinalizations).set({ controllerHostHash: OPERATOR_ATTESTED_CONTROLLER_HOST, updatedAt: now })
.where(and(eq(nativeRunFinalizations.runId, input.runId), coordinator.controllerHostHash === null
? isNull(nativeRunFinalizations.controllerHostHash) : eq(nativeRunFinalizations.controllerHostHash, coordinator.controllerHostHash),
eq(nativeRunFinalizations.controllerPid, coordinator.controllerPid!),
eq(nativeRunFinalizations.controllerBootId, coordinator.controllerBootId!)))
.returning({ runId: nativeRunFinalizations.runId });
if (!updated.length) throw changed();
eq(nativeRunFinalizations.controllerBootId, coordinator.controllerBootId!),
eq(nativeRunFinalizations.leaseOwner, coordinator.leaseOwner),
eq(nativeRunFinalizations.controllerGeneration, coordinator.controllerGeneration),
eq(nativeRunFinalizations.attempt, coordinator.attempt),
eq(nativeRunFinalizations.phase, coordinator.phase),
coordinator.resultId === null ? isNull(nativeRunFinalizations.resultId)
: eq(nativeRunFinalizations.resultId, coordinator.resultId),
sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text = ${expiry.epoch}`))
.returning();
if (!updated) throw changed();
const proof = { controller: recorded, actorId: input.actorId, stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() };
await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now,
evidence: { ...action.evidence, controllerStop: proof },
nextAction: "The previous server was confirmed stopped. Recovery resumes the run from its saved state.",
nextAction: "The previous server was confirmed stopped. Recovery will inspect the saved run. Separate provider containment may still be required.",
}).where(eq(issueRecoveryActions.id, action.id));
// The controller-only action is now resolved. Publish its responsible
// next owner in this same transaction: an observed/no-checkpoint run is
// not necessarily admitted by any deferred recovery scanner.
if (coordinator.resultId === null) {
const { remoteProcess } = await readNativeRemoteExecutionOwnership(tx, { run, issueId: task.id });
await surfaceNativeControllerNamespaceAttention(tx, { run, coordinator: updated, expiryEpoch: expiry.epoch, remoteProcess });
}
const publication = await persistActivity(tx, { companyId: input.companyId, actorType: "user", actorId: input.actorId,
action: "issue.native_controller_stop_confirmed", entityType: "issue", entityId: input.issueId, runId: input.runId,
details: { recoveryActionId: input.actionId, ...proof } });
@@ -1,7 +1,7 @@
import { spawn } from "node:child_process";
import { randomUUID } from "node:crypto";
import { once } from "node:events";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { and, eq } from "drizzle-orm";
import {
activityLog,
@@ -12,10 +12,12 @@ import {
environmentLeases,
heartbeatRuns,
issues,
issueRecoveryActions,
nativeRunFinalizations,
subscribeDatabaseWork,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { agentExecutionsHaveStopped } from "../agent-execution-stop.js";
import { readProcessStartedAt } from "../hot-restart.js";
import { currentProcessSpaceHash } from "../controller-process-identity.js";
@@ -31,6 +33,8 @@ import {
settleUnstartedNativeSessionResume,
} from "./native-failed-run-owner-release.js";
import * as controllerProcess from "../controller-process-identity.js";
const LEASE_FAILURE = 'Failed to acquire lease for environment "Managed computer" (sandbox): Sandbox creation has failed';
describe("failed native run owner release", () => {
@@ -63,9 +67,10 @@ describe("failed native run owner release", () => {
errorCode?: string;
error?: string;
coordinator: Partial<typeof nativeRunFinalizations.$inferInsert>;
runId?: string;
}) {
const issueId = randomUUID();
const runId = randomUUID();
const runId = input.runId ?? randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Resume the native run", status: "in_progress",
assigneeAgentId: agentId, workMode: "standard" });
await db.insert(heartbeatRuns).values({
@@ -80,8 +85,10 @@ describe("failed native run owner release", () => {
providerSessionId: "provider-session", activeTurnId: "active-turn" },
},
});
const current = await controllerProcess.currentControllerProcess();
expect(current.hostHash).not.toBe("unknown");
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1,
...input.coordinator } as typeof nativeRunFinalizations.$inferInsert);
controllerHostHash: current.hostHash, ...input.coordinator } as typeof nativeRunFinalizations.$inferInsert);
return { runId, issueId };
}
@@ -166,6 +173,194 @@ describe("failed native run owner release", () => {
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
});
it.each(["null", "foreign", "unknown-current"] as const)("keeps a failed namespace-held source out of automatic expired-owner release: %s", async mode => {
const actual = await controllerProcess.currentControllerProcess();
expect(actual.hostHash).not.toBe("unknown");
const recordedPid = 2_000_000_001;
const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed",
coordinator: { leaseOwner: `foreign-controller:${randomUUID()}`, leaseExpiresAt: new Date(0),
controllerBootId: randomUUID(), controllerPid: recordedPid,
controllerHostHash: mode === "null" ? null : mode === "foreign" ? `foreign:${actual.hostHash}` : actual.hostHash,
recoveryState: "awaiting_evidence" } });
await db.update(heartbeatRuns).set({ processPid: recordedPid, processGroupId: recordedPid })
.where(eq(heartbeatRuns.id, runId));
const before = await coordinator(runId);
const originalKill = process.kill.bind(process);
const probes: number[] = [];
const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
if (Math.abs(pid) === recordedPid && signal === 0) {
probes.push(pid);
throw Object.assign(new Error("controlled local PID absent; not foreign stop proof"), { code: "ESRCH" });
}
return originalKill(pid, signal);
});
const identity = mode === "unknown-current" ? vi.spyOn(controllerProcess, "currentProcessSpaceHash")
.mockReturnValue("unknown") : null;
try {
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
expect(probes).toEqual([]);
expect(await coordinator(runId)).toEqual(before);
const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId));
expect(attention).toMatchObject({ cause: "native_restart_recovery_blocked", attemptCount: 0, wakePolicy: null, monitorPolicy: null,
evidence: { runId, reason: "controller_process_space_unverified" } });
for (let inspection = 0; inspection < 3; inspection++) {
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await executionControlSchedule(db).finalization()).toBeNull();
}
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId))).toEqual([attention]);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.action, "native.failed_run_owner_released")))).toHaveLength(0);
} finally { identity?.mockRestore(); probe.mockRestore(); }
});
it.each(["timed_out", "interrupted"] as const)("publishes quiet operator attention after an already %s owner expires", async status => {
const { runId, issueId } = await nativeRun({ status, coordinator: { leaseOwner: `foreign:${randomUUID()}`,
controllerBootId: randomUUID(), controllerHostHash: null, leaseExpiresAt: new Date(0) } });
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
expect(action).toMatchObject({ ownerType: "board", cause: "native_restart_recovery_blocked", attemptCount: 0,
evidence: { runId, reason: "controller_process_space_unverified", controllerAuthority: { digest: expect.any(String) } } });
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) });
});
it("keeps empty deadline and release scans from stamping controller ownership", async () => {
const stamped = vi.fn();
const unsubscribe = controllerProcess.onControllerStamp(stamped);
try {
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(stamped).not.toHaveBeenCalled();
} finally { unsubscribe(); }
});
it("retains a foreign owner's original deadline until its returned unstarted claim settles", async () => {
const leaseExpiresAt = new Date(Date.now() + 60_000);
const leaseOwner = `default:resume:${randomUUID()}`;
// This dispatched claim returned before the executor took an attempt. Its
// exact returned token/attempt, not the PID or lease age, authorizes release.
const { runId, issueId } = await nativeRun({ status: "failed", errorCode: "setup_failed",
coordinator: { leaseOwner, leaseExpiresAt, controllerBootId: randomUUID(),
controllerHostHash: `foreign:${currentProcessSpaceHash()}` } });
const before = await coordinator(runId);
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime());
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
expect(await coordinator(runId)).toEqual(before);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
expect(await settleUnstartedNativeSessionResume(db, { runId, leaseOwner, attempt: before.attempt })).toBe(true);
expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null });
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
});
it("surfaces a foreign owner only at its original expiry without stamping or probing it", async () => {
const leaseExpiresAt = new Date(Date.now() + 60_000);
const recordedPid = 2_000_000_001;
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
leaseOwner: `foreign:${randomUUID()}`, leaseExpiresAt, controllerBootId: randomUUID(),
controllerPid: recordedPid, controllerHostHash: null } });
const before = await coordinator(runId);
const stamped = vi.fn();
const unsubscribe = controllerProcess.onControllerStamp(stamped);
const kill = process.kill.bind(process);
const probes: number[] = [];
const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
if (Math.abs(pid) === recordedPid && signal === 0) {
probes.push(pid);
throw Object.assign(new Error("foreign PID is not local stop evidence"), { code: "ESRCH" });
}
return kill(pid, signal);
});
try {
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime());
expect(await releaseAbandonedNativeOwners(db, new Date(leaseExpiresAt.getTime() - 1))).toBe(0);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]);
expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0);
const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
expect(attention).toMatchObject({ status: "active", ownerType: "board", attemptCount: 0,
wakePolicy: null, monitorPolicy: null, evidence: { runId, reason: "controller_process_space_unverified" } });
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0);
expect(await coordinator(runId)).toEqual(before);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([attention]);
expect(probes).toEqual([]);
expect(stamped).not.toHaveBeenCalled();
} finally { probe.mockRestore(); unsubscribe(); }
});
it("finishes each requested namespace inspection pass and preserves the original local deadline", async () => {
const foreign = [];
for (let index = 0; index < 5; index++) foreign.push(await nativeRun({ status: "failed", coordinator: {
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } }));
const localDeadline = new Date(Date.now() + 600_000);
await nativeRun({ status: "failed", coordinator: { leaseOwner: `local:${randomUUID()}`, leaseExpiresAt: localDeadline } });
for (let page = 0; page < 2; page++) {
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
}
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId));
expect(rows).toHaveLength(foreign.length);
expect(rows.every(row => row.attemptCount === 0 && row.wakePolicy === null && row.monitorPolicy === null)).toBe(true);
// Deadline reads do not request another pass or turn surfaced holds into timers.
for (let read = 0; read < 2; read++) {
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
}
// A later explicit mutation inspection must recheck all retained authorities.
// Finish its known three-page pass before expecting the local timer again.
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime());
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime());
const reinspected = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId));
const byId = (a: typeof rows[number], b: typeof rows[number]) => a.id.localeCompare(b.id);
expect(reinspected.sort(byId)).toEqual([...rows].sort(byId));
});
it("keeps a lower-sorting arrival durable when its notification coalesces between inspection pages", async () => {
for (let index = 1; index <= 4; index++) await nativeRun({ status: "failed",
runId: `ffffffff-ffff-4fff-8fff-${String(index).padStart(12, "0")}`, coordinator: {
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
const lower = await nativeRun({ status: "failed", runId: "00000000-0000-4000-8000-000000000001", coordinator: {
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
// Simulate one coalesced dirty signal by continuing the actual existing
// page, not resetting its cursor or creating a second artificial worker.
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull();
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toEqual([]);
expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0);
expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toHaveLength(1);
expect(await releaseAbandonedNativeOwners(db, new Date(), 5)).toBe(0);
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
});
it("surfaces a preserved namespace waiter when an unrelated action is actually resolved", async () => {
const { runId, issueId } = await nativeRun({ status: "failed", coordinator: {
leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } });
const service = issueRecoveryActionService(db);
const unrelated = await service.upsertSourceScoped({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
cause: "independent-test-recovery", fingerprint: "independent-test-identity", ownerType: "board",
nextAction: "Resolve this independently owned action first.", wakePolicy: null, monitorPolicy: null, attemptCount: 0 });
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0].id).toBe(unrelated.id);
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
expect(await committedWakes(DELIVERY_QUEUES.executionFinalization, () => service.resolveActiveForIssue({ companyId,
sourceIssueId: issueId, actionId: unrelated.id, status: "resolved", outcome: "restored" }))).toBeGreaterThan(0);
expect(await releaseAbandonedNativeOwners(db)).toBe(0);
const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId));
expect(rows).toHaveLength(2);
expect(rows.find(row => row.id === unrelated.id)).toMatchObject({ status: "resolved", fingerprint: "independent-test-identity" });
expect(rows.find(row => row.id !== unrelated.id)).toMatchObject({ attemptCount: 0,
evidence: { runId, reason: "controller_process_space_unverified" } });
expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull();
});
it("keeps an unexpired owner blocking the stop check and schedules its expiry", async () => {
const leaseExpiresAt = new Date(Date.now() + 10 * 60_000);
const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed", error: LEASE_FAILURE,
@@ -1,15 +1,21 @@
import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { currentProcessSpaceHash } from "../controller-process-identity.js";
import type { ExecutionReconciliation } from "@paperclipai/shared";
import { nativeControllerAuthorityDigest, readNativeRemoteExecutionOwnership, isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js";
import { settleNativeTerminalWake } from "./native-run-projection.js";
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm";
import { activityLog, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db";
import { logger } from "../../middleware/logger.js";
import { adapterExecutionControls } from "../adapter-execution-control.js";
import { proveOwnerGone, type LocalOwnerStamp } from "../owner-proof.js";
import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
import { logActivity, persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { LEGACY_WORKSPACE_RECOVERY_KEY } from "../workspace-restore-recovery-state.js";
import { activeNativeExecutionRunIds } from "./native-active-runs.js";
import { isCurrentNativeControllerProcess, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
import { currentNativeControllerIdentity, isCurrentNativeControllerProcess, surfaceNativeControllerNamespaceAttention, recordLegacyHostFallbackProof } from "./native-restart-recovery.js";
import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "./native-owner-stamps.js";
/**
@@ -37,7 +43,10 @@ type Release =
/** The caller dispatched this claim and its execution has returned. */
| { kind: "unstarted_resume"; runId: string; leaseOwner: string; attempt: number }
/** A background pass found an expired owner and must prove it stopped. */
| { kind: "expired_owner"; runId: string; leaseOwner: string; attempt: number; controllerGeneration: number };
| { kind: "expired_owner"; runId: string; leaseOwner: string; attempt: number; controllerGeneration: number }
| { kind: "operator_containment"; runId: string; leaseOwner: string; attempt: number;
companyId: string; issueId: string; actionId: string; authorityDigest: string; actorId: string; controllerGeneration: number;
actionOutcome: ExecutionReconciliation["actionOutcome"]; outcomeEvidence: string };
/** Environment leases that `agentExecutionsHaveStopped` would still treat as live. */
const unsettledRunEnvironmentLease = sql`exists (
@@ -70,19 +79,44 @@ function abandonedOwnerCondition() {
)!;
}
/**
* The earliest lease expiry the background pass will need to inspect. Sandbox
* cleanup is excluded, so an unreleased lease is never polled for here. An
* expired row whose process or controller may still be live keeps the shared
* finalization retry cadence until that process exits.
*/
/** Local owners retain their release deadline. Foreign owners retain the same
* original deadline until expiry surfaces attention owned by the board. */
async function localOwnerSpaceCondition() {
const processSpace = currentProcessSpaceHash();
const native = await currentNativeControllerIdentity();
return or(
and(eq(nativeRunFinalizations.controllerBootId, native.bootId), eq(nativeRunFinalizations.controllerPid, native.pid)),
processSpace !== "unknown" ? eq(nativeRunFinalizations.controllerHostHash, processSpace) : sql`false`,
)!;
}
export async function nextAbandonedNativeOwnerReleaseAt(db: Db): Promise<Date | null> {
// A finite remaining inspection page is work; a surfaced foreign owner is not
// a timer. The existing cursor is cleared after the last lookahead page.
if (abandonedOwnerReleaseCursor !== null) return new Date();
const local = await localOwnerSpaceCondition();
// Notifications can coalesce while a partial page advances. A lower-sorting
// arrival without any active attention remains durable inspection work, so
// completing the higher pages cannot consume its only wake.
const [unattended] = await db.select({ at: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
.where(and(abandonedOwnerCondition(), sql`not coalesce(${local}, false)`,
sql`${nativeRunFinalizations.leaseOwner} not like 'native-finalizer:%'`,
sql`${nativeRunFinalizations.leaseOwner} not like 'native-cleanup:%'`,
sql`not exists (select 1 from ${issueRecoveryActions} attention
where attention.company_id = ${nativeRunFinalizations.companyId}
and attention.source_issue_id = ${nativeRunFinalizations.issueId}
and attention.status in ('active', 'escalated'))`,
)).orderBy(asc(nativeRunFinalizations.leaseExpiresAt)).limit(1);
const [row] = await db.select({ at: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
.where(abandonedOwnerCondition())
.where(and(abandonedOwnerCondition(), sql`coalesce(${local}, false)`))
.orderBy(asc(nativeRunFinalizations.leaseExpiresAt)).limit(1);
return row?.at ?? null;
if (!unattended?.at) return row?.at ?? null;
if (!row?.at) return unattended.at;
return unattended.at < row.at ? unattended.at : row.at;
}
/** A proof that used the one-release fallback for a row saved before its PID
@@ -132,20 +166,94 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, release.runId));
if (!binding) return false;
// Same issue-before-coordinator-before-run order as admission and failure settlement.
await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, binding.companyId),
const [task] = await tx.select({ id: issues.id, assigneeAgentId: issues.assigneeAgentId,
status: issues.status, executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues).where(and(eq(issues.companyId, binding.companyId),
eq(issues.id, binding.issueId))).for("update");
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, binding.companyId), eq(nativeRunFinalizations.runId, release.runId),
eq(nativeRunFinalizations.issueId, binding.issueId))).for("update");
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, binding.companyId), eq(heartbeatRuns.id, release.runId))).for("update");
if (!coordinator || !run || run.runtimeMode !== "native" || !RELEASABLE_RUN_STATUSES.includes(run.status)
|| coordinator.phase !== "observed" || coordinator.resultId !== null
if (!coordinator || !run || run.runtimeMode !== "native"
|| !(RELEASABLE_RUN_STATUSES.includes(run.status) || (release.kind === "operator_containment" && run.status === "running"))
|| !(coordinator.phase === "observed" || (release.kind === "operator_containment" && coordinator.phase === "retryable_failure"))
|| coordinator.resultId !== null
|| coordinator.leaseOwner !== release.leaseOwner || coordinator.attempt !== release.attempt) return false;
if (release.kind === "expired_owner") {
if (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > now
|| coordinator.controllerGeneration !== release.controllerGeneration
|| run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null) return false;
let containedExpiryEpoch: string | null = null;
if (release.kind === "operator_containment") {
const [action] = await tx.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, release.companyId), eq(issueRecoveryActions.sourceIssueId, release.issueId),
eq(issueRecoveryActions.id, release.actionId))).for("update");
const [expiry] = await tx.select({ epoch: sql<string | null>`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
const authority = action?.evidence.controllerAuthority;
const recordedDigest = authority && typeof authority === "object" && !Array.isArray(authority)
&& (authority as Record<string, unknown>).schema === "paperclip.native-controller-authority.v1"
? (authority as Record<string, unknown>).digest : null;
if (!["completed", "not_performed", "mixed"].includes(release.actionOutcome)
|| release.outcomeEvidence.trim().length < 20 || release.outcomeEvidence.length > 12_000
|| !task || !action || action.kind !== "active_run_watchdog" || action.ownerType !== "board"
|| !["active", "escalated"].includes(action.status) || action.cause !== "native_restart_recovery_blocked"
|| action.fingerprint !== `native-controller-space:${run.id}:${release.authorityDigest}` || action.evidence.runId !== run.id
|| action.evidence.reason !== "controller_process_space_unverified"
|| recordedDigest !== release.authorityDigest || !expiry
|| nativeControllerAuthorityDigest(coordinator, expiry.epoch) !== release.authorityDigest
|| coordinator.companyId !== release.companyId || coordinator.issueId !== release.issueId
|| action.returnOwnerAgentId !== run.agentId || task.assigneeAgentId !== run.agentId
|| ["done", "cancelled"].includes(task.status)
|| (task.executionRunId !== run.id && !(task.executionRunId === null && RELEASABLE_RUN_STATUSES.includes(run.status)))
|| (task.checkoutRunId !== null && task.checkoutRunId !== run.id)
|| !coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > now
|| coordinator.leaseOwner?.startsWith("native-finalizer:") || coordinator.leaseOwner?.startsWith("native-cleanup:")
|| coordinator.failureDetail?.successorRunId || isCurrentNativeControllerProcess(coordinator)
|| isNativeRunnerOwnershipHeld(run) || (run.controllerLeaseExpiresAt !== null && run.controllerLeaseExpiresAt > now)
|| run.runnerProfileJson?.nativeWorkspaceSync != null || run.runnerProfileJson?.remoteWorkspace != null
|| run.runnerProfileJson?.nativeControllerRecovery != null
|| run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null
|| adapterExecutionControls.has(run.id) || activeNativeExecutionRunIds.has(run.id)) return false;
if (task.executionRunId === null) {
// Issue lock serializes admission. A cleared failure pointer does not
// authorize containment of an older/replaced source.
const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId),
or(eq(heartbeatRuns.nativeIssueId, coordinator.issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${coordinator.issueId}`,
sql`exists (select 1 from ${nativeRunFinalizations} newer_owner
where newer_owner.run_id = ${heartbeatRuns.id} and newer_owner.company_id = ${run.companyId}
and newer_owner.issue_id = ${coordinator.issueId})`),
ne(heartbeatRuns.id, run.id),
sql`${heartbeatRuns.createdAt} >= (select source.created_at from ${heartbeatRuns} source
where source.id = ${run.id} and source.company_id = ${run.companyId})`,
)).limit(1);
if (newer) return false;
}
const [heldElsewhere] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, run.id), or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1);
const [work] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, run.companyId), eq(workspaceOperations.heartbeatRunId, run.id),
inArray(workspaceOperations.status, ["queued", "running"]))).limit(1);
if (heldElsewhere || work) return false;
containedExpiryEpoch = expiry.epoch;
// The explicit original-host controller+provider attestation replaces a
// meaningless local PID observation only in this exact authority branch.
} else if (release.kind === "expired_owner") {
if (!coordinator.leaseExpiresAt || coordinator.controllerGeneration !== release.controllerGeneration
|| run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null
|| coordinator.leaseExpiresAt > now) return false;
// An expired lease does not make foreign PID numbers local evidence.
// Keep this boot's exact owner semantics; every other owner needs the
// same known kernel/PID space before either recorded-process probe.
if (!isCurrentNativeControllerProcess(coordinator)) {
const current = currentProcessSpaceHash();
if (current === "unknown" || coordinator.controllerHostHash !== current) {
if (!coordinator.leaseOwner?.startsWith("native-finalizer:") && !coordinator.leaseOwner?.startsWith("native-cleanup:")) {
const [expiry] = await tx.select({ epoch: sql<string | null>`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
const { remoteProcess } = await readNativeRemoteExecutionOwnership(tx, { run, issueId: coordinator.issueId });
await surfaceNativeControllerNamespaceAttention(tx, { run, coordinator, expiryEpoch: expiry?.epoch ?? null, remoteProcess });
}
return false;
}
}
const [heldElsewhere] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, run.id), or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1);
if (heldElsewhere || await runProcessMayBeLive(run, now, fallbacks)
@@ -159,7 +267,8 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis
eq(nativeRunResults.schemaStatus, "accepted"))).limit(1);
if (accepted) return false;
const failureCode = coordinator.failureCode ?? run.errorCode ?? FALLBACK_FAILURE_CODE;
const failureCode = release.kind === "operator_containment" ? "native_restart_recovery_blocked"
: coordinator.failureCode ?? run.errorCode ?? FALLBACK_FAILURE_CODE;
const scope = { companyId: run.companyId, agentId: run.agentId, runId: run.id, issueId: coordinator.issueId };
await notifyExecutionRunWork(tx, scope);
await notifyNativeFinalizationWork(tx, scope);
@@ -177,21 +286,49 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis
...coordinator.failureDetail,
...(run.error ? { message: run.error.slice(0, 2_000) } : {}),
originalFailureCode: run.errorCode ?? failureCode,
releasedOwner: { reason: release.kind, leaseOwner: release.leaseOwner, attempt: release.attempt },
releasedOwner: release.kind === "operator_containment"
? { schema: "paperclip.native-controller-containment.v1", reason: release.kind, actionId: release.actionId,
controllerGeneration: coordinator.controllerGeneration, controllerBootId: coordinator.controllerBootId,
controllerHostHash: coordinator.controllerHostHash, authorityDigest: release.authorityDigest, attempt: release.attempt,
actorId: release.actorId, recordedAt: now.toISOString(), controllerStopped: true, providerStopped: true,
actionOutcome: release.actionOutcome, outcomeEvidence: release.outcomeEvidence,
priorProcess: { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt?.toISOString() ?? null } }
: { reason: release.kind, leaseOwner: release.leaseOwner, attempt: release.attempt },
recoveryOwner: { kind: "board" },
nextAction: NEXT_ACTION,
},
updatedAt: now,
}).where(and(
eq(nativeRunFinalizations.runId, run.id),
eq(nativeRunFinalizations.phase, "observed"),
eq(nativeRunFinalizations.phase, release.kind === "operator_containment" ? coordinator.phase : "observed"),
isNull(nativeRunFinalizations.resultId),
eq(nativeRunFinalizations.leaseOwner, release.leaseOwner),
eq(nativeRunFinalizations.attempt, release.attempt),
release.kind === "operator_containment" ? and(
eq(nativeRunFinalizations.controllerGeneration, release.controllerGeneration),
sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text = ${containedExpiryEpoch}`,
coordinator.controllerHostHash === null ? isNull(nativeRunFinalizations.controllerHostHash)
: eq(nativeRunFinalizations.controllerHostHash, coordinator.controllerHostHash),
) : undefined,
)).returning({ runId: nativeRunFinalizations.runId });
if (!updated) return false;
await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, updatedAt: now })
.where(eq(heartbeatRuns.id, run.id));
const [settledRun] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, updatedAt: now,
...(release.kind === "operator_containment" ? { status: "failed", finishedAt: now,
errorCode: failureCode, error: "The original controller and provider were contained by a board operator. Review the recorded action outcomes before explicitly retrying.",
processPid: null, processGroupId: null, processStartedAt: null, executionControlDeadlineAt: null } : {}),
}).where(eq(heartbeatRuns.id, run.id)).returning();
if (release.kind === "operator_containment" && settledRun) {
await preserveNativeFailedWorkspace(tx, settledRun);
await settleNativeTerminalWake(tx, settledRun);
await notifyNativeStatusReassessment(tx, scope);
// Durable activity only: this call may be a savepoint of board resolution,
// so publishing a live event here would precede the outer commit.
await persistActivity(tx, { companyId: run.companyId, actorType: "user", actorId: release.actorId,
action: "native.controller_containment_recorded", entityType: "heartbeat_run", entityId: run.id,
agentId: run.agentId, runId: run.id, details: { recoveryActionId: release.actionId,
authorityDigest: release.authorityDigest, replay: "not_authorized" } });
return true;
}
for (const use of fallbacks) {
await recordLegacyHostFallbackProof(tx, run, { allowed: true, reason: use.reason, legacyHostFallback: true }, use.source);
}
@@ -253,10 +390,10 @@ export async function releaseAbandonedNativeOwners(db: Db, now = new Date(), pag
.where(and(abandonedOwnerCondition(), lte(nativeRunFinalizations.leaseExpiresAt, now),
after ? gt(nativeRunFinalizations.runId, after) : undefined))
.orderBy(asc(nativeRunFinalizations.runId))
.limit(pageSize);
let candidates = await page(abandonedOwnerReleaseCursor);
if (candidates.length === 0 && abandonedOwnerReleaseCursor !== null) candidates = await page(null);
abandonedOwnerReleaseCursor = candidates.length === pageSize ? candidates[candidates.length - 1]!.runId : null;
.limit(pageSize + 1);
const rows = await page(abandonedOwnerReleaseCursor);
const candidates = rows.slice(0, pageSize);
abandonedOwnerReleaseCursor = rows.length > pageSize ? candidates[candidates.length - 1]!.runId : null;
let released = 0;
for (const candidate of candidates) {
try {
@@ -270,3 +407,105 @@ export async function releaseAbandonedNativeOwners(db: Db, now = new Date(), pag
}
return released;
}
/** This entry is called only by the board-authorized reconciliation owner.
* Its opaque authority is reloaded and checked under the normal settlement
* locks; the operator receipt never becomes a fabricated provider result. */
export async function settleOperatorContainedNativeOwner(db: Db, input: {
companyId: string; issueId: string; runId: string; actionId: string;
authorityDigest: string; actorId: string;
actionOutcome: ExecutionReconciliation["actionOutcome"]; outcomeEvidence: string;
}): Promise<boolean> {
const [owner] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
eq(nativeRunFinalizations.runId, input.runId)));
if (!owner?.leaseOwner) return false;
return settleFailedRunOwner(db, { kind: "operator_containment", ...input,
leaseOwner: owner.leaseOwner, attempt: owner.attempt, controllerGeneration: owner.controllerGeneration }, new Date());
}
/** Committed operator evidence is a separate authority from an observed local
* stop. It permits only a new, independently authorized Retry of this source;
* it never proves a PID was observed dead or admits provider replay. */
export async function hasCommittedNativeControllerContainment(
db: Pick<Db, "select">,
run: Run,
issueId: string,
): Promise<boolean> {
if (run.runtimeMode !== "native" || run.nativeIssueId !== issueId || run.status !== "failed"
|| run.nativePhase !== "terminal_failure" || !run.finishedAt
|| run.errorCode !== "native_restart_recovery_blocked"
|| run.processPid !== null || run.processGroupId !== null || run.processStartedAt !== null
|| adapterExecutionControls.has(run.id) || activeNativeExecutionRunIds.has(run.id)
|| (run.controllerLeaseExpiresAt !== null && run.controllerLeaseExpiresAt > new Date())
|| run.runnerProfileJson?.nativeWorkspaceSync != null || run.runnerProfileJson?.remoteWorkspace != null
|| run.runnerProfileJson?.nativeControllerRecovery != null
|| run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null) return false;
const [task] = await db.select().from(issues).where(and(eq(issues.companyId, run.companyId),
eq(issues.id, issueId))).for("share", { noWait: true });
if (!task || task.assigneeAgentId !== run.agentId || ["done", "cancelled"].includes(task.status)) return false;
// This is a committed source-containment fact, not admission authority.
// Existing chat/queue owners separately bind current checkout/execution and
// the exact admitted successor; a legitimate successor does not undo the
// source's original-host containment.
const [owner] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.issueId, issueId),
eq(nativeRunFinalizations.runId, run.id))).for("share", { noWait: true });
if (!owner || owner.phase !== "terminal_failure" || owner.recoveryState !== "blocked"
|| owner.failureCode !== "native_restart_recovery_blocked" || owner.leaseOwner || owner.leaseExpiresAt
|| owner.nextAttemptAt || owner.resultId || owner.assessmentId || owner.decisionId
|| owner.failureDetail?.successorRunId) return false;
const value = owner.failureDetail?.releasedOwner;
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const receipt = value as Record<string, unknown>;
if (receipt.schema !== "paperclip.native-controller-containment.v1" || receipt.reason !== "operator_containment"
|| typeof receipt.actionId !== "string" || typeof receipt.authorityDigest !== "string"
|| !/^[a-f0-9]{64}$/.test(receipt.authorityDigest)
|| receipt.attempt !== owner.attempt || receipt.controllerGeneration !== owner.controllerGeneration
|| receipt.controllerBootId !== owner.controllerBootId || receipt.controllerHostHash !== owner.controllerHostHash
|| receipt.controllerStopped !== true || receipt.providerStopped !== true
|| !["completed", "not_performed", "mixed"].includes(String(receipt.actionOutcome))
|| typeof receipt.outcomeEvidence !== "string" || receipt.outcomeEvidence.trim().length < 20
|| receipt.outcomeEvidence.length > 12_000 || typeof receipt.actorId !== "string" || !receipt.actorId
|| typeof receipt.recordedAt !== "string" || !Number.isFinite(Date.parse(receipt.recordedAt))) return false;
const [action] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
eq(issueRecoveryActions.id, receipt.actionId))).for("share", { noWait: true });
const authority = action?.evidence.controllerAuthority;
const decision = action?.evidence.executionReconciliation;
if (!action || action.kind !== "active_run_watchdog" || action.ownerType !== "board"
|| action.status !== "resolved" || action.outcome !== "restored" || !action.resolvedAt
|| action.returnOwnerAgentId !== run.agentId || action.cause !== "native_restart_recovery_blocked"
|| action.fingerprint !== `native-controller-space:${run.id}:${receipt.authorityDigest}`
|| action.evidence.runId !== run.id || action.evidence.reason !== "controller_process_space_unverified"
|| action.evidence.continuationDelivery !== "not_requested"
|| !authority || typeof authority !== "object" || Array.isArray(authority)
|| !("schema" in authority) || authority.schema !== "paperclip.native-controller-authority.v1"
|| !("digest" in authority) || authority.digest !== receipt.authorityDigest
|| !decision || typeof decision !== "object" || Array.isArray(decision)) return false;
const reconciliation = decision as Record<string, unknown>;
if (reconciliation.runId !== run.id || reconciliation.controllerStopped !== true || reconciliation.providerStopped !== true
|| reconciliation.controllerAuthorityDigest !== receipt.authorityDigest || reconciliation.actorId !== receipt.actorId
|| reconciliation.actionOutcome !== receipt.actionOutcome || reconciliation.outcomeEvidence !== receipt.outcomeEvidence) return false;
const [audit] = await db.select({ id: activityLog.id }).from(activityLog).where(and(
eq(activityLog.companyId, run.companyId), eq(activityLog.entityType, "heartbeat_run"),
eq(activityLog.entityId, run.id), eq(activityLog.actorType, "user"), eq(activityLog.actorId, receipt.actorId),
eq(activityLog.action, "native.controller_containment_recorded"),
sql`${activityLog.details}->>'recoveryActionId' = ${action.id}`,
sql`${activityLog.details}->>'authorityDigest' = ${receipt.authorityDigest}`,
sql`${activityLog.details}->>'replay' = 'not_authorized'`)).limit(1);
if (!audit) return false;
const [blocked] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id),
or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1);
const [accepted] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id),
eq(nativeRunResults.schemaStatus, "accepted"))).limit(1);
const [work] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, run.companyId), eq(workspaceOperations.heartbeatRunId, run.id),
inArray(workspaceOperations.status, ["queued", "running"]))).limit(1);
const [remote] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
ne(environmentLeases.provider, "local"))).limit(1);
return !blocked && !accepted && !work && !remote;
}
@@ -11,17 +11,16 @@ type ControllerRow = Pick<typeof nativeRunFinalizations.$inferSelect, "controlle
| "controllerProcessStartedAt"> & {
controllerHostHash?: string | null;
controllerGeneration?: number | null;
leaseOwner?: string | null;
};
/** NRF records the controller's boot, PID, start time and, since #19868, its
* PID space (`controller_host_hash`). Rows saved before that keep the
* one-release local proof on single-host servers (`processSpaceUnrecorded`,
* recorded as `legacy_host_unknown_dead_pid_proof`, #19883). */
/** Native controller ownership requires its recorded kernel/PID space before
* a local PID observation. Independent durable stop evidence remains valid. */
export function nativeControllerOwnerStamp(row: ControllerRow, scope: string): LocalOwnerStamp {
return {
kind: "local", scope, generation: row.controllerGeneration ?? null, host: null,
kind: "local", scope, generation: row.controllerGeneration ?? null, fenceToken: row.leaseOwner, host: null,
processSpace: row.controllerHostHash ?? null,
processSpaceUnrecorded: row.controllerHostHash == null,
processSpaceUnrecorded: row.controllerHostHash == null, requireRecordedProcessSpace: true,
bootId: row.controllerBootId, pid: row.controllerPid,
processStartedAt: row.controllerProcessStartedAt?.toISOString() ?? null, processGroupId: null,
};
@@ -34,6 +34,7 @@ import {
heartbeatRunEvents,
heartbeatRuns,
issues,
issueRecoveryActions,
nativeRunFinalizations,
} from "@paperclipai/db";
import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
@@ -49,7 +50,7 @@ import { isUniqueViolation } from "../../db-errors.js";
import { logger } from "../../middleware/logger.js";
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js";
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
import { nativeControllerAuthorityDigest, isNativeRunnerOwnershipHeld, readNativeRemoteExecutionOwnership } from "./native-runner-ownership.js";
import { nativeRestartContinuationInFlight, nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
import { retireSupersededNativeRetry } from "./native-superseded-retry.js";
import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js";
@@ -415,6 +416,7 @@ export async function evaluateNativeControllerTakeover(input: {
isProcessAlive?: (pid: number) => boolean;
readProcessStartedAt?: (pid: number) => Promise<Date | null>;
requireStoppedController?: boolean;
requireRecordedProcessSpace?: boolean;
remoteLeaseFenced?: boolean;
}): Promise<NativeControllerTakeoverDecision> {
const { owner, now } = input;
@@ -474,12 +476,13 @@ export async function evaluateNativeControllerTakeover(input: {
pidExists: input.isProcessAlive ?? base.pidExists,
readStartedAt: input.readProcessStartedAt ?? base.readStartedAt,
};
const needsBootProof = owner.controllerHostHash != null
&& (owner.controllerHostHash !== probe.processSpace || owner.controllerHostHash === UNKNOWN_PROCESS_SPACE);
const needsBootProof = input.requireRecordedProcessSpace || (owner.controllerHostHash != null
&& (owner.controllerHostHash !== probe.processSpace || owner.controllerHostHash === UNKNOWN_PROCESS_SPACE));
const proof = await proveOwnerGone({
kind: "local", scope: "native_controller", generation: null, host: null,
processSpace: recordsHost ? owner.controllerHostHash ?? null : probe.processSpace,
processSpaceUnrecorded: recordsHost && owner.controllerHostHash == null,
...(input.requireRecordedProcessSpace ? { requireRecordedProcessSpace: true } : {}),
bootId: needsBootProof ? owner.controllerBootId ?? null : null, touchedAt: owner.updatedAt?.toISOString() ?? null,
pid: priorPid, processStartedAt: owner.controllerProcessStartedAt?.toISOString() ?? null,
processGroupId: null,
@@ -927,6 +930,47 @@ function appendBoundedRecoveryHistory(entry: Record<string, unknown>) {
)`;
}
/** Shared by restart inspection and failed-owner cleanup under their existing
* issue/coordinator/run locks. Publishing attention never grants a new turn. */
export async function surfaceNativeControllerNamespaceAttention(db: Db, input: {
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "agentId">;
coordinator: typeof nativeRunFinalizations.$inferSelect;
expiryEpoch: string | null;
remoteProcess: boolean;
}): Promise<void> {
const reason = input.remoteProcess ? "remote_controller_process_space_unverified" : "controller_process_space_unverified";
const attention = issueRecoveryActionService(db);
const digest = nativeControllerAuthorityDigest(input.coordinator, input.expiryEpoch);
const fingerprint = `native-controller-space:${input.run.id}:${digest}`;
const [existing] = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, input.run.companyId), eq(issueRecoveryActions.sourceIssueId, input.coordinator.issueId),
inArray(issueRecoveryActions.status, ["active", "escalated"]),
)).for("update").limit(1);
// One incident per exact authority. A repeat neither spends a retry
// attempt nor emits a new wake. Renewed/replaced authority gets a new
// incident, preserving the prior confirmation/history. An unrelated
// active recovery owner is never overwritten by this observation.
const sameSource = existing?.cause === "native_restart_recovery_blocked"
&& existing.evidence.runId === input.run.id
&& ["controller_process_space_unverified", "remote_controller_process_space_unverified"].includes(String(existing.evidence.reason));
if (!existing || (sameSource && existing.fingerprint !== fingerprint)) {
await attention.upsertSourceScoped({
companyId: input.run.companyId, sourceIssueId: input.coordinator.issueId,
kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: input.run.agentId,
cause: "native_restart_recovery_blocked", fingerprint,
evidence: { runId: input.run.id, reason, executorStopEvidence: "unverified",
...(!input.remoteProcess ? { controllerAuthority: { schema: "paperclip.native-controller-authority.v1",
digest, leaseExpiresAt: input.coordinator.leaseExpiresAt!.toISOString() } } : {}) },
nextAction: input.remoteProcess
? "Use the task's ordinary Stop action and its environment provider's stop/cleanup recovery. Wait for that exact controller/provider acknowledgement. Local PID checks and local containment confirmation cannot authorize remote recovery."
: "Stop the original controller and provider on their original host. After the saved lease expires, a board operator must record both stops and review action outcomes. Recording containment does not retry the task.",
createOnly: !existing, supersedeOnIdentityChange: true,
attemptCount: 0, maxAttempts: null, timeoutAt: null, lastAttemptAt: null,
wakePolicy: null, monitorPolicy: null,
});
}
}
export async function claimNativeRestartRecoveries(input: {
db: Db;
controller?: NativeControllerIdentity;
@@ -948,6 +992,7 @@ export async function claimNativeRestartRecoveries(input: {
const now = input.now ?? new Date();
const controller =
input.controller ?? (await currentNativeControllerIdentity());
const controllerHostHash = currentProcessSpaceHash();
const candidateQuery = input.db
.select({
runId: heartbeatRuns.id,
@@ -1022,6 +1067,7 @@ export async function claimNativeRestartRecoveries(input: {
.select({
run: heartbeatRuns,
coordinator: nativeRunFinalizations,
controllerLeaseExpiryEpoch: sql<string | null>`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text`,
issueExecutionRunId: issues.executionRunId,
issueCheckoutRunId: issues.checkoutRunId,
issueAssigneeAgentId: issues.assigneeAgentId,
@@ -1075,12 +1121,8 @@ export async function claimNativeRestartRecoveries(input: {
const storedRemoteWorkspace = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace);
const remoteWorkspace = readNativeWorkspaceSyncReference(profile.nativeWorkspaceSync)
?? (storedRemoteWorkspace.success ? storedRemoteWorkspace.data : null);
const [pluginTask] = await tx.select({ id: environmentLeases.id,
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
}).from(environmentLeases).where(and(
eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.issueId, issue.id),
eq(environmentLeases.heartbeatRunId, row.run.id), sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
)).orderBy(desc(environmentLeases.createdAt)).limit(1);
const { pluginTask, remoteProcess } = await readNativeRemoteExecutionOwnership(tx as unknown as Db,
{ run: row.run, issueId: issue.id });
const failedRemoteRetry = row.run.status === "failed" && row.coordinator.phase === "retryable_failure"
&& row.coordinator.failureDetail?.recoveryMode === "exact_checkpoint_resume" && (remoteWorkspace !== null || pluginTask !== undefined);
// A same-run bootstrap retry cannot take back an issue lock that moved
@@ -1200,22 +1242,51 @@ export async function claimNativeRestartRecoveries(input: {
return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const;
}
const remoteSandbox = profile.nativeWorkspaceSync != null || profile.remoteWorkspace != null;
const remoteProcess = remoteSandbox || !!pluginTask;
const durableAuthority = (remoteWorkspace || pluginTask) && await hasNativeRunnerCheckpoint(tx, {
companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id,
});
const takeover = await evaluateNativeControllerTakeover({
owner: row.coordinator,
now,
remoteLeaseFenced: Boolean(durableAuthority),
coordinatedPreviousController:
input.coordinatedPreviousController ?? null,
});
// Lease expiry alone does not prove a local in-flight owner stopped.
// Only this boot's own PID or the same known process space can make a
// local PID observation meaningful. Durable remote leases keep their
// existing released/expired authority instead of using local probes.
const processSpaceUnverified = !durableAuthority
&& Boolean(row.coordinator.leaseOwner && row.coordinator.leaseExpiresAt)
&& !isCurrentNativeControllerProcess(row.coordinator)
&& (controllerHostHash === "unknown" || row.coordinator.controllerHostHash !== controllerHostHash);
const nonExecutionOwner = row.coordinator.leaseOwner?.startsWith("native-finalizer:")
|| row.coordinator.leaseOwner?.startsWith("native-cleanup:");
const evaluatedTakeover = nonExecutionOwner
? { allowed: false, reason: "native_finalization_or_cleanup_owner" }
: await evaluateNativeControllerTakeover({
owner: row.coordinator,
now,
requireRecordedProcessSpace: processSpaceUnverified,
remoteLeaseFenced: Boolean(durableAuthority),
coordinatedPreviousController:
input.coordinatedPreviousController ?? null,
});
const independentControllerWait = !evaluatedTakeover.allowed
&& ["controller_lease_held", "controller_lease_grace"].includes(evaluatedTakeover.reason);
// Controller death does not prove a foreign local Runner or provider
// stopped. Remote execution never uses local PID probes and still owes
// its existing checkpoint or never-launched proof below.
const independentlyStoppedRemoteController = remoteProcess && evaluatedTakeover.allowed
&& ["controller_stopped", "controller_fenced"].includes(evaluatedTakeover.reason);
const namespaceAttentionNeeded = processSpaceUnverified && !nonExecutionOwner
&& !independentControllerWait && !independentlyStoppedRemoteController;
const takeover = namespaceAttentionNeeded
? { allowed: false, reason: remoteProcess ? "remote_controller_process_space_unverified" : "controller_process_space_unverified" }
: evaluatedTakeover;
await recordLegacyHostFallbackProof(tx as unknown as Db, row.run, takeover, "native_restart_recovery");
if (!takeover.allowed && ["controller_host_unrecorded", "controller_host_unprovable"].includes(takeover.reason)) {
await requestNativeControllerStopConfirmation(tx as unknown as Db, row.coordinator);
}
if (!takeover.allowed) {
if (namespaceAttentionNeeded) {
await surfaceNativeControllerNamespaceAttention(tx as unknown as Db, {
run: row.run, coordinator: row.coordinator, expiryEpoch: row.controllerLeaseExpiryEpoch, remoteProcess,
});
}
const previous = row.coordinator.recoveryHistory.at(-1);
const waitAlreadyRecorded = previous?.disposition === "awaiting_evidence" && previous.reason === takeover.reason;
const event = historyEntry({
@@ -1697,8 +1768,8 @@ export async function claimNativeRestartRecoveries(input: {
// The owner row is locked: grant the full lease at this write.
leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`,
controllerBootId: controller.bootId,
controllerHostHash,
controllerPid: controller.pid,
controllerHostHash: controller.hostHash ?? null,
controllerProcessStartedAt: controller.processStartedAt,
controllerGeneration: generation,
recoveryState:
@@ -1739,6 +1810,17 @@ export async function claimNativeRestartRecoveries(input: {
} as const;
}
// A successful ordinary claim has now reconciled this exact earlier
// authority. Retire only its namespace attention; stale/other incidents
// remain owned and no operator confirmation is fabricated.
await issueRecoveryActionService(tx as unknown as Db).resolveActiveForIssue({
companyId: row.run.companyId, sourceIssueId: row.coordinator.issueId,
kind: "active_run_watchdog", cause: "native_restart_recovery_blocked",
fingerprint: `native-controller-space:${row.run.id}:${nativeControllerAuthorityDigest(row.coordinator, row.controllerLeaseExpiryEpoch)}`,
status: "resolved", outcome: "restored",
resolutionNote: "Normal restart recovery verified the previous authority and committed its exact replacement claim.",
}, tx);
if (
claimKind !== "reattach_existing_runner" &&
claimKind !== "reattach_remote_runner"
@@ -1,8 +1,40 @@
import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
import { createHash } from "node:crypto";
import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
import { readSystemRecovery } from "../system-recovery-state.js";
type NativeRemoteExecutionScope = {
run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "runnerProfileJson">;
issueId: string;
};
export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs: readonly NativeRemoteExecutionScope[]) {
const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`;
const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId,
issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId,
metadata: sql<Record<string, unknown>>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`,
}).from(environmentLeases).where(and(
or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId),
eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.heartbeatRunId, input.run.id)))),
sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`,
)).orderBy(desc(environmentLeases.createdAt)) : [];
const byScope = new Map<string, typeof leases[number]>();
for (const lease of leases) {
const scope = key(lease.companyId, lease.issueId!, lease.runId!);
if (!byScope.has(scope)) byScope.set(scope, lease);
}
return inputs.map(input => {
const pluginTask = byScope.get(key(input.run.companyId, input.issueId, input.run.id));
return { pluginTask, remoteProcess: input.run.runnerProfileJson?.nativeWorkspaceSync != null
|| input.run.runnerProfileJson?.remoteWorkspace != null || !!pluginTask };
});
}
export async function readNativeRemoteExecutionOwnership(db: Db, input: NativeRemoteExecutionScope) {
return (await readNativeRemoteExecutionOwnershipForRuns(db, [input]))[0]!;
}
export const nativeManualAdmissionOwnerColumns = {
runId: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
issueId: nativeRunFinalizations.issueId, phase: nativeRunFinalizations.phase,
@@ -79,3 +111,16 @@ export function nativeRunnerOwnershipNotHeldCondition() {
), false
)`;
}
/** Opaque binding for an operator's exact observed controller authority. Never
* expose the token itself. expiryEpoch must come from Postgres numeric epoch
* text, which is timezone independent and retains timestamp microseconds. */
export function nativeControllerAuthorityDigest(owner: Pick<typeof nativeRunFinalizations.$inferSelect,
"companyId" | "issueId" | "runId" | "leaseOwner" | "controllerGeneration" | "controllerBootId" |
"controllerHostHash" | "controllerPid" | "controllerProcessStartedAt">, expiryEpoch: string | null): string {
return createHash("sha256").update(JSON.stringify([
"paperclip.native-controller-authority.v1", owner.companyId, owner.issueId, owner.runId,
owner.leaseOwner, owner.controllerGeneration, owner.controllerBootId, owner.controllerHostHash,
owner.controllerPid, owner.controllerProcessStartedAt?.toISOString() ?? null, expiryEpoch,
])).digest("hex");
}
@@ -1,3 +1,4 @@
import { currentControllerProcess, currentProcessSpaceHash } from "../controller-process-identity.js";
import { renewNativeSessionExecutionLease, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js";
import { NativeControllerRecoveryWaitingError } from "./native-controller-recovery.js";
@@ -810,6 +811,15 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
}
}, 360_000);
// These historical controllers belong to the same kernel/PID namespace as
// the real local processes this fixture starts. Other-space/legacy NULL
// authority has separate explicit negatives; do not invent a constant hash.
async function knownFixtureControllerHostHash() {
const current = await currentControllerProcess();
expect(current.hostHash, "Same-host recovery fixture requires actual known process space").not.toBe("unknown");
return current.hostHash;
}
async function persistRestartEvidence(input: {
fixture: Awaited<ReturnType<typeof seedRun>>;
runnerPid: number;
@@ -863,6 +873,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
leaseOwner: "dead-controller",
leaseExpiresAt: new Date(now.getTime() + 60_000),
controllerBootId: "dead-controller-boot",
controllerHostHash: await knownFixtureControllerHostHash(),
controllerPid: 2_000_000_000,
controllerProcessStartedAt: new Date("2026-09-04T11:00:00.000Z"),
controllerGeneration: 1,
@@ -895,6 +906,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
leaseOwner: "dead-controller",
leaseExpiresAt: new Date(now.getTime() + 60_000),
controllerBootId: "dead-controller-boot",
controllerHostHash: await knownFixtureControllerHostHash(),
controllerPid: 2_000_000_003,
controllerProcessStartedAt: new Date("2026-09-04T11:00:00.000Z"),
controllerGeneration: 1,
@@ -2238,6 +2250,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
leaseOwner: "dead-controller",
leaseExpiresAt: new Date(now.getTime() + 60_000),
controllerBootId: "dead-controller-boot",
controllerHostHash: await knownFixtureControllerHostHash(),
controllerPid: 2_000_000_002,
controllerProcessStartedAt: new Date("2026-09-04T10:00:00.000Z"),
controllerGeneration: 4,
@@ -2280,6 +2293,73 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
]);
});
it.each(["stopped", "fenced"] as const)("keeps foreign local Runner and provider ownership after its controller is %s", async state => {
const fixture = await seedRun(`FOREIGN-LOCAL-${state}-${randomUUID()}`);
const oldBoot = randomUUID(), foreignSpace = `foreign:${currentProcessSpaceHash()}`;
const runnerPid = 2_000_000_001, runnerGroup = 2_000_000_002, providerPid = 2_000_000_003;
const controllerPid = 2_000_000_004, startedAt = new Date(Date.now() - 600_000);
// Historical in-flight local execution on the other process space. A
// controller-instance stop/fence says nothing about these saved children.
await fixture.db.update(heartbeatRuns).set({ processPid: runnerPid, processGroupId: runnerGroup,
processStartedAt: startedAt, processHostHash: foreignSpace,
runnerProfileJson: { ...fixture.preparedProfile, sessionCheckpoint: {
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId,
sessionId: fixture.native.normalizedSessionId },
providerSessionId: "foreign-provider-session", activeTurnId: "foreign-in-flight-turn",
process: { providerPid, providerProcessStartedAt: startedAt.toISOString() },
} },
}).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: oldBoot, controllerPid,
controllerProcessStartedAt: startedAt, controllerHostHash: foreignSpace,
leaseOwner: `foreign-in-flight:${randomUUID()}`, leaseExpiresAt: new Date(0), attempt: 1, controllerGeneration: 1,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
await fixture.db.insert(controllerInstances).values({ bootId: oldBoot, processSpace: foreignSpace,
pid: controllerPid, leaseExpiresAt: new Date(Date.now() - 1_000),
...(state === "stopped" ? { stoppedAt: new Date() } : { fencedAt: new Date() }),
});
registerControllerInstanceDb(fixture.db);
const [before] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
const probes: number[] = [], recordedPids = new Set([runnerPid, runnerGroup, providerPid, controllerPid]);
const kill = process.kill.bind(process);
const pidProbe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
if (signal === 0 && recordedPids.has(Math.abs(pid))) {
probes.push(pid);
throw Object.assign(new Error("Absent local PID is not foreign execution stop proof"), { code: "ESRCH" });
}
return kill(pid, signal);
});
const claimed = vi.fn(), stamped = vi.fn(), unsubscribe = onControllerStamp(stamped);
try {
const dispositions = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId], onClaim: claimed });
const [after] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
const stopped = await hasNativeLocalProcessStop(fixture.db, companyId, fixture.runId);
expect(probes).toEqual([]);
expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: fixture.runId,
reason: "controller_process_space_unverified" }]);
expect(claimed).not.toHaveBeenCalled();
expect(stamped).not.toHaveBeenCalled();
expect(stopped).toBe(false);
expect(after).toMatchObject({ phase: before.phase, attempt: before.attempt,
controllerGeneration: before.controllerGeneration, controllerBootId: before.controllerBootId,
controllerPid: before.controllerPid, controllerHostHash: before.controllerHostHash,
leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, resultId: null });
expect(run).toMatchObject({ status: "running", processPid: runnerPid, processGroupId: runnerGroup,
processStartedAt: startedAt, processHostHash: foreignSpace });
expect(run.runnerProfileJson).toEqual({ ...fixture.preparedProfile, sessionCheckpoint: {
identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId,
sessionId: fixture.native.normalizedSessionId },
providerSessionId: "foreign-provider-session", activeTurnId: "foreign-in-flight-turn",
process: { providerPid, providerProcessStartedAt: startedAt.toISOString() },
} });
const [attention] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId));
expect(attention).toMatchObject({ status: "active", cause: "native_restart_recovery_blocked",
evidence: { runId: fixture.runId, reason: "controller_process_space_unverified" } });
expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toEqual([]);
} finally { unsubscribe(); pidProbe.mockRestore(); resetControllerInstanceLeaseForTests(); }
});
it("recovers a preparation with no runner PID once its foreign controller stops", async () => {
const fixture = await seedRun("REMOTE-PREPARATION-OWNER");
const oldBoot = randomUUID();
@@ -2295,6 +2375,9 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
registerControllerInstanceDb(fixture.db);
const stamped = vi.fn(), unsubscribe = onControllerStamp(stamped);
try {
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [randomUUID()] })).toEqual([]);
expect(stamped).not.toHaveBeenCalled();
const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] });
expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "controller_lease_held", retryAt: expect.any(Date) });
@@ -2316,6 +2399,52 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
} finally { unsubscribe(); resetControllerInstanceLeaseForTests(); }
});
it.each(["grace", "fenced"] as const)("uses durable foreign controller %s evidence before classifying its namespace", async state => {
const fixture = await seedRun(`REMOTE-CONTROLLER-${state}-${randomUUID()}`);
const oldBoot = randomUUID();
await fixture.db.update(heartbeatRuns).set({
runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace() },
}).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: oldBoot, controllerPid: 7,
controllerProcessStartedAt: new Date(Date.now() - 600_000), controllerHostHash: "previous-pid-space",
leaseOwner: "previous-preparation", leaseExpiresAt: new Date(0), attempt: 1, controllerGeneration: 1,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
await fixture.db.insert(controllerInstances).values({ bootId: oldBoot, processSpace: "previous-pid-space",
pid: 7, leaseExpiresAt: new Date(Date.now() - 1_000),
...(state === "fenced" ? { fencedAt: new Date() } : {}),
});
registerControllerInstanceDb(fixture.db);
const originalKill = process.kill.bind(process);
const pidProbe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => {
if (pid === 7 && signal === 0) throw new Error("A foreign controller PID must not be probed");
return originalKill(pid, signal);
});
const stamped = vi.fn(), unsubscribe = onControllerStamp(stamped);
try {
const [before] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [disposition] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "hard", runIds: [fixture.runId] });
const [after] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
if (state === "grace") {
expect(disposition).toMatchObject({ kind: "awaiting_evidence", reason: "controller_lease_grace", retryAt: expect.any(Date) });
expect(stamped).not.toHaveBeenCalled();
expect(after).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt,
controllerBootId: before.controllerBootId, controllerHostHash: before.controllerHostHash,
controllerGeneration: before.controllerGeneration, attempt: before.attempt });
await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, restartKind: "hard", runIds: [fixture.runId] });
const [repeated] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(repeated.updatedAt).toEqual(after.updatedAt);
} else {
expect(disposition).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId });
expect(stamped).toHaveBeenCalledOnce();
expect(after).toMatchObject({ controllerGeneration: 2, attempt: 1, resultId: null });
}
expect((await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0])
.toMatchObject({ status: "running", processPid: null });
expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toEqual([]);
} finally { unsubscribe(); pidProbe.mockRestore(); resetControllerInstanceLeaseForTests(); }
});
it("recovers an unclaimed remote startup once without requiring a provider checkpoint", async () => {
const fixture = await seedRun("REMOTE-UNCLAIMED");
await fixture.db.update(heartbeatRuns).set({
@@ -2473,7 +2602,8 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
const bootId = randomUUID();
const leaseOwner = `${bootId}:1:${randomUUID()}`;
await fixture.db.update(nativeRunFinalizations).set({
attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: bootId, controllerPid: 2_000_000_001,
attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: bootId,
controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: 2_000_000_001,
controllerProcessStartedAt: new Date(Date.now() - 60_000), leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
}).where(eq(nativeRunFinalizations.runId, runId));
if (options.checkpoint !== false) {
@@ -2708,6 +2838,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
await fixture.db.update(nativeRunFinalizations).set({
attempt: options.attempt ?? 1, controllerGeneration: 1,
controllerBootId: options.liveController ? current.bootId : randomUUID(),
controllerHostHash: await knownFixtureControllerHostHash(),
controllerPid: options.liveController ? current.pid : 2_000_000_001,
controllerProcessStartedAt: options.liveController ? current.processStartedAt : new Date(Date.now() - 60_000),
leaseOwner: `previous:${randomUUID()}`,
@@ -1,3 +1,4 @@
import { currentControllerProcess } from "../controller-process-identity.js";
import { activeNativeExecutionRunIds } from "./native-active-runs.js";
import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js";
import { readIdleLocalWork, markIdleStartupComplete, markIdleIngressTracked } from "../idle-local-work.js";
@@ -9810,6 +9811,7 @@ describe("native session execution lease fencing", () => {
expect(profileWrites).toBe(2);
expect(nullWrites).toBe(0);
const claim = updates.find(update => update.table === nativeRunFinalizations && update.values.phase === "observed" && update.values.leaseOwner)!.values;
expect(claim.controllerHostHash).toBe((await currentControllerProcess()).hostHash);
expect(recordAdmission).toHaveBeenCalledExactlyOnceWith(db, { session, identity, waitForClaim: false,
admissionEpoch: (profile.nativeProviderPackAdmissionState as Record<string, unknown>).admissionEpoch,
claim: { leaseOwner: claim.leaseOwner, attempt: claim.attempt, controllerGeneration: claim.controllerGeneration, controllerBootId: claim.controllerBootId } });
@@ -1,3 +1,4 @@
import { currentControllerProcess } from "../controller-process-identity.js";
import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js";
import { isGovernancePauseStop } from "../governance-pause-stop.js";
import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js";
@@ -9902,6 +9903,7 @@ async function executePaperclipNativeSessionAttempt(
const leaseOwner =
input.leaseOwner ?? `${effectiveRunnerInstanceId}:${randomUUID()}`;
const controller = await currentNativeControllerIdentity();
const controllerHostHash = (await currentControllerProcess()).hostHash;
const executionLeaseTtlMs = input.runnerExecutionTarget?.kind === "remote"
? REMOTE_CONTROLLER_LEASE_MS : NATIVE_SESSION_EXECUTION_LEASE_TTL_MS;
let attempt: number;
@@ -10087,8 +10089,8 @@ async function executePaperclipNativeSessionAttempt(
// Start the full lease after claim locks and lineage reads finish.
leaseExpiresAt: sql`clock_timestamp() + ${executionLeaseTtlMs} * interval '1 millisecond'`,
controllerBootId: controller.bootId,
controllerHostHash,
controllerPid: controller.pid,
controllerHostHash: controller.hostHash ?? null,
controllerProcessStartedAt: controller.processStartedAt,
controllerGeneration: nextControllerGeneration,
failureCode: null,
+26
View File
@@ -45,6 +45,32 @@ describe("proveOwnerGone: local owners", () => {
expect(await proveOwnerGone(stamp(), probe())).toEqual({ verdict: "gone", reason: "pid_dead" });
});
it.each([null, "foreign-space", "unknown"])("does not probe a recorded-space-required owner in %s space", async processSpace => {
const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null);
const result = await proveOwnerGone(stamp({ processSpace, processSpaceUnrecorded: processSpace === null,
requireRecordedProcessSpace: true }), probe({ pidExists, readStartedAt }));
expect(result).toEqual({ verdict: "unprovable", reason: processSpace === null ? "process_space_unrecorded"
: processSpace === "unknown" ? "unknown_process_space" : "different_process_space" });
expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled();
});
it.each(["stopped", "fenced"] as const)("retains independent controller-instance %s evidence for an unrecorded Native space", async state => {
const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null);
const controllerInstance = vi.fn(async () => state);
expect(await proveOwnerGone(stamp({ processSpace: null, processSpaceUnrecorded: true,
requireRecordedProcessSpace: true }), probe({ pidExists, readStartedAt, controllerInstance })))
.toEqual({ verdict: "gone", reason: state === "stopped" ? "controller_stopped" : "controller_fenced" });
expect(controllerInstance).toHaveBeenCalledOnce();
expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled();
});
it("accepts audited operator stop evidence without probing a recorded-space-required owner", async () => {
const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null);
expect(await proveOwnerGone(stamp({ processSpace: "operator_attested", requireRecordedProcessSpace: true }),
probe({ pidExists, readStartedAt }))).toEqual({ verdict: "gone", reason: "operator_attested" });
expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled();
});
it("judges a row saved before its PID space locally for one release, on single-host servers only", async () => {
const legacy = stamp({ host: null, processSpace: null, processSpaceUnrecorded: true });
expect(await proveOwnerGone(legacy, probe())).toEqual({ verdict: "gone", reason: "pid_dead", legacyHostFallback: true });
+4 -2
View File
@@ -33,6 +33,7 @@ export type LocalOwnerStamp = {
/** Record kind or subsystem that wrote the stamp, for reports only. */
scope: string;
generation: number | null;
fenceToken?: string | null;
/** Hostname at stamp time. Informational; `processSpace` is the proof key. */
host: string | null;
/** Hash of host, kernel boot and PID namespace (`currentProcessSpaceHash`). */
@@ -58,6 +59,7 @@ export type LocalOwnerStamp = {
* An ephemeral (multi-host) server never judges them.
*/
processSpaceUnrecorded?: boolean;
requireRecordedProcessSpace?: boolean;
/** When the owned record was last written, if the caller knows (for the controller lease proof). */
touchedAt?: string | null;
};
@@ -247,7 +249,7 @@ export async function proveOwnerGone(
async function proveLocalOwnerGone(stamp: LocalOwnerStamp, probe: ProcessProbe): Promise<OwnerProof> {
const proof = await proveLocalProcessGone(stamp, probe);
if (proof.verdict !== "unprovable" || !CROSS_SPACE_REASONS.has(proof.reason) || !stamp.bootId || !probe.controllerInstance
if (proof.verdict !== "unprovable" || (!CROSS_SPACE_REASONS.has(proof.reason) && !(stamp.requireRecordedProcessSpace && proof.reason === "process_space_unrecorded")) || !stamp.bootId || !probe.controllerInstance
|| validPid(stamp.processGroupId ?? null)) return proof;
// Another host or PID space: the owner's server boot answers through its lease. Only a record the
// server process itself held qualifies; agent processes in a recorded group may outlive it.
@@ -272,7 +274,7 @@ async function proveLocalProcessGone(stamp: LocalOwnerStamp, probe: ProcessProbe
// fallback on single-host servers; an ephemeral server cannot assume the
// PID was recorded here, so a board operator decides.
const legacy = !stamp.processSpace && stamp.processSpaceUnrecorded === true;
if (legacy && ephemeralServerEnabled()) return { verdict: "unprovable", reason: "process_space_unrecorded" };
if (legacy && (stamp.requireRecordedProcessSpace || ephemeralServerEnabled())) return { verdict: "unprovable", reason: "process_space_unrecorded" };
if (!validPid(stamp.pid) || (!stamp.processSpace && !legacy)) return { verdict: "unprovable", reason: "missing_identity" };
if (legacy) {
const proof = await proveLocalOwnerGone({ ...stamp, processSpace: probe.processSpace, host: null,
@@ -33,7 +33,7 @@ import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js
import { OWNER_GONE_ATTESTATION_KEY, attestOwnerGoneByRecordVersion, requestOwnerGoneConfirmation } from "./owner-gone-attestation.js";
import {
requestNativeAcceptedResultLeaseOperatorAction,
settleNativeAcceptedResultLease,
settleNativeAcceptedResultLease, classifyNativeAcceptedResultLeaseOwners,
} from "../native-runtime/native-accepted-result-lease-settlement.js";
import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js";
import { settleAbandonedRemoteSnapshotSave, settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js";
@@ -656,6 +656,11 @@ const agentLifecycleLeaseKind: OwnableKind = {
},
};
const acceptedNativeResultLease = nrfKind("native_accepted_result_lease", "Accepted-result executor lease", "accepted_result",
and(isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, TERMINAL_PHASES),
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup,
sql`${nativeRunFinalizations.leaseExpiresAt} > now()`));
export const OWNABLE_KINDS: readonly OwnableKind[] = [
// Native runtime and Runner
nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership",
@@ -671,10 +676,13 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [
// exact lease so finalization runs now (RC-41).
// Outstanding leases only: an expired lease no longer holds the result,
// so it is not in progress for this kind (finalization reclaims it).
...nrfKind("native_accepted_result_lease", "Accepted-result executor lease", "accepted_result",
and(isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, TERMINAL_PHASES),
isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup,
sql`${nativeRunFinalizations.leaseExpiresAt} > now()`)),
...acceptedNativeResultLease,
listInProgress: async (db, limit, offset) => {
const listed = await acceptedNativeResultLease.listInProgress!(db, limit, offset);
return { ...listed, records: await classifyNativeAcceptedResultLeaseOwners(db, listed.records) };
},
listForRuns: async (db, runIds) => classifyNativeAcceptedResultLeaseOwners(db,
await acceptedNativeResultLease.listForRuns!(db, runIds)),
settle: settleNativeAcceptedResultLease,
onUnprovable: requestNativeAcceptedResultLeaseOperatorAction,
},
@@ -1,11 +1,16 @@
// @vitest-environment jsdom
import { canBoardConfirmOwnerStop } from "../lib/recovery-reconcile";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { createRoot } from "react-dom/client";
import { flushSync } from "react-dom";
import type { AnchorHTMLAttributes, ReactElement } from "react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { Agent, IssueRecoveryAction } from "@paperclipai/shared";
import { IssueRecoveryActionCard, deriveRecoveryCardState } from "./IssueRecoveryActionCard";
import { IssueRecoveryActionCard, NativeControllerContainmentRecovery, deriveRecoveryCardState } from "./IssueRecoveryActionCard";
const containmentApi = vi.hoisted(() => ({ resolveRecoveryAction: vi.fn() }));
vi.mock("../api/issues", () => ({ issuesApi: containmentApi }));
vi.mock("@/lib/router", () => ({
Link: ({ children, to, ...props }: AnchorHTMLAttributes<HTMLAnchorElement> & { to: string }) => (
@@ -1168,3 +1173,93 @@ it("requires owner-stop recovery even for board false-positive dispositions", ()
expect(node.querySelector("[data-testid='recovery-action-resolve-trigger']")).toBeNull();
expect(node.textContent).not.toContain("False positive");
});
describe("original-host controller containment in task recovery", () => {
let client: QueryClient;
const digest = "a".repeat(64);
const runId = "00000000-0000-4000-8000-000000000041";
const onRecorded = vi.fn();
beforeEach(() => {
client = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } });
containmentApi.resolveRecoveryAction.mockReset().mockResolvedValue({});
onRecorded.mockReset();
});
afterEach(() => client.clear());
function mount(canManage = true, reason = "controller_process_space_unverified", authority = digest) {
const action = buildAction({ kind: "active_run_watchdog", ownerType: "board", cause: "native_restart_recovery_blocked",
evidence: { runId, reason, controllerAuthority: { schema: "paperclip.native-controller-authority.v1", digest: authority } } });
const issue = { id: "ordinary-task", status: "in_progress" as const, activeRecoveryAction: action };
const element = <QueryClientProvider client={client}><NativeControllerContainmentRecovery key={authority} issue={issue}
canManage={canManage} onRecorded={onRecorded} /></QueryClientProvider>;
if (!root) return render(element);
act(() => root?.render(element));
return container!;
}
function fillFields(node: HTMLDivElement) {
act(() => {
const area = node.querySelector("textarea")!;
Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(area,
"Verified the original deployment controller and provider stopped; completed actions were reviewed.");
area.dispatchEvent(new Event("input", { bubbles: true }));
const select = node.querySelector("select")!;
select.value = "mixed";
select.dispatchEvent(new Event("change", { bubbles: true }));
});
}
function recordButton(node: HTMLDivElement) {
return Array.from(node.querySelectorAll("button")).find(button => button.textContent === "Record containment")!;
}
it("requires separate controller and provider confirmations on an ordinary task", async () => {
const node = mount();
expect(node.textContent).toContain("original host");
expect(node.textContent).toContain("does not retry");
expect(node.textContent).not.toContain(digest);
fillFields(node);
const boxes = node.querySelectorAll('[role="checkbox"]');
expect(recordButton(node).disabled).toBe(true);
click(boxes[0]);
expect(recordButton(node).disabled).toBe(true);
click(boxes[1]);
expect(recordButton(node).disabled).toBe(false);
click(recordButton(node));
await new Promise(resolve => setTimeout(resolve, 0));
expect(containmentApi.resolveRecoveryAction).toHaveBeenCalledOnce();
expect(containmentApi.resolveRecoveryAction).toHaveBeenCalledWith("ordinary-task", expect.objectContaining({
outcome: "restored", sourceIssueStatus: "in_progress", executionReconciliation: {
runId, controllerStopped: true, providerStopped: true, controllerAuthorityDigest: digest,
actionOutcome: "mixed", outcomeEvidence: "Verified the original deployment controller and provider stopped; completed actions were reviewed.",
},
}));
});
it("does not expose a confirmation form to non-board runtime managers", () => {
const node = mount(false);
expect(node.querySelector("button")).toBeNull();
expect(node.textContent).toContain("company owner");
expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled();
});
it.each(["owner", "admin", "operator", "viewer"] as const)("shows the containment form only for a board %s with owner-stop authority", role => {
const allowed = canBoardConfirmOwnerStop("company", { source: "session", user: null, userId: "board-user", keyId: null, companyIds: ["company"], isInstanceAdmin: false,
memberships: [{ companyId: "company", status: "active", membershipRole: role }] });
const node = mount(allowed);
expect(node.querySelectorAll('[role="checkbox"]').length).toBe(role === "owner" || role === "admin" ? 2 : 0);
expect(Boolean(node.querySelector("button"))).toBe(role === "owner" || role === "admin");
expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled();
});
it("does not offer local containment for remote or unrelated recovery", () => {
const node = mount(true, "remote_controller_process_space_unverified");
expect(node.querySelector("section")).toBeNull();
mount(true, "some_other_hold");
expect(node.querySelector("section")).toBeNull();
});
it("clears old confirmations when the server supplies a different authority snapshot", () => {
const node = mount();
fillFields(node);
for (const box of node.querySelectorAll('[role="checkbox"]')) click(box);
expect(recordButton(node).disabled).toBe(false);
mount(true, "controller_process_space_unverified", "b".repeat(64));
expect(recordButton(node).disabled).toBe(true);
expect(node.querySelector("textarea")?.value).toBe("");
expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled();
});
});
@@ -1,3 +1,9 @@
import { useMutation } from "@tanstack/react-query";
import { issuesApi } from "../api/issues";
import { describeError } from "../api/errors";
import { Checkbox } from "./ui/checkbox";
import { NativeSelect } from "./ui/select";
import type { Issue } from "@paperclipai/shared";
import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared";
import { useWorkspaceIsolationControls } from "@/hooks/useWorkspaceIsolationControls";
import { requiresExecutionReconciliation } from "@paperclipai/shared";
@@ -1510,3 +1516,55 @@ export function IssueRecoveryActionCard({
export type { IssueRecoveryActionStatus };
export default IssueRecoveryActionCard;
/** Separate original-host confirmations: the opaque authority is supplied by
* the server snapshot, never entered or edited by the operator. */
export function NativeControllerContainmentRecovery({ issue, canManage, onRecorded }: {
issue: Pick<Issue, "id" | "status" | "activeRecoveryAction">; canManage: boolean; onRecorded: () => void;
}) {
const action = issue.activeRecoveryAction;
const [controllerStopped, setControllerStopped] = useState(false);
const [providerStopped, setProviderStopped] = useState(false);
const [outcome, setOutcome] = useState<"" | "completed" | "not_performed" | "mixed">("");
const [evidence, setEvidence] = useState("");
const authority = action?.evidence?.controllerAuthority;
const digest = authority && typeof authority === "object" && !Array.isArray(authority)
? (authority as Record<string, unknown>).digest : undefined;
const runId = action?.evidence?.runId;
const submit = useMutation({ mutationFn: async () => {
if (!action || typeof runId !== "string" || typeof digest !== "string" || !outcome
|| !controllerStopped || !providerStopped) throw new Error("Confirm both original-host stops and review the action outcomes.");
return issuesApi.resolveRecoveryAction(issue.id, { actionId: action.id, outcome: "restored", sourceIssueStatus: issue.status,
executionReconciliation: { runId, controllerStopped: true, providerStopped: true,
controllerAuthorityDigest: digest, actionOutcome: outcome, outcomeEvidence: evidence.trim() } });
}, onSuccess: onRecorded });
if (!action || action.kind !== "active_run_watchdog" || action.ownerType !== "board"
|| action.cause !== "native_restart_recovery_blocked" || action.evidence?.reason !== "controller_process_space_unverified"
|| !["active", "escalated"].includes(action.status) || typeof digest !== "string") return null;
return <section aria-label="Original controller recovery">
<h3>Verify the original execution stopped</h3>
<p>Paperclip cannot compare process IDs across hosts or kernel sessions. Stop both the controller and provider on the original host before recording evidence. The saved lease must also expire and environment/workspace cleanup must finish. Recording containment does not retry the task.</p>
{canManage ? <>
<Label><Checkbox checked={controllerStopped} onCheckedChange={value => setControllerStopped(value === true)} /> I verified the original controller stopped on its original host.</Label>
<Label><Checkbox checked={providerStopped} onCheckedChange={value => setProviderStopped(value === true)} /> I verified the original provider and its tools stopped on their original host.</Label>
<Label>Reviewed external action outcomes
<NativeSelect value={outcome} onChange={event => {
const value = event.target.value;
if (value === "completed" || value === "not_performed" || value === "mixed" || value === "") setOutcome(value);
}}>
<option value="">Select only after reviewing the effects</option>
<option value="completed">All completed</option>
<option value="not_performed">None performed</option>
<option value="mixed">Some completed and some not performed</option>
</NativeSelect>
</Label>
<Label>Original-host containment evidence and actions that must not be repeated
<Textarea maxLength={12000} value={evidence} onChange={event => setEvidence(event.target.value)} />
</Label>
<p>If any action outcome is still unknown, keep this hold. After evidence and cleanup settle, use the task’s ordinary Retry action separately.</p>
<Button disabled={submit.isPending || !controllerStopped || !providerStopped || !outcome || evidence.trim().length < 20}
onClick={() => submit.mutate()}>{submit.isPending ? "Recording…" : "Record containment"}</Button>
{submit.error && <p role="alert">{describeError(submit.error).body}</p>}
</> : <p>A company owner, company administrator, or instance administrator must review this original-host execution.</p>}
</section>;
}
@@ -492,7 +492,7 @@ describe("governed retained-file recovery", () => {
expect(api.confirmController).toHaveBeenCalledExactlyOnceWith("issue-1", { actionId: "action-1", runId: "run-1",
stopEvidence: "The old server container was replaced and is not running." });
expect(api.resume).not.toHaveBeenCalled();
expect(container.textContent).toContain("Resuming the run…");
expect(container.textContent).toContain("Controller stop recorded. Checking recovery…");
});
it("confirms the unrecorded server holding a stuck resource, for board owners only", async () => {
@@ -78,7 +78,7 @@ export function WorkspaceExportRecovery({ action, className, issueId, companyId,
copy={{ waiting: "This run’s previous server was recorded before Paperclip stored which host it ran on, so Paperclip can’t verify that it stopped. The run’s saved state is kept.",
question: "How did you confirm that the previous server stopped?",
placeholder: "For example: the old server container was replaced and is no longer running.",
resuming: "Resuming the run…" }} /></>;
resuming: "Controller stop recorded. Checking recovery…" }} /></>;
}
// A resource held by a server recorded before Paperclip stored which server held it.
if (action?.cause === OWNER_UNPROVABLE && issueId && companyId && isOwnerGoneRequired(action, issueId, companyId)) {
+9
View File
@@ -1,3 +1,4 @@
import { NativeControllerContainmentRecovery } from "../components/IssueRecoveryActionCard";
import { useIssueIdBatches } from "@/hooks/useIssueIdBatches";
import { TaskChatLoadingShell } from "@/components/task-chat/TaskChatLoadingShell";
import { useConversationHistoryPages, type ConversationHistoryPage } from "@/hooks/useConversationHistoryPages";
@@ -95,6 +96,7 @@ import { instanceSettingsApi } from "../api/instanceSettings";
import { accessApi } from "../api/access";
import {
canBoardManageRuntime,
canBoardConfirmOwnerStop,
canBoardResolveRecoveryAction,
isRetainedWorkspaceExportCause,
readRecoveryReconcileWorkspaceId,
@@ -3735,6 +3737,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
selectedCompanyId,
boardAccess,
);
const canConfirmOwnerStop = canBoardConfirmOwnerStop(selectedCompanyId, boardAccess);
// Retained-file recovery reports whether it offers its own control. The task's
// Retry stays visible whenever it does not, so a retryable hold never dead-ends.
const [retainedRecoveryOwnsRetry, setRetainedRecoveryOwnsRetry] = useState(true);
@@ -8474,6 +8477,12 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
: null,
onRetry: (actionId) => retryDispositionRecovery.mutateAsync(actionId).then(() => undefined),
}}>
<NativeControllerContainmentRecovery key={`${issue.activeRecoveryAction?.id}:${JSON.stringify(issue.activeRecoveryAction?.evidence?.controllerAuthority)}`} issue={issue}
canManage={canConfirmOwnerStop} onRecorded={() => {
invalidateIssueCollections();
void queryClient.invalidateQueries({ queryKey: queryKeys.issues.detail(issueId!) });
void queryClient.invalidateQueries({ queryKey: queryKeys.issues.runs(issueId!) });
}} />
<IssueDetailChatTab
executionBlocked={Boolean(issue.executionBlocker)}
interruption={issue.interruption}