From a3e7e63330060f8d177e99bfcb26dddf6d6e8b52 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 12:06:11 -0600 Subject: [PATCH] Guard Native recovery with process-space proof and explicit containment (#19813) ## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- doc/execution-semantics.md | 54 +- maps/agent-access-map/recovery.md | 11 + maps/feature-map/recovery.md | 8 + .../shared/src/types/execution-projection.ts | 3 + packages/shared/src/validators/issue.test.ts | 31 + packages/shared/src/validators/issue.ts | 18 +- .../chat-channels.integration.test.ts | 245 +++++ ...eartbeat-native-restart-suspension.test.ts | 947 +++++++++++++++++- ...at-observed-workspace-preservation.test.ts | 12 +- .../helpers/native-restart-fixture.ts | 6 +- ...ue-agent-mutation-ownership-routes.test.ts | 16 + .../ownership-reconciliation-report.test.ts | 6 +- server/src/routes/issues.ts | 32 +- .../chat-channels/failed-run-retry.ts | 9 +- .../chat-channels/task-retry-authority.ts | 6 +- .../services/controller-process-identity.ts | 16 +- .../controller-process-space-identity.test.ts | 71 ++ .../src/services/execution-projection.test.ts | 18 + server/src/services/execution-projection.ts | 21 + .../services/execution-recovery-resolution.ts | 54 +- server/src/services/issue-recovery-actions.ts | 11 +- ...native-accepted-result-lease-settlement.ts | 35 +- .../native-controller-host-recovery.test.ts | 88 +- .../native-controller-host-recovery.ts | 70 +- .../native-failed-run-owner-release.test.ts | 201 +++- .../native-failed-run-owner-release.ts | 297 +++++- .../native-runtime/native-owner-stamps.ts | 11 +- .../native-runtime/native-restart-recovery.ts | 118 ++- .../native-runtime/native-runner-ownership.ts | 49 +- ...unner-restart-recovery.integration.test.ts | 133 ++- .../native-session-executor.test.ts | 2 + .../native-runtime/native-session-executor.ts | 4 +- server/src/services/owner-proof.test.ts | 26 + server/src/services/owner-proof.ts | 6 +- .../services/recovery/ownership-registry.ts | 18 +- .../IssueRecoveryActionCard.test.tsx | 97 +- ui/src/components/IssueRecoveryActionCard.tsx | 58 ++ .../WorkspaceExportRecovery.control.test.tsx | 2 +- ui/src/components/WorkspaceExportRecovery.tsx | 2 +- ui/src/pages/IssueDetail.tsx | 9 + 40 files changed, 2674 insertions(+), 147 deletions(-) create mode 100644 server/src/services/controller-process-space-identity.test.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index c479e918c3f..27a3999f5e0 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1899,8 +1899,13 @@ A native run can accept its result and then lose its server to a hard kill before finalization. Its result stays leased to the controller of the earlier boot, which cannot release the lease the way a graceful drain does. Startup native recovery releases that lease at once when the lease belongs to a -different boot and its recorded controller process has exited or its PID now -names another process. Only the executor's lease carries that controller +different boot in the same known kernel and PID namespace, and its recorded +controller process has exited or its PID now names another process. The +executor saves that namespace with its claim. Linux uses its boot UUID and PID +namespace; macOS uses its kernel boot-session UUID. An unreadable or unsupported +identity, a different namespace, or an older claim without this proof keeps the +original lease expiry and does not probe the recorded PID. Only the executor's +lease carries that controller identity; a finalizer or cleanup lease is never released this way and expires on its own short lease. The stored result is then finalized without another @@ -1908,22 +1913,59 @@ provider turn, and saved follow-up messages are reconsidered. A live owner, an owner without a recorded process, a lease of the current boot, and a remote workspace keep the lease until it expires. Lease age alone is never proof. +Before claiming an in-flight local native run, restart recovery also requires +this same known process space. Expiry alone does not prove a local controller +stopped. A missing, foreign or unreadable namespace creates one board-owned +recovery action for the exact saved authority, without probing the recorded PID, +spending retry attempts, or arming a polling loop. Renewed authority gets a new +opaque fingerprint and preserves the previous action. Remote and plugin tasks +retain their ordinary Stop/provider-cleanup path and do not offer the local +containment confirmation. + +When automatic recovery cannot prove a local owner stopped, an operator can +record containment through the task's recovery action after the original lease +expires. The operator separately confirms the controller and provider/tools +stopped on their original host, and records reviewed action outcomes and evidence. +The server rechecks board `runtime:confirm_owner_stop` permission for a company +owner, company administrator or instance administrator, plus company, assignee, source binding, +opaque authority, exact expiry (including microseconds), pending Stop, accepted +results and cleanup obligations under the normal ownership locks. A renewed or +replaced owner invalidates the confirmation. This is explicit operator evidence, +not a local process-stop observation. + +Containment uses normal failed-owner settlement and preserves usage uncertainty, +workspaces and outcome evidence. Containment records the canonical `restored` +outcome while keeping the task status unchanged, including when it is already +`todo`. A separate ordinary Retry then consumes that committed evidence. It does not accept a provider result or queue a +replacement. Only exact unadmitted Retry receipts for the contained source are +closed with an inspectable reason; accepted or newer work stays intact. After +cleanup and outcome review, the user makes a new ordinary Retry request under +current authorization. Unknown outcomes keep the hold; reviewed completed or +mixed effects remain in the continuation context and never authorize automatic +replay. A later successful ordinary recovery claim resolves only its exact prior +namespace attention. + A recorded process is judged only in the PID space that recorded it. Native controller rows store `controller_host_hash` and run processes store `process_host_hash` (`"remote"` for a sandbox process). A dead PID on this host never proves that a process on another host, in another PID namespace, or in a sandbox is gone; such a record keeps its lease, fence or staleness TTL. -Rows saved before those columns existed have no recorded host. For one release -only, a single-host server keeps the previous local proof for them, and each use +Rows saved before those columns existed have no recorded host. The legacy +compatibility helper and run-process sweep retain one-release single-host local +proof where their existing owners permit it. Each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry, so the release that removes this fallback can confirm that no rows still depend on it. An ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`) never uses the fallback. Its native controller rows with no recorded host wait as `controller_host_unrecorded`, with a board recovery action to confirm that the previous server stopped. Its run processes with no recorded host are left to the run's Stop control. The next -release removes the fallback and treats every row with no recorded host as -unprovable. +release removes that compatibility fallback. The strict Native in-flight claim, +early accepted-result release, and failed-owner guards above do not use it. +A previous-server-only confirmation is not same-namespace proof and does not +confirm that a provider or its tools stopped. It cannot override those guards. +Its notification requests recovery inspection; unfinished ownership can still +require the separate containment and ordinary Retry described above. A server boot that holds work keeps a controller instance lease (`controller_instances`): a row named by its boot id, with an epoch and a lease diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index 4d1d4d23f3e..0dccbc827c6 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -100,6 +100,17 @@ Gaps: Component host test doesn't prove actual filesystem/provider repair. ## `bounded-continuation` +Original-host Native controller containment requires board-only +`runtime:confirm_owner_stop` authority on the existing recovery resolution route: +a company owner, company administrator, or instance administrator. Agents cannot submit it. +It requires separate controller/provider stop confirmations, reviewed outcomes, +a server-bound exact authority digest, elapsed lease and unchanged locked +source/assignee/cleanup fences. It records containment without queuing work; a +separate ordinary task Retry is required. Unknown process space never authorizes +a local PID probe or automatic physical takeover, even after expiry. +Evidence: `source-only` — candidate authority, route, UI and real-owner controls; +no deployed/original-host stop verification is claimed. + Capability ID: `recovery/bounded-continuation`. Ordinary task Retry uses `POST /api/issues/{id}/retry`. This is authenticated diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index 340529aceb6..f434bc8b16d 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -64,6 +64,14 @@ and the [disconnect recovery journey](../../tests/e2e/disconnect-write-recovery. ### `source-task` +A Native run held for unknown controller process space remains blocked even +after its lease expires. A company owner, company administrator, or instance +administrator can record separate original-host controller and provider/tool stop confirmations with reviewed +action outcomes through the existing task recovery card. Recording containment +does not replay work. After normal settlement/cleanup, use a separate ordinary +Retry; stale authority or unresolved cleanup must keep it blocked. This candidate +path is source-only pending its focused real-owner qualification. + Open the affected task from Tasks, Inbox/Blocked, search, or a direct task link. Observe one working, backoff, failed or waiting-for-user state. Use Retry after an infrastructure failure. No process or file certification is required. diff --git a/packages/shared/src/types/execution-projection.ts b/packages/shared/src/types/execution-projection.ts index 39c7af04e52..22fce6bdcd7 100644 --- a/packages/shared/src/types/execution-projection.ts +++ b/packages/shared/src/types/execution-projection.ts @@ -85,6 +85,9 @@ export function requiresExecutionReconciliation( export interface ExecutionReconciliation { runId: string; providerStopped: true; + /** Only the exact namespace-unverified Native ownership hold accepts these. */ + controllerStopped?: true; + controllerAuthorityDigest?: string; actionOutcome: "completed" | "not_performed" | "mixed"; outcomeEvidence: string; /** Operator evidence bound to this failed run; required after workspace restore failure. */ diff --git a/packages/shared/src/validators/issue.test.ts b/packages/shared/src/validators/issue.test.ts index 2ffd5488cda..9a10bf1112d 100644 --- a/packages/shared/src/validators/issue.test.ts +++ b/packages/shared/src/validators/issue.test.ts @@ -369,6 +369,37 @@ describe("issue validators", () => { } }).success).toBe(true); }); + it.each(["backlog", "in_progress", "blocked"] as const)("allows a complete controller containment request to retain %s status for locked route validation", sourceIssueStatus => { + const input = { actionId: "22222222-2222-4222-8222-222222222222", outcome: "restored", sourceIssueStatus, + executionReconciliation: { runId: "11111111-1111-4111-8111-111111111111", providerStopped: true, + controllerStopped: true, controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed", + outcomeEvidence: "The original-host controller and provider stopped; action outcomes were reviewed." } }; + // The real historical HTTP request failed here before its route could + // enforce same-status/current-authority checks. Syntax grants no authority. + expect(resolveIssueRecoveryActionSchema.parse(input)).toEqual(input); + }); + + it.each(["controller-only", "digest-only", "missing-action", "wrong-outcome", "partial-with-workspace-evidence"] as const)( + "rejects incomplete controller containment even when ordinary status validation would pass: %s", mode => { + const complete = { runId: "11111111-1111-4111-8111-111111111111", providerStopped: true, + controllerStopped: true, controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed", + outcomeEvidence: "The original-host controller and provider stopped; action outcomes were reviewed." }; + const ordinary = { runId: complete.runId, providerStopped: complete.providerStopped, + actionOutcome: complete.actionOutcome, outcomeEvidence: complete.outcomeEvidence }; + const input = { actionId: mode === "missing-action" ? undefined : "22222222-2222-4222-8222-222222222222", + outcome: mode === "wrong-outcome" ? "blocked" : "restored", + sourceIssueStatus: mode === "wrong-outcome" ? "blocked" : "todo", + executionReconciliation: mode === "controller-only" ? { ...ordinary, controllerStopped: true } + : mode === "digest-only" ? { ...ordinary, controllerAuthorityDigest: complete.controllerAuthorityDigest } + : mode === "partial-with-workspace-evidence" ? { ...ordinary, controllerStopped: true, + workspaceRepairEvidence: "Retained workspace files were repaired and inspected separately." } + : complete }; + const result = resolveIssueRecoveryActionSchema.safeParse(input); + expect(result.success).toBe(false); + if (!result.success) expect(result.error.issues).toContainEqual(expect.objectContaining({ path: ["executionReconciliation"] })); + }, + ); + it("allows cancelled recovery resolutions to atomically restore the source issue status", () => { expect( resolveIssueRecoveryActionSchema.parse({ diff --git a/packages/shared/src/validators/issue.ts b/packages/shared/src/validators/issue.ts index 8a8affbcdcf..3fbdc7cac53 100644 --- a/packages/shared/src/validators/issue.ts +++ b/packages/shared/src/validators/issue.ts @@ -652,6 +652,8 @@ export const resolveIssueRecoveryActionSchema = z .object({ runId: z.string().guid(), providerStopped: z.literal(true), + controllerStopped: z.literal(true).optional(), + controllerAuthorityDigest: z.string().regex(/^[0-9a-f]{64}$/).optional(), actionOutcome: z.enum(["completed", "not_performed", "mixed"]), outcomeEvidence: z.string().trim().min(20).max(12000), workspaceRepairEvidence: z.string().trim().min(20).max(12000).optional(), @@ -668,10 +670,20 @@ export const resolveIssueRecoveryActionSchema = z if (value.workspaceBaseRef && (!value.actionId || value.outcome !== "restored" || value.sourceIssueStatus !== "todo" || value.executionReconciliation)) { ctx.addIssue({ code: z.ZodIssueCode.custom, path: ["workspaceBaseRef"], message: "Branch repair must retry the exact recovery action" }); } + const reconciliation = value.executionReconciliation; + const controllerContainment = reconciliation?.controllerStopped !== undefined + || reconciliation?.controllerAuthorityDigest !== undefined; + if (controllerContainment && (!value.actionId || value.outcome !== "restored" + || reconciliation?.controllerStopped !== true || !reconciliation?.controllerAuthorityDigest)) { + ctx.addIssue({ code: z.ZodIssueCode.custom, path: ["executionReconciliation"], + message: "Controller containment requires the exact recovery action, both stop confirmations, and its authority digest" }); + return; + } if (value.outcome === "restored") { - // A retained-source repair records evidence without changing task state. - // The route verifies the exact server-owned source and unchanged status. - if (value.executionReconciliation?.workspaceRepairEvidence) return; + // Source repair and complete controller containment record evidence + // without changing task state. The route checks the exact locked source, + // unchanged status, board authority, lease expiry, and cleanup evidence. + if (value.executionReconciliation?.workspaceRepairEvidence || controllerContainment) return; if ( value.sourceIssueStatus !== "todo" && value.sourceIssueStatus !== "done" && diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index e5af378da22..9ceb3a0ee18 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1,3 +1,4 @@ +import { releaseAbandonedNativeOwners } from "../services/native-runtime/native-failed-run-owner-release.js"; import { guardSlackFixtureNetwork } from "./helpers/slack-cloud/transport.js"; import { chatHistoryActionBlocks } from "./helpers/chat-history-proof.js"; import type { NativeExecutionInput, NativeSession, NativeSessionBackend, PrpEvent } from "../vendor/paperclip-runner/index.js"; @@ -67914,6 +67915,250 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); + it.each(["blocked_owner_queued", "todo_owner_useful", "todo_admin_useful", "todo_operator_denied"] as const)( + "admits a separate connected-conversation Retry only after committed original-controller containment: %s", async mode => { + const f = await failedChatRetryFixture("telegram", undefined, true); + const companyId = f.fixture.companyId, issueId = f.issue.id, agentId = f.fixture.assignedAgentId; + let heartbeat: ReturnType | undefined; + const useful = mode.endsWith("useful"), denied = mode === "todo_operator_denied"; + const role = mode === "todo_admin_useful" ? "admin" : denied ? "operator" : "owner"; + // Initial task state and real membership: no status/receipt rewrite occurs + // during containment or the new Retry journey. + await db.update(companyMemberships).set({ membershipRole: role }).where(and( + eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, "owner-user"))); + if (mode.startsWith("todo")) await db.update(issues).set({ status: "todo" }).where(eq(issues.id, issueId)); + const settings = instanceSettingsService(db), previousSettings = await settings.getExperimental(); + const workingDirectory = useful ? mkdtempSync(path.join(os.tmpdir(), "chat-contained-todo-")) : null; + let providerTurns = 0; + const answer = "The separately retried todo request completed once after original-host containment."; + + // Historical initial facts: the original connected request was admitted, + // failed after a recorded launch on a foreign host, and retained an expired + // result-less owner. No stop/result or successful provider outcome is seeded. + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, nativePhase: "observed", + processPid: 2_000_000_001, processGroupId: 2_000_000_001, processStartedAt: new Date(0), + runnerProfileJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: f.runId, phase: "observed", + leaseOwner: "native-exec:historical-connected-source", leaseExpiresAt: new Date(Date.now() - 1_000), + controllerBootId: randomUUID(), controllerHostHash: "historical-foreign-process-space", + controllerPid: 2_000_000_001, controllerProcessStartedAt: new Date(0), controllerGeneration: 1, attempt: 1 }); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: f.runId, + eventType: "native.process_identity_recorded", payload: { targetKind: "local", hostname: "original-foreign-host", + processPid: 2_000_000_001, processGroupId: 2_000_000_001, processStartedAt: new Date(0).toISOString() } }); + try { + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + const [action] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.cause, "native_restart_recovery_blocked"), eq(issueRecoveryActions.status, "active"))); + expect(action).toBeDefined(); + const authority = action.evidence.controllerAuthority; + if (!authority || typeof authority !== "object" || Array.isArray(authority) + || !("digest" in authority) || typeof authority.digest !== "string") throw new Error("Normal exact controller attention required"); + const app = express(); app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); app.use(errorHandler); + const [task] = await readIssuesById(issueId); + const evidence = "Verified original controller and provider stopped on the original host; retained completed external actions."; + expect(task.status).toBe(mode.startsWith("todo") ? "todo" : "blocked"); + const beforeRuns = await readHeartbeatRunsByCompanyId(companyId); + const beforeWakes = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId)); + const beforeOwner = (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]; + const containment = await request(app).post(`/api/issues/${issueId}/recovery-actions/resolve`).send({ actionId: action.id, + outcome: "restored", sourceIssueStatus: task.status, executionReconciliation: { runId: f.runId, + controllerStopped: true, providerStopped: true, controllerAuthorityDigest: authority.digest, + actionOutcome: "mixed", outcomeEvidence: evidence } }); + if (denied) { + expect(containment.status).toBe(403); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(beforeOwner); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]).toEqual(action); + expect(await readHeartbeatRunsByCompanyId(companyId)).toEqual(beforeRuns); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId))).toEqual(beforeWakes); + // A deliberate click may persist an intent; only the REAL normal + // queue may decide admission. The transport fixture's receipt stub is + // not sufficient evidence that an uncontained source cannot execute. + const unexpectedBackend = vi.fn((): NativeSessionBackend => { + throw new Error("An uncontained foreign owner must never create a new provider backend"); + }); + await settings.updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true }); + heartbeat = heartbeatService(db, { nativeSessionBackendFactory: unexpectedBackend }); + f.wakeup.mockImplementation(heartbeat.wakeup); + const idempotencyKey = randomUUID(); + const retryInput = { observedRunId: f.runId, idempotencyKey }; + const click = await request(app).post(`/api/issues/${issueId}/retry`).send(retryInput); + expect(click.status).toBe(202); + expect(click.body.requestId).toEqual(expect.any(String)); + expect(click.body.chatActionId).toEqual(expect.any(String)); + const deniedRetry = await f.service.processFailedChatRunRetry(click.body.chatActionId); + expect(deniedRetry).toMatchObject({ status: "failed", runId: null }); + const [deniedAction] = await db.select().from(chatActions).where(eq(chatActions.id, click.body.chatActionId)); + expect(deniedAction).toMatchObject({ status: "failed", result: { code: "failed_chat_retry_not_authorized" } }); + await reconcileTaskChatRetryReceipts(db, { companyId, requestId: click.body.requestId }); + const [intent] = await readAgentWakeupRequestsById(click.body.requestId); + expect(intent).toMatchObject({ status: "failed", runId: null, payload: { taskRetry: { + sourceRunId: f.runId, actorId: "owner-user", actorSource: "session", chatActionId: click.body.chatActionId, + } } }); + // A lost HTTP response may be retried with the same key. It replays + // this exact denied intent, never creates another execution authority. + const replay = await request(app).post(`/api/issues/${issueId}/retry`).send(retryInput); + expect(replay.status).toBe(202); + expect(replay.body).toMatchObject({ requestId: click.body.requestId, chatActionId: click.body.chatActionId }); + expect(await f.service.processFailedChatRunRetry(replay.body.chatActionId)).toMatchObject({ status: "failed", runId: null }); + expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), + eq(agentWakeupRequests.idempotencyKey, `task-retry:${idempotencyKey}`)))).toHaveLength(1); + await heartbeat.resumeTaskRetry(companyId, click.body.requestId); + expect(unexpectedBackend).not.toHaveBeenCalled(); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(beforeOwner); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]).toEqual(action); + expect(await readHeartbeatRunsByCompanyId(companyId)).toEqual(beforeRuns); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(0); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0); + return; + } + expect(containment.status).toBe(200); + const [settled] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(settled).toMatchObject({ phase: "terminal_failure", leaseOwner: null, resultId: null, + failureCode: "native_restart_recovery_blocked", failureDetail: { releasedOwner: { reason: "operator_containment" } } }); + const [resolved] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)); + expect(resolved).toMatchObject({ status: "resolved", evidence: { continuationDelivery: "not_requested" } }); + if (!useful) expect(resolved.outcome).toBe("restored"); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0); + expect(await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.eventType, "native.local_process_stopped")))).toHaveLength(0); + const direct = { companyId, issueId, agentId, failedRunId: f.runId, initiatedByUserId: "owner-user" }; + // These are corrupt/stale control receipts, not provider outcomes. Restore + // their exact committed beforeimages before the real new Retry below. + for (const invalid of ["wrong_digest", "changed_generation", "not_resolved"] as const) { + try { + if (invalid === "wrong_digest") await db.update(issueRecoveryActions).set({ evidence: { + ...resolved.evidence, controllerAuthority: { schema: "paperclip.native-controller-authority.v1", digest: "0".repeat(64) }, + } }).where(eq(issueRecoveryActions.id, action.id)); + if (invalid === "changed_generation") await db.update(nativeRunFinalizations) + .set({ controllerGeneration: settled.controllerGeneration + 1 }).where(eq(nativeRunFinalizations.runId, f.runId)); + if (invalid === "not_resolved") await db.update(issueRecoveryActions).set({ status: "cancelled" }) + .where(eq(issueRecoveryActions.id, action.id)); + await expect(db.transaction(tx => f.service.prepareFailedChatRunRetry(tx, direct))) + .rejects.toMatchObject({ details: { code: "chat_failed_run_retry_not_authorized" } }); + } finally { + await db.update(issueRecoveryActions).set({ evidence: resolved.evidence, status: resolved.status }) + .where(eq(issueRecoveryActions.id, action.id)); + await db.update(nativeRunFinalizations).set({ controllerGeneration: settled.controllerGeneration }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + } + } + // Existing real-queue pattern: another task occupies capacity, so this + // assertion proves admission and exact provenance, not provider execution. + const backend = (execution: NativeExecutionInput): NativeSessionBackend => { + const turnId = randomUUID(), evidenceId = randomUUID(), evidenceRef = `work_product:${evidenceId}`; + const identity = { companyId, agentId, issueId, runId: execution.binding.runId, + sessionId: execution.session.normalizedSessionId! }; + const result = structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT); + result.summary = answer; + result.completionClaim.contractRevision = execution.completionContract.contract.revision; + result.completionClaim.criteria = execution.completionContract.contract.criteria.map(criterion => ({ + criterionId: criterion.id, status: "satisfied", evidenceRefs: [evidenceRef] })); + result.evidence = [{ kind: "work_product", ref: evidenceRef }]; result.verification[0]!.artifactRef = evidenceRef; + let started = () => {}; const turnStarted = new Promise(resolve => { started = resolve; }); + const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; + const session: NativeSession = { + identity: () => identity, async capabilities() { return capabilities; }, + async *events() { + await turnStarted; const [run] = await readHeartbeatRunsById(identity.runId); + yield { schema: "paperclip.prp.event.v1", sourceEventId: `${identity.runId}:terminal`, sourceSeq: 1, + sourceInstanceId: run.runnerInstanceId!, sourceKind: "runner", runId: identity.runId, + normalizedSessionId: identity.sessionId, turnId, eventType: "turn.completed", schemaVersion: 1, + priority: 0, emittedAt: new Date().toISOString(), payload: {} } satisfies PrpEvent; + }, + async startTurn() { providerTurns += 1; started(); return { turnId }; }, + async result() { await turnStarted; return { result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }; }, + async snapshot() { return { backendKind: "mock", sessionId: "contained-todo-fixture", identity, + providerSessionId: `provider-${identity.runId}`, activeTurnId: null, pendingRuntimeRequests: [], lineage: [], cursor: "1" }; }, + async close() {}, + }; + return { + async descriptor() { return { kind: "mock", name: "contained-todo-retry", version: "1", capabilities, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; }, + async openSession() { + await db.insert(issueWorkProducts).values({ id: evidenceId, companyId, issueId, + type: "artifact", provider: "paperclip", title: "Contained todo request result", status: "ready_for_review", reviewState: "approved" }); + return session; + }, + async recoverSession() { throw new Error("Containment Retry must create a fresh authorized session, never replay the old provider"); }, + }; + }; + if (useful) await settings.updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true }); + await db.update(agents).set({ adapterType: "paperclip_runner", status: useful ? "idle" : "running", + ...(useful ? { adapterConfig: { provider: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", + cwd: workingDirectory, workspaceStrategy: { type: "project_primary" } } } : {}), + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId)); + if (!useful) await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", startedAt: new Date() }); + heartbeat = heartbeatService(db, useful ? { nativeSessionBackendFactory: backend } : {}); + f.wakeup.mockImplementation(heartbeat.wakeup); + const classifyDenial = (value: unknown) => value === "chat_failed_run_retry_not_authorized" + || value === "failed_chat_retry_not_authorized" ? value : "withheld_unknown"; + const click = await request(app).post(`/api/issues/${issueId}/retry`) + .send({ observedRunId: f.runId, idempotencyKey: randomUUID() }); + expect(click.status, JSON.stringify({ phase: "ordinary_retry_response", status: click.status, + code: classifyDenial(click.body?.code ?? click.body?.details?.code) })).toBe(202); + expect(resolved.outcome).toBe("restored"); + expect(click.body.chatActionId).toBeTruthy(); + const outcome = await f.service.processFailedChatRunRetry(click.body.chatActionId); + const [retryAction] = await db.select().from(chatActions).where(eq(chatActions.id, click.body.chatActionId)); + const outcomeDiagnostic = JSON.stringify({ phase: "durable_chat_retry", status: outcome.status, + code: classifyDenial(retryAction?.result?.code) }); + if (useful) expect(["queued", "running", "succeeded"], outcomeDiagnostic).toContain(outcome.status); + else expect(outcome.status, outcomeDiagnostic).toBe("queued"); + const [wake] = await readAgentWakeupRequestsById(click.body.chatActionId); + expect(wake).toMatchObject({ ...(!useful ? { status: "queued" } : {}), runId: expect.any(String) }); + const [successor] = await readHeartbeatRunsById(wake.runId!); + expect(successor).toMatchObject({ ...(!useful ? { status: "queued" } : {}), retryOfRunId: f.runId, contextSnapshot: { + source: "chat:telegram", wakeCommentIds: [f.action.payload.commentId], taskRetryRequestId: click.body.requestId, + } }); + expect(successor.contextSnapshot?.forceFreshSession).toBe(true); + // Use the same canonical projection as heartbeat/task-context before + // execution. The chat payload carries a source reference, never a copy + // of operator authority or a fabricated provider outcome. + const { buildExecutionContinuation } = await import("../services/execution-continuation.js"); + const continuation = await buildExecutionContinuation({ db, companyId, issueId, agentId, + runId: successor.id, context: successor.contextSnapshot ?? {}, summary: null, exposeLowTrustRaw: false }); + expect(continuation.trigger.sourceRunId).toBe(f.runId); + expect(continuation.recoveryOutcomes).toContainEqual({ recoveryActionId: action.id, + decision: expect.objectContaining({ runId: f.runId, controllerStopped: true, providerStopped: true, + actionOutcome: "mixed", outcomeEvidence: evidence }) }); + await f.service.processFailedChatRunRetry(click.body.chatActionId); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ resultId: null, failureDetail: { releasedOwner: { actionOutcome: "mixed", outcomeEvidence: evidence } } }); + if (useful) { + await heartbeat.resumeQueuedRuns(); + await vi.waitFor(async () => { + const [completed] = await readHeartbeatRunsById(successor.id); + expect(completed.status, JSON.stringify({ phase: "useful_successor", status: completed.status, + errorCode: completed.errorCode, nativePhase: completed.nativePhase })).toBe("succeeded"); + expect(completed.resultJson?.nativeResult).toMatchObject({ summary: answer }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, successor.id)))[0].phase).toBe("committed"); + expect(providerTurns).toBe(1); + await f.service.processPendingPublications(); await f.service.processPendingPublications(); + expect((await readChatPublicationsByEndpointId(f.endpoint.id)).filter(publication => publication.state === "published" + && typeof publication.payload.text === "string" && publication.payload.text.includes(answer))).toHaveLength(1); + }, { timeout: 10_000, interval: 25 }); + const [completed] = await readHeartbeatRunsById(successor.id); + expect(completed.resultJson?.nativeResult).toMatchObject({ summary: answer }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, successor.id)))[0].phase).toBe("committed"); + expect(providerTurns).toBe(1); + expect((await readChatPublicationsByEndpointId(f.endpoint.id)).filter(publication => publication.state === "published" + && typeof publication.payload.text === "string" && publication.payload.text.includes(answer))).toHaveLength(1); + await f.service.processFailedChatRunRetry(click.body.chatActionId); + await heartbeat.resumeTaskRetry(companyId, click.body.requestId); + expect(providerTurns).toBe(1); + } + } finally { + await f.service.shutdown(); if (heartbeat) await heartbeat.drainActiveRunExecutions(); + if (useful || denied) await settings.updateExperimental(previousSettings); + if (workingDirectory) rmSync(workingDirectory, { recursive: true, force: true }); + } + }); + it.each(["stopped", "real_queue", "unknown_executor", "source_edited", "cancelled_stopped", "cancelled_unknown"] as const)( "admits ordinary chat task Retry with a missing checkpoint only after containment: %s", async kind => { const context = await failedChatRetryFixture("telegram", undefined, kind === "real_queue"); diff --git a/server/src/__tests__/heartbeat-native-restart-suspension.test.ts b/server/src/__tests__/heartbeat-native-restart-suspension.test.ts index edf2f07afeb..49a7dd3385f 100644 --- a/server/src/__tests__/heartbeat-native-restart-suspension.test.ts +++ b/server/src/__tests__/heartbeat-native-restart-suspension.test.ts @@ -1,13 +1,30 @@ +import { confirmNativeControllerStopped, requestNativeControllerStopConfirmation } from "../services/native-runtime/native-controller-host-recovery.js"; +import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js"; +import { nativeControllerAuthorityDigest } from "../services/native-runtime/native-runner-ownership.js"; +import { cancelHeartbeatNativeRun } from "../services/heartbeat/run-control.js"; +import { claimCancellationRequest } from "../services/native-runtime/native-cancellation-request.js"; +import express from "express"; +import request from "supertest"; +import * as heartbeatModule from "../services/heartbeat.js"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/error-handler.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { waitForCondition } from "./helpers/native-restart-fixture.js"; +import { requestTaskRetry } from "../services/task-retry.js"; +import type { HeartbeatServiceOptions } from "../services/heartbeat.js"; +import type { NativeRunIdentity, NativeSession, PrpEvent } from "@paperclipai/paperclip-runner"; import { randomUUID } from "node:crypto"; +import * as controllerProcess from "../services/controller-process-identity.js"; import { and, eq, sql } from "drizzle-orm"; -import { activityLog, authUsers, budgetReservations, issueRecoveryActions, companyMemberships, completionContracts, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations } from "@paperclipai/db"; -import { expect, it, vi } from "vitest"; +import { activityLog, agents, createDb, agentWakeupRequests, authUsers, budgetReservations, companyMemberships, completionContracts, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations } from "@paperclipai/db"; +import { describe, expect, it, vi } from "vitest"; import { subscribeDatabaseWork } from "@paperclipai/db/work-signals"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { resolveLocalRuntimeApiUrl } from "../runtime-api.js"; import { heartbeatService } from "../services/heartbeat.js"; import { instanceSettingsService } from "../services/instance-settings.js"; -import { classifyNativeRunnerRecoveryEvidence, currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js"; +import { claimNativeRestartRecoveries, classifyNativeRunnerRecoveryEvidence, currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js"; import type { NativeSessionBackend } from "@paperclipai/paperclip-runner"; import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js"; import { CONTROL_PLANE_CONFORMANCE_OPEN, CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js"; @@ -15,16 +32,28 @@ import { describeEmbeddedPostgres, nativeRestartFixture } from "./helpers/native import { currentControllerBootId, currentProcessSpaceHash } from "../services/controller-process-identity.js"; import { createOwnershipReportState, runOwnershipReconciliationPass, settleOwnableKind, type OwnershipReport } from "../services/recovery/ownership-invariant-report.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; -import { registerOwnershipReconciliationWork } from "../services/native-runtime/native-finalization-reconciler.js"; +import { reconcileNativeFinalizations, registerOwnershipReconciliationWork } from "../services/native-runtime/native-finalization-reconciler.js"; import { OWNABLE_KINDS } from "../services/recovery/ownership-registry.js"; -import { proveOwnerGone } from "../services/owner-proof.js"; +import { currentProcessProbe, proveOwnerGone } from "../services/owner-proof.js"; describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shutdown", () => { const fixture = nativeRestartFixture(); - async function seedRecoverableRun(label: string) { - const f = await fixture.seedRun(label); + async function freshExecutionAgent() { + const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 }); + const [template] = await db.select().from(agents).where(eq(agents.id, fixture.current.agentId)); + const agentId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId: template.companyId, + name: "Isolated useful recovery execution", role: template.role, status: "active", + adapterType: template.adapterType, adapterConfig: template.adapterConfig, + runtimeConfig: template.runtimeConfig, permissions: template.permissions }); + return { db, agentId }; + } + + async function seedRecoverableRun(label: string, isolatedAgent = false) { + const isolated = isolatedAgent ? await freshExecutionAgent() : undefined; + const f = await fixture.seedRun(label, isolated?.db, { agentId: isolated?.agentId }); await instanceSettingsService(f.db).updateExperimental({ enableNativeRunner: true }); const userId = randomUUID(); await f.db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.invalid`, @@ -134,8 +163,12 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut // saved when it reaches the provider session. const savedAttempts: number[] = []; const reachedProvider = async () => { - const [owner] = await f.db.select({ attempt: nativeRunFinalizations.attempt }).from(nativeRunFinalizations) + const [owner] = await f.db.select({ attempt: nativeRunFinalizations.attempt, + controllerHostHash: nativeRunFinalizations.controllerHostHash }).from(nativeRunFinalizations) .where(eq(nativeRunFinalizations.runId, f.runId)); + const actualProcess = await controllerProcess.currentControllerProcess(); + expect(actualProcess.hostHash).not.toBe("unknown"); + expect(owner.controllerHostHash).toBe(actualProcess.hostHash); savedAttempts.push(owner.attempt); throw new Error("stop at provider dispatch"); }; @@ -256,6 +289,8 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut leaseOwner: leaseOwner(f.native.runnerInstanceId), leaseExpiresAt, controllerBootId: controller.bootId, + // Local-positive accepted-result fixtures model the actual executor claim. + controllerHostHash: (await controllerProcess.currentControllerProcess()).hostHash, controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 1, @@ -304,7 +339,8 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut const providerDispatch = vi.fn(() => { throw new Error("an accepted result must not open another provider turn"); }); const next = heartbeatService(f.db, { nativeSessionBackendFactory: providerDispatch }); try { - await next.recoverNativeRunsAfterRestart(new Date()); + await next.recoverNativeRunsAfterRestart(new Date(), [f.runId]); + await reconcileNativeFinalizations(f.db, [f.runId]); } finally { await next.drainActiveRunExecutions(); } expect(providerDispatch).not.toHaveBeenCalled(); @@ -338,14 +374,25 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut }) { await f.db.update(nativeRunFinalizations).set({ controllerBootId: controller.bootId, controllerPid: controller.pid, controllerProcessStartedAt: controller.processStartedAt, - controllerHostHash: controller.hostHash ?? null }) + ...(Object.hasOwn(controller, "hostHash") ? { controllerHostHash: controller.hostHash ?? null } : {}) }) .where(eq(nativeRunFinalizations.runId, f.runId)); } + function acceptedRunKinds(f: Awaited>) { + const kind = OWNABLE_KINDS.find(candidate => candidate.id === "native_accepted_result_lease"); + if (!kind?.listForRuns) throw new Error("Native accepted-result run listing required"); + const listForRuns = kind.listForRuns; + return [{ ...kind, listInProgress: async (db: typeof f.db) => ({ + records: await listForRuns(db, [f.runId]), truncated: false, + }) }]; + } + async function restart(f: Awaited>) { const providerDispatch = vi.fn(() => { throw new Error("an accepted result must not open another provider turn"); }); const next = heartbeatService(f.db, { nativeSessionBackendFactory: providerDispatch }); - try { await next.recoverNativeRunsAfterRestart(new Date()); } - finally { await next.drainActiveRunExecutions(); } + try { + await next.recoverNativeRunsAfterRestart(new Date(), [f.runId]); + await reconcileNativeFinalizations(f.db, [f.runId]); + } finally { await next.drainActiveRunExecutions(); } expect(providerDispatch).not.toHaveBeenCalled(); const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); @@ -400,6 +447,141 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut expect({ status: run.status, phase: owner.phase }).toEqual({ status: "succeeded", phase: "committed" }); }, 90_000); + it("keeps an unexpired accepted-result executor lease without process-space proof", async () => { + const f = await seedAcceptedResult("ACCEPTED-UNPROVEN-PROCESS-SPACE", runner => `${runner}:${randomUUID()}`); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, + processStartedAt: new Date(Date.now() - 60_000) }); + await f.db.update(nativeRunFinalizations).set({ controllerHostHash: null }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(before.controllerHostHash).toBeNull(); + const now = new Date(); + expect(before.leaseExpiresAt?.getTime()).toBeGreaterThan(now.getTime()); + expect(before).toMatchObject({ phase: "workspace_finalizing", resultId: expect.any(String) }); + const isProcessAlive = vi.fn(() => false); + const readProcessStartedAt = vi.fn(async () => null); + const released = await settleOwnableKind(f.db, "native_accepted_result_lease", { + now, runIds: [f.runId], probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt }, + }); + expect(released).toEqual([]); + expect(isProcessAlive).not.toHaveBeenCalled(); + expect(readProcessStartedAt).not.toHaveBeenCalled(); + const [after] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(after).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, + controllerBootId: before.controllerBootId, controllerPid: before.controllerPid, + phase: before.phase, resultId: before.resultId }); + const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(run.status).toBe("running"); + expect(await f.db.select({ id: nativeRunResults.id }).from(nativeRunResults) + .where(eq(nativeRunResults.runId, f.runId))).toHaveLength(1); + }, 60_000); + + it.each(["foreign", "unknown-current"] as const)("retains an accepted-result lease without probing an owner in %s process space", async mode => { + const f = await seedAcceptedResult(`ACCEPTED-SPACE-${mode}`, runner => `${runner}:${randomUUID()}`); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, processStartedAt: new Date(Date.now() - 60_000) }); + const actual = await controllerProcess.currentControllerProcess(); + if (mode === "foreign") await f.db.update(nativeRunFinalizations).set({ controllerHostHash: `foreign:${actual.hostHash}` }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const isProcessAlive = vi.fn(() => false), readProcessStartedAt = vi.fn(async () => null); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now: new Date(), runIds: [f.runId], probe: { ...currentProcessProbe(), processSpace: mode === "unknown-current" ? "unknown" : actual.hostHash, deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt } })).toEqual([]); + expect(isProcessAlive).not.toHaveBeenCalled(); + expect(readProcessStartedAt).not.toHaveBeenCalled(); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]) + .toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, controllerHostHash: before.controllerHostHash }); + }, 60_000); + + it.each(["dead", "recycled"] as const)("releases only the exact same-space %s owner and finalizes its accepted result once", async mode => { + const f = await seedAcceptedResult(`ACCEPTED-LOCAL-${mode}`, runner => `${runner}:${randomUUID()}`); + const birth = new Date(Date.now() - 60_000); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, processStartedAt: birth }); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(before.controllerHostHash).toBe((await controllerProcess.currentControllerProcess()).hostHash); + const now = new Date(); + const isProcessAlive = vi.fn(() => mode === "recycled"); + const readProcessStartedAt = vi.fn(async () => new Date(birth.getTime() + 2_000)); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now, runIds: [f.runId], probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt } })) + .toEqual([{ kind: "native_accepted_result_lease", recordId: f.runId, runId: f.runId, + reason: mode === "dead" ? "pid_dead" : "pid_reused" }]); + expect(isProcessAlive).toHaveBeenCalledExactlyOnceWith(2_000_000_001); + const [released] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(released).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: now, resultId: before.resultId }); + for (let attempt = 0; attempt < 2; attempt++) { + const { run, owner } = await restart(f); + expect({ status: run.status, phase: owner.phase }).toEqual({ status: "succeeded", phase: "committed" }); + } + expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("done"); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(1); + expect(await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1); + }, 90_000); + + it.each(["same-birth", "unreadable-birth", "missing-recorded-birth", "ambiguous-birth"] as const)( + "retains a same-space live owner with %s evidence", async mode => { + const f = await seedAcceptedResult(`ACCEPTED-HOLD-${mode}`, runner => `${runner}:${randomUUID()}`); + const birth = new Date(Date.now() - 60_000); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, + processStartedAt: mode === "missing-recorded-birth" ? null : birth }); + const isProcessAlive = vi.fn(() => true); + const readProcessStartedAt = vi.fn(async () => mode === "unreadable-birth" ? null + : new Date(birth.getTime() + (mode === "ambiguous-birth" ? 500 : 0))); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now: new Date(), runIds: [f.runId], probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt } })).toEqual([]); + expect(isProcessAlive).toHaveBeenCalledExactlyOnceWith(2_000_000_001); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].leaseExpiresAt) + .toEqual(f.leaseExpiresAt); + }, 60_000); + + it.each(["lease-token", "boot", "namespace"] as const)("loses dead-owner compare-and-set when %s changes after observation", async changed => { + const f = await seedAcceptedResult(`ACCEPTED-CAS-${changed}`, runner => `${runner}:${randomUUID()}`); + const birth = new Date(Date.now() - 60_000); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, processStartedAt: birth }); + const replacement = randomUUID(); + const readProcessStartedAt = vi.fn(async () => { + await f.db.update(nativeRunFinalizations).set(changed === "lease-token" ? { leaseOwner: `new-owner:${replacement}` } + : changed === "boot" ? { controllerBootId: replacement } : { controllerHostHash: `new-space:${replacement}` }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + return new Date(birth.getTime() + 2_000); + }); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now: new Date(), runIds: [f.runId], + probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, + pidExists: () => true, readStartedAt: readProcessStartedAt } })).toEqual([]); + expect(readProcessStartedAt).toHaveBeenCalledExactlyOnceWith(2_000_000_001); + const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(owner.leaseExpiresAt).toEqual(f.leaseExpiresAt); + expect(changed === "lease-token" ? owner.leaseOwner : changed === "boot" ? owner.controllerBootId : owner.controllerHostHash) + .toBe(changed === "lease-token" ? `new-owner:${replacement}` : changed === "boot" ? replacement : `new-space:${replacement}`); + }, 60_000); + + it.each(["current-boot", "live-owner", "finalizer", "cleanup", "remote-sync", "remote-workspace"] as const)( + "never probes or releases excluded accepted-result authority: %s", async exclusion => { + const prefix = exclusion === "finalizer" ? "native-finalizer" : exclusion === "cleanup" ? "native-cleanup" : "executor"; + const f = await seedAcceptedResult(`ACCEPTED-EXCLUDE-${exclusion}`, () => `${prefix}:${randomUUID()}`); + const self = await currentNativeControllerIdentity(); + await killedController(f, { bootId: exclusion === "current-boot" ? self.bootId : randomUUID(), + pid: 2_000_000_001, processStartedAt: new Date(Date.now() - 60_000) }); + if (exclusion.startsWith("remote-")) await f.db.update(heartbeatRuns).set({ runnerProfileJson: { + ...f.preparedProfile, [exclusion === "remote-sync" ? "nativeWorkspaceSync" : "remoteWorkspace"]: {}, + } }).where(eq(heartbeatRuns.id, f.runId)); + const isProcessAlive = vi.fn(() => false), readProcessStartedAt = vi.fn(async () => null); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now: new Date(), runIds: [f.runId], probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt }, + skipRunIds: { has: runId => exclusion === "live-owner" && runId === f.runId } })).toEqual([]); + expect(isProcessAlive).not.toHaveBeenCalled(); expect(readProcessStartedAt).not.toHaveBeenCalled(); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].leaseExpiresAt) + .toEqual(f.leaseExpiresAt); + }, 60_000); + + it("finalizes a legacy NULL-proof accepted result by ordinary expiry without provider replay", async () => { + const f = await seedAcceptedResult("ACCEPTED-LEGACY-EXPIRED", runner => `${runner}:${randomUUID()}`); + await killedController(f, { bootId: randomUUID(), pid: 2_000_000_001, processStartedAt: null }); + await f.db.update(nativeRunFinalizations).set({ controllerHostHash: null, leaseExpiresAt: new Date(Date.now() - 1_000) }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const isProcessAlive = vi.fn(() => false), readProcessStartedAt = vi.fn(async () => null); + expect(await settleOwnableKind(f.db, "native_accepted_result_lease", { now: new Date(), runIds: [f.runId], probe: { ...currentProcessProbe(), deployment: null, controllerInstance: undefined, pidExists: isProcessAlive, readStartedAt: readProcessStartedAt } })).toEqual([]); + expect(isProcessAlive).not.toHaveBeenCalled(); expect(readProcessStartedAt).not.toHaveBeenCalled(); + const { run, owner } = await restart(f); + expect({ status: run.status, phase: owner.phase }).toEqual({ status: "succeeded", phase: "committed" }); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(1); + }, 90_000); + it("keeps the accepted result leased while its earlier-boot controller process is still alive", async () => { // Another live server (or an orphaned controller of an earlier boot) can // still be finalizing. Age and boot ID alone never prove that it stopped. @@ -428,6 +610,688 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut .toEqual({ status: "running", phase: "workspace_finalizing", leaseExpiresAt: f.leaseExpiresAt }); }, 60_000); + describe("restart claim process-space authority", () => { + const priorPid = 2_000_000_001; + async function seedClaim(label: string, expires: "live" | "expired", hash: string | null, isolatedAgent = false) { + const f = await seedRecoverableRun(label, isolatedAgent); + await f.db.update(nativeRunFinalizations).set({ + leaseOwner: `old-controller:${randomUUID()}`, leaseExpiresAt: new Date(Date.now() + (expires === "live" ? 1_200_000 : -1_000)), + controllerBootId: randomUUID(), controllerHostHash: hash, controllerPid: priorPid, + controllerProcessStartedAt: new Date(Date.now() - 60_000), controllerGeneration: 1, + }).where(eq(nativeRunFinalizations.runId, f.runId)); + return f; + } + function observeOwnerPid(alive = false) { + const original = process.kill.bind(process); + const observed: number[] = []; + const spy = vi.spyOn(process, "kill").mockImplementation((pid, signal) => { + if (pid === priorPid && signal === 0) { + observed.push(pid); + if (alive) return true; + throw Object.assign(new Error("controlled recorded owner absent in this PID space"), { code: "ESRCH" }); + } + return original(pid, signal); + }); + return { observed, restore: () => spy.mockRestore() }; + } + it.each([ + { mode: "null", expires: "live" }, { mode: "foreign", expires: "live" }, + { mode: "unknown-current", expires: "live" }, { mode: "foreign", expires: "expired" }, + ] as const)("holds $mode $expires local restart claim without an owner PID probe", async ({ mode, expires }) => { + const actual = await controllerProcess.currentControllerProcess(); + expect(actual.hostHash).not.toBe("unknown"); + const f = await seedClaim(`CLAIM-SPACE-${mode}-${expires}`, expires, + mode === "null" ? null : mode === "foreign" ? `foreign:${actual.hostHash}` : actual.hostHash); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(before.resultId).toBeNull(); + const unknown = mode === "unknown-current" ? vi.spyOn(controllerProcess, "currentProcessSpaceHash") + .mockReturnValue("unknown") : null; + const probe = observeOwnerPid(); + const onClaim = vi.fn(); + try { + const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }); + expect(probe.observed).toEqual([]); + expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_process_space_unverified" }]); + expect(onClaim).not.toHaveBeenCalled(); + const [after] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(after).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, + controllerBootId: before.controllerBootId, controllerPid: before.controllerPid, controllerHostHash: before.controllerHostHash, + controllerGeneration: before.controllerGeneration, attempt: before.attempt, phase: before.phase, resultId: null }); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + } finally { probe.restore(); unknown?.mockRestore(); } + }, 60_000); + + it("keeps a real controller-only attestation from authorizing an in-flight claim", async () => { + const f = await seedClaim("CLAIM-CONTROLLER-ONLY", "expired", "foreign-process-space"); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + await requestNativeControllerStopConfirmation(f.db, before); + const [action] = await f.db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.sourceIssueId, f.issueId), eq(issueRecoveryActions.status, "active"))); + expect(action.cause).toBe("native_controller_host_unverified"); + await expect(confirmNativeControllerStopped({ db: f.db, companyId: fixture.current.companyId, + issueId: f.issueId, runId: f.runId, actionId: action.id, actorId: "board-user", + stopEvidence: "The original controller stopped; provider effects have not been contained." })) + .resolves.toEqual({ runId: f.runId, status: "queued" }); + const probe = observeOwnerPid(), onClaim = vi.fn(); + try { + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_process_space_unverified" }]); + expect(probe.observed).toEqual([]); + expect(onClaim).not.toHaveBeenCalled(); + const [after] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(after).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, + controllerGeneration: before.controllerGeneration, attempt: before.attempt, resultId: null, + controllerHostHash: "operator_attested" }); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]) + .toMatchObject({ status: "running", resultJson: null }); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + } finally { probe.restore(); } + }, 60_000); + + it.each(["live", "expired"] as const)("claims a same-space dead owner with a %s lease and stamps the pending claim", async expires => { + const actual = await controllerProcess.currentControllerProcess(); + expect(actual.hostHash).not.toBe("unknown"); + const f = await seedClaim(`CLAIM-LOCAL-${expires}`, expires, actual.hostHash); + const probe = observeOwnerPid(); + const onClaim = vi.fn(); + try { + const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }); + expect(probe.observed).toEqual([priorPid]); + expect(onClaim).toHaveBeenCalledOnce(); + expect(dispositions).toHaveLength(1); + expect(dispositions[0]).toMatchObject({ runId: f.runId, leaseOwner: expect.any(String) }); + const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(owner.controllerHostHash).toBe(actual.hostHash); + expect(owner.controllerPid).toBe(process.pid); + expect(owner.controllerBootId).toBe((await currentNativeControllerIdentity()).bootId); + expect(owner.resultId).toBeNull(); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + } finally { probe.restore(); } + }, 60_000); + + it("keeps this boot's actual PID fenced even when its old stamp is NULL", async () => { + const f = await seedClaim("CLAIM-CURRENT-NULL", "live", null); + const current = await currentNativeControllerIdentity(); + await f.db.update(nativeRunFinalizations).set({ controllerBootId: current.bootId, + controllerPid: current.pid, controllerProcessStartedAt: current.processStartedAt }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const onClaim = vi.fn(); + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive" }]); + expect(onClaim).not.toHaveBeenCalled(); + }, 60_000); + + it("does not let a same-space expired lease replace a live owner", async () => { + const actual = await controllerProcess.currentControllerProcess(); + expect(actual.hostHash).not.toBe("unknown"); + const f = await seedClaim("CLAIM-EXPIRED-LIVE", "expired", actual.hostHash); + // A real current PID and birth keep the original evaluator's live-owner + // contract; no expiry-only takeover is introduced by the new fence. + const current = await currentNativeControllerIdentity(); + await f.db.update(nativeRunFinalizations).set({ controllerPid: current.pid, + controllerProcessStartedAt: current.processStartedAt }).where(eq(nativeRunFinalizations.runId, f.runId)); + const onClaim = vi.fn(); + const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }); + expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_still_alive" }]); + expect(onClaim).not.toHaveBeenCalled(); + }, 60_000); + + async function namespaceHold(label: string, live = false, isolatedAgent = false) { + const f = await seedClaim(label, live ? "live" : "expired", "foreign-process-space", isolatedAgent); + const [source] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + if (!source.responsibleUserId) throw new Error("A genuine responsible operator is required"); + await f.db.update(issues).set({ responsibleUserId: source.responsibleUserId }).where(eq(issues.id, f.issueId)); + await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] }); + const [action] = await f.db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, fixture.current.companyId), eq(issueRecoveryActions.sourceIssueId, f.issueId), + eq(issueRecoveryActions.status, "active"))); + const authority = action?.evidence.controllerAuthority; + if (!authority || typeof authority !== "object" || Array.isArray(authority) + || !("digest" in authority) || typeof authority.digest !== "string") throw new Error("Expected exact server-owned controller authority"); + return { ...f, action, digest: authority.digest, userId: source.responsibleUserId }; + } + function boardRoutes(f: Pick>, "db" | "userId">, actor?: Express.Request["actor"]) { + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { + req.actor = actor ?? { type: "board", userId: f.userId, source: "session", companyIds: [fixture.current.companyId], + memberships: [{ companyId: fixture.current.companyId, membershipRole: "owner", status: "active" }] }; + next(); + }); + app.use("/api", issueRoutes(f.db, createStorageService(createLocalDiskStorageProvider(fixture.current.runtimeRoot)))); + app.use(errorHandler); + return app; + } + const containmentEvidence = "Deployment operator verified the original host controller, provider and tools are stopped; external action outcomes were reviewed."; + function confirmation(f: Awaited>) { + return { actionId: f.action.id, outcome: "restored", sourceIssueStatus: "in_progress", + executionReconciliation: { runId: f.runId, controllerStopped: true, providerStopped: true, + controllerAuthorityDigest: f.digest, actionOutcome: "mixed", outcomeEvidence: containmentEvidence } }; + } + + it("keeps repeated namespace attention idempotent without spending a recovery budget", async () => { + const f = await namespaceHold("CLAIM-ATTENTION-IDEMPOTENT"); + const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + for (let scan = 0; scan < 3; scan++) await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] }); + const rows = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ id: f.action.id, attemptCount: 0, maxAttempts: null, wakePolicy: null, monitorPolicy: null, + updatedAt: f.action.updatedAt, lastAttemptAt: f.action.lastAttemptAt }); + expect(JSON.stringify(rows)).not.toContain(owner.leaseOwner!); + expect(rows[0].fingerprint).toBe(`native-controller-space:${f.runId}:${f.digest}`); + }, 60_000); + + it("supersedes exact namespace attention after a microsecond renewal without overwriting its history", async () => { + const f = await namespaceHold("CLAIM-ATTENTION-RENEWAL"); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`${nativeRunFinalizations.leaseExpiresAt} + interval '1 microsecond'` }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const [renewed] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(renewed.leaseOwner).toBe(before.leaseOwner); + expect(renewed.leaseExpiresAt).toEqual(before.leaseExpiresAt); // JS drops the changed microsecond. + await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] }); + const rows = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(rows).toHaveLength(2); + expect(rows.find(row => row.id === f.action.id)).toMatchObject({ status: "cancelled", evidence: f.action.evidence }); + const current = rows.find(row => row.status === "active"); + expect(current).toMatchObject({ attemptCount: 0, maxAttempts: null, wakePolicy: null, monitorPolicy: null }); + expect(current?.fingerprint).not.toBe(f.action.fingerprint); + }, 60_000); + + it("does not overwrite an unrelated active recovery identity with namespace attention", async () => { + const f = await seedClaim("CLAIM-OTHER-ATTENTION", "expired", "foreign-process-space"); + const [prior] = await f.db.insert(issueRecoveryActions).values({ companyId: fixture.current.companyId, + sourceIssueId: f.issueId, kind: "active_run_watchdog", ownerType: "board", cause: "uncertain_external_action", + fingerprint: `other:${f.runId}`, evidence: { runId: f.runId, originalEvidence: "keep this exact incident" }, + nextAction: "Review this independent external action first.", attemptCount: 2, maxAttempts: 3 }).returning(); + await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] }); + expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual([prior]); + }, 60_000); + + it("rejects a stale dialog after a same-token microsecond-only lease renewal", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-MICROSECOND-CAS"); + await f.db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`${nativeRunFinalizations.leaseExpiresAt} + interval '1 microsecond'` }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect(response.body.details?.code).toBe("native_controller_containment_stale"); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + expect((await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)))[0].status).toBe("active"); + }, 60_000); + + it.each(["token", "boot", "generation", "namespace", "assignee", "renewed-live-lease"] as const) + ("rejects controller containment after the %s fence changes", async changed => { + const f = await namespaceHold(`CLAIM-CONTAINMENT-${changed}`); + if (changed === "assignee") await f.db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId)); + else await f.db.update(nativeRunFinalizations).set(changed === "token" ? { leaseOwner: randomUUID() } + : changed === "boot" ? { controllerBootId: randomUUID() } + : changed === "generation" ? { controllerGeneration: sql`${nativeRunFinalizations.controllerGeneration} + 1` } + : changed === "namespace" ? { controllerHostHash: "replaced-space" } : { leaseExpiresAt: new Date(Date.now() + 60_000) }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + }, 60_000); + + it("rejects the old provider-only confirmation for the new controller hold", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-EXPLICIT-CONTROLLER"); + const body = confirmation(f); + const { controllerStopped: _stopped, controllerAuthorityDigest: _digest, ...providerOnly } = body.executionReconciliation; + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`) + .send({ ...body, sourceIssueStatus: "todo", executionReconciliation: providerOnly }); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].leaseOwner).not.toBeNull(); + }, 60_000); + + it("rejects unknown action outcomes instead of turning containment into replay permission", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-UNKNOWN-OUTCOME"); + const body = confirmation(f); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`) + .send({ ...body, executionReconciliation: { ...body.executionReconciliation, actionOutcome: "unknown" } }); + expect(response.status, JSON.stringify(response.body)).toBe(400); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + }, 60_000); + + it.each(["native-finalizer", "native-cleanup"] as const)("never uses an executor's PID to claim %s authority", async holder => { + const actual = await controllerProcess.currentControllerProcess(); + const f = await seedClaim(`CLAIM-NONEXECUTOR-${holder}`, "expired", actual.hostHash); + await f.db.update(nativeRunFinalizations).set({ leaseOwner: `${holder}:${randomUUID()}` }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + const probe = observeOwnerPid(); + try { + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "native_finalization_or_cleanup_owner" }]); + expect(probe.observed).toEqual([]); + expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual([]); + } finally { probe.restore(); } + }, 60_000); + + it("keeps remote-marked no-checkpoint ownership out of the local containment form", async () => { + const f = await seedClaim("CLAIM-REMOTE-NO-CHECKPOINT", "expired", "foreign-process-space"); + // The claim route treats any retained remote-workspace field as remote; + // an incomplete descriptor must never grant a local containment bypass. + await f.db.update(heartbeatRuns).set({ runnerProfileJson: { ...f.preparedProfile, remoteWorkspace: { state: "prepared" } } }) + .where(eq(heartbeatRuns.id, f.runId)); + const probe = observeOwnerPid(); + try { + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "remote_controller_process_space_unverified" }]); + expect(probe.observed).toEqual([]); + const [action] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(action.evidence.controllerAuthority).toBeUndefined(); + expect(action.nextAction).toContain("ordinary Stop"); + } finally { probe.restore(); } + }, 60_000); + + // A controlled provider uses the normal SDK event/result contract. It does + // not write an accepted result, task status, usage, or finalization directly. + function retryBackend(db: Awaited>["db"], opened: string[]) { + const factory: NonNullable = execution => { + const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; + let identity: NativeRunIdentity; + let sourceInstanceId = ""; + const turnId = randomUUID(); + let begin: () => void = () => {}; + const started = new Promise(resolve => { begin = resolve; }); + const result = { ...CONTROL_PLANE_CONFORMANCE_RESULT, summary: "The explicitly retried task produced this useful completion.", + completionClaim: { ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, + contractRevision: execution.completionContract.contract.revision, + criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, + status: "satisfied" as const, evidenceRefs: ["event:2"] })) } }; + const session: NativeSession = { + identity: () => identity, capabilities: async () => capabilities, + startTurn: async () => { begin(); return { turnId }; }, + async *events() { + await started; + const event = (sourceSeq: number, eventType: PrpEvent["eventType"], payload: Record): PrpEvent => ({ + schema: "paperclip.prp.event.v1", sourceKind: "runner", sourceInstanceId, + sourceEventId: `${sourceInstanceId}:${sourceSeq}`, sourceSeq, runId: identity.runId, + normalizedSessionId: identity.sessionId, turnId, schemaVersion: 1, priority: 0, + emittedAt: new Date().toISOString(), eventType, payload }); + yield event(1, "turn.started", {}); + yield event(2, "item.completed", { kind: "agentMessage", channel: "final", text: result.summary }); + yield event(3, "turn.completed", { status: "completed" }); + }, + result: async () => ({ result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }), + snapshot: async () => ({ backendKind: "mock", sessionId: "contained-retry-session", identity, + providerSessionId: "contained-retry-provider", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }), + close: async () => {}, + }; + return { descriptor: async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, execution.binding.runId)); + if (!run.runnerInstanceId) throw new Error("A real admitted runner identity is required"); + sourceInstanceId = run.runnerInstanceId; + return { kind: "mock", name: "contained-retry", version: "1", capabilities, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; + }, openSession: async input => { identity = input.identity; opened.push(identity.runId); return session; }, + recoverSession: async snapshot => { identity = snapshot.identity; opened.push(identity.runId); return { recovered: true, session }; } }; + }; + return factory; + } + + async function completeContainedRetry(f: Awaited>) { + const [before] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const oldClick = await requestTaskRetry({ db: f.db, companyId: fixture.current.companyId, issueId: f.issueId, + actorId: f.userId, actorSource: "session", observedRunId: f.runId, idempotencyKey: randomUUID() }); + expect(oldClick.accepted).toBe(true); + const opened: string[] = []; + const services: ReturnType[] = []; + const actualFactory = heartbeatModule.heartbeatService; + const factory = vi.spyOn(heartbeatModule, "heartbeatService").mockImplementation((db, options) => { + const service = actualFactory(db, { ...options, nativeSessionBackendFactory: retryBackend(db, opened) }); + services.push(service); return service; + }); + try { + const app = boardRoutes(f); + const [currentTask] = await f.db.select().from(issues).where(eq(issues.id, f.issueId)); + const response = await request(app).post(`/api/issues/${f.issueId}/recovery-actions/resolve`) + .send({ ...confirmation(f), sourceIssueStatus: currentTask.status }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + const [source] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(source).toMatchObject({ status: "failed", usageJson: before.usageJson, costAccountingPending: before.costAccountingPending }); + const [coordinator] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(coordinator).toMatchObject({ phase: "terminal_failure", resultId: null, leaseOwner: null, + failureDetail: { releasedOwner: { actionOutcome: "mixed", outcomeEvidence: containmentEvidence } } }); + const [action] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)); + expect(action.evidence.continuationDelivery).toBe("not_requested"); + expect(action.evidence.executionReconciliation).toMatchObject({ controllerAuthorityDigest: f.digest }); + expect(action.evidence.executionReconciliation).toMatchObject({ controllerStopped: true, providerStopped: true, + actionOutcome: "mixed", outcomeEvidence: containmentEvidence }); + const [old] = await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, oldClick.requestId!)); + // A Retry while the source is running already coalesced into that + // accepted run. It is not an unadmitted receipt and must stay history. + expect(old).toMatchObject(before.status === "running" + ? { status: "coalesced", runId: f.runId } + : { status: "skipped", runId: null, error: "controller_containment_requires_new_retry" }); + expect(opened).toEqual([]); + expect(await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1); + const retried = await request(app).post(`/api/issues/${f.issueId}/retry`).send({ observedRunId: f.runId, idempotencyKey: randomUUID() }); + expect(retried.status, JSON.stringify(retried.body)).toBe(202); + expect(retried.body.requestId).not.toBe(oldClick.requestId); + await waitForCondition("normal explicit Retry terminal successor and wakes to settle", async () => { + const [wake] = await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retried.body.requestId)); + if (!wake?.runId || wake.runId === f.runId) return false; + const [successor] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId)); + return !!successor?.finishedAt + && ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(successor.status) + && services.every(service => service.getTaskDrainStatus().quiescent); + }); + const [wake] = await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retried.body.requestId)); + expect(wake.runId).not.toBeNull(); + expect(wake.runId).not.toBe(f.runId); + expect(opened).toEqual([wake.runId]); + const [successor] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, wake.runId!)); + expect(successor.status).toBe("succeeded"); + expect(successor.contextSnapshot?.executionContinuation).toMatchObject({ recoveryOutcomes: expect.arrayContaining([ + { recoveryActionId: f.action.id, decision: expect.objectContaining({ actionOutcome: "mixed", outcomeEvidence: containmentEvidence }) }, + ]) }); + expect(successor.resultJson?.nativeResult).toMatchObject({ summary: "The explicitly retried task produced this useful completion." }); + const [task] = await f.db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(task.status).toBe("done"); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, successor.id))).toHaveLength(1); + } finally { + factory.mockRestore(); + for (const service of services) await service.drainActiveRunExecutions(); + } + } + + it("records original-host containment then completes one useful successor only after a new ordinary Retry", async () => { + await completeContainedRetry(await namespaceHold("CLAIM-CONTAINMENT-ORDINARY-RETRY", false, true)); + }, 90_000); + + /** Historical INITIAL fixture, not a failed execution manufactured by the + * tested owners. Preparation produced the normal contract/profile; these + * are the complete persisted abandoned-source facts at journey start: + * failed+finished, setup_failed, no result/usage added, cleared issue and + * checkout pointers, same assignee/responsible operator, observed resultless + * expired foreign claim with its original token/generation/PID/birth. */ + async function seedHistoricalFailedForeignSource(label: string, isolatedAgent = false) { + const f = await seedClaim(label, "expired", "foreign-process-space", isolatedAgent); + const [source] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + if (!source.responsibleUserId) throw new Error("A genuine responsible operator is required"); + await f.db.transaction(async tx => { + await tx.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), errorCode: "setup_failed", + error: "Historical pre-executor failure with retained foreign ownership", controllerLeaseExpiresAt: null }) + .where(eq(heartbeatRuns.id, f.runId)); + await tx.update(issues).set({ executionRunId: null, checkoutRunId: null, responsibleUserId: source.responsibleUserId }) + .where(eq(issues.id, f.issueId)); + }); + const [initialRun] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [initialOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const [initialTask] = await f.db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(initialRun).toMatchObject({ status: "failed", errorCode: "setup_failed", usageJson: source.usageJson }); + expect(initialRun.finishedAt).not.toBeNull(); + expect(initialOwner).toMatchObject({ phase: "observed", resultId: null, controllerHostHash: "foreign-process-space" }); + expect(initialOwner.leaseOwner).not.toBeNull(); + expect(initialOwner.leaseExpiresAt!.getTime()).toBeLessThan(Date.now()); + expect(initialTask).toMatchObject({ executionRunId: null, checkoutRunId: null, assigneeAgentId: initialRun.agentId }); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + return { ...f, userId: source.responsibleUserId, initialRun, initialOwner, initialTask }; + } + + async function inspectHistoricalFailedSource(f: Awaited>) { + // Actual normal startup/mutation finalization duty. No status, token, + // lease, provider result or usage is rewritten by this fixture now. + const inspected = await reconcileAbandonedExecutionControl(f.db); + expect(inspected.releasedOwners).toBe(0); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(f.initialOwner); + const [action] = await f.db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId), + eq(issueRecoveryActions.status, "active"))); + const authority = action?.evidence.controllerAuthority; + if (!authority || typeof authority !== "object" || Array.isArray(authority) + || !("digest" in authority) || typeof authority.digest !== "string") + throw new Error("Normal historical failed-owner inspection must publish exact containment authority"); + return { ...f, action, digest: authority.digest }; + } + + it("contains a historical already-failed foreign source with cleared pointer then completes a new ordinary Retry", async () => { + const initial = await seedHistoricalFailedForeignSource("CLAIM-HISTORICAL-FAILED-CLEARED-RETRY", true); + // The journey begins after the declared INITIAL state above exists. + await completeContainedRetry(await inspectHistoricalFailedSource(initial)); + }, 90_000); + + it("rejects cleared-pointer containment when a newer canonical Native coordinator binds the issue", async () => { + const f = await inspectHistoricalFailedSource(await seedHistoricalFailedForeignSource("CLAIM-CONTAINMENT-NEWER-COORDINATOR")); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const newerId = randomUUID(); + await f.db.insert(heartbeatRuns).values({ id: newerId, companyId: fixture.current.companyId, + agentId: fixture.current.agentId, status: "queued", runtimeMode: "native", contextSnapshot: { issueId: f.issueId }, nativeIssueId: f.issueId }); + await f.db.insert(nativeRunFinalizations).values({ runId: newerId, companyId: fixture.current.companyId, + issueId: f.issueId, phase: "observed", attempt: 0 }); + const [currentTask] = await f.db.select().from(issues).where(eq(issues.id, f.issueId)); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`) + .send({ ...confirmation(f), sourceIssueStatus: currentTask.status }); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(before); + expect((await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)))[0].status).toBe("active"); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toEqual([]); + }, 90_000); + + it("automatically completes one useful successor after a first-admission preparation failure without containment", async () => { + const isolated = await freshExecutionAgent(); + const userId = randomUUID(); + await isolated.db.insert(authUsers).values({ id: userId, name: "Operator", email: `${userId}@example.invalid`, + createdAt: new Date(), updatedAt: new Date() }); + await isolated.db.insert(companyMemberships).values({ companyId: fixture.current.companyId, + principalType: "user", principalId: userId, membershipRole: "owner" }); + await instanceSettingsService(isolated.db).updateExperimental({ enableNativeRunner: true }); + const [task] = await isolated.db.insert(issues).values({ companyId: fixture.current.companyId, + title: "Recover a genuine first preparation failure", status: "in_progress", workMode: "standard", + assigneeAgentId: isolated.agentId, responsibleUserId: userId }).returning(); + const opened: string[] = []; + const backend = vi.fn(retryBackend(isolated.db, opened)); + let failedPreparationRunId: string | null = null; + const service = heartbeatService(isolated.db, { nativeSessionBackendFactory: backend, + beforeNativeRuntimeSelection: async runId => { + // A single dependency failure belongs to the original admission. The + // normal automatic successor must use the real controlled SDK backend. + if (failedPreparationRunId !== null) return; + failedPreparationRunId = runId; + expect(await isolated.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId))).toEqual([]); + throw new Error("Controlled first-admission preparation failure before native selection and provider dispatch"); + } }); + try { + const admitted = await service.wakeup(isolated.agentId, { source: "on_demand", triggerDetail: "manual", + reason: "issue_assigned", requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true, + payload: { issueId: task.id }, contextSnapshot: { issueId: task.id, actorId: userId, responsibleUserId: userId } }); + expect(admitted).not.toBeNull(); + await waitForCondition("normal automatic preparation-recovery terminal successor and wakes to settle", async () => { + const successors = await isolated.db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, fixture.current.companyId), eq(heartbeatRuns.agentId, isolated.agentId), + eq(heartbeatRuns.retryOfRunId, admitted!.id))); + return successors.length === 1 && !!successors[0].finishedAt + && ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(successors[0].status) + && service.getTaskDrainStatus().quiescent; + }); + expect(failedPreparationRunId).toBe(admitted!.id); + const [source] = await isolated.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, admitted!.id)); + expect(source).toMatchObject({ status: "failed", responsibleUserId: userId, processPid: null }); + expect(source.resultJson?.executionRecovery).toMatchObject({ kind: "bootstrap", providerWorkStarted: false }); + expect(await isolated.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, source.id))).toEqual([]); + const successors = await isolated.db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, fixture.current.companyId), eq(heartbeatRuns.agentId, isolated.agentId), + eq(heartbeatRuns.retryOfRunId, source.id))); + expect(successors).toHaveLength(1); + const successor = successors[0]; + expect(successor).toMatchObject({ status: "succeeded", retryOfRunId: source.id, + responsibleUserId: userId, nativeIssueId: task.id }); + expect(successor.id).not.toBe(source.id); + expect(opened).toEqual([successor.id]); + expect(backend.mock.calls.map(([execution]) => execution.binding.runId)).not.toContain(source.id); + expect(successor.resultJson?.nativeResult).toMatchObject({ summary: "The explicitly retried task produced this useful completion." }); + const wakes = await isolated.db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, fixture.current.companyId), eq(agentWakeupRequests.runId, successor.id), + eq(agentWakeupRequests.reason, "issue_continuation_needed"))); + expect(wakes).toHaveLength(1); + expect((await isolated.db.select().from(issues).where(eq(issues.id, task.id)))[0].status).toBe("done"); + expect(await isolated.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, source.id))).toEqual([]); + expect(await isolated.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, successor.id))).toHaveLength(1); + const actions = await isolated.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, task.id)); + expect(actions.every(action => action.evidence.reason !== "controller_process_space_unverified" + && action.evidence.controllerAuthority === undefined)).toBe(true); + } finally { await service.drainActiveRunExecutions(); } + }, 90_000); + + it("requires new Retry only for exact unadmitted receipts and keeps other accepted work", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-UNADMITTED"); + // Existing failed-owner fixtures model the supported failure outside the + // executor: status failed with the result-less observed claim still held. + // No provider outcome, accepted result or cost is supplied by this setup. + await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), errorCode: "native_session_resume_failed" }) + .where(eq(heartbeatRuns.id, f.runId)); + const old = await requestTaskRetry({ db: f.db, companyId: fixture.current.companyId, issueId: f.issueId, + actorId: f.userId, actorSource: "session", observedRunId: f.runId, idempotencyKey: randomUUID() }); + const [before] = await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, old.requestId!)); + expect(before).toMatchObject({ status: "deferred_issue_execution", runId: null }); + const other = await namespaceHold("CLAIM-OTHER-ACCEPTED-WORK"); + const accepted = await requestTaskRetry({ db: other.db, companyId: fixture.current.companyId, issueId: other.issueId, + actorId: other.userId, actorSource: "session", observedRunId: other.runId, idempotencyKey: randomUUID() }); + const [unrelatedBefore] = await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, accepted.requestId!)); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect((await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, old.requestId!)))[0]) + .toMatchObject({ status: "skipped", runId: null, error: "controller_containment_requires_new_retry", payload: before.payload }); + expect((await f.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, accepted.requestId!)))[0]).toEqual(unrelatedBefore); + const next = await requestTaskRetry({ db: f.db, companyId: fixture.current.companyId, issueId: f.issueId, + actorId: f.userId, actorSource: "session", observedRunId: f.runId, idempotencyKey: randomUUID() }); + expect(next.accepted).toBe(true); + expect(next.requestId).not.toBe(old.requestId); + }, 60_000); + + it.each(["environment", "workspace"] as const)("retains containment authority while %s cleanup is unsettled", async kind => { + const f = await namespaceHold(`CLAIM-CONTAINMENT-CLEANUP-${kind}`); + if (kind === "environment") await f.db.insert(environmentLeases).values({ companyId: fixture.current.companyId, + issueId: f.issueId, heartbeatRunId: f.runId, provider: "local", status: "active" }); + else await f.db.insert(workspaceOperations).values({ companyId: fixture.current.companyId, issueId: f.issueId, + heartbeatRunId: f.runId, phase: "workspace_finalize", status: "running", cwd: fixture.current.runtimeRoot }); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running"); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].leaseOwner).not.toBeNull(); + }, 60_000); + + it("preserves a result actually accepted after the namespace attention was published", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-ACCEPTED-RESULT", true); + const [contract] = await f.db.select().from(completionContracts).where(eq(completionContracts.issueId, f.issueId)); + await f.db.update(heartbeatRuns).set({ completionContractId: contract.id, completionContractSha256: contract.canonicalSha256 }) + .where(eq(heartbeatRuns.id, f.runId)); + const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const identity = { companyId: fixture.current.companyId, issueId: f.issueId, runId: f.runId, + agentId: fixture.current.agentId, sessionId: run.nativeSessionId! }; + const port = new PaperclipControlPlanePort(f.db, { ...identity, completionContractId: contract.id, + completionContractSha256: contract.canonicalSha256, sourceInstanceId: run.runnerInstanceId!, + controlPlaneSourceInstanceId: `accepted-after-attention:${f.runId}` }); + await port.openRun({ ...CONTROL_PLANE_CONFORMANCE_OPEN, identity, sourceInstanceId: run.runnerInstanceId! }); + await port.completeRun({ result: { ...CONTROL_PLANE_CONFORMANCE_RESULT, completionClaim: { + ...CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim, contractRevision: f.native.completionContract.revision, + criteria: f.native.completionContract.criterionIds.map(criterionId => ({ criterionId, status: "satisfied" as const, evidenceRefs: ["event:2"] })), + } }, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, callerResultId: `accepted-after-attention:${f.runId}` }); + const [accepted] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(accepted.resultId).not.toBeNull(); + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(409); + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0].resultId).toBe(accepted.resultId); + expect(await f.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(1); + }, 60_000); + + it.each([false, true] as const)("keeps a correlated Stop from automatically releasing foreign ownership (ephemeral: %s)", async ephemeral => { + const previous = process.env.PAPERCLIP_EPHEMERAL_SERVER; + if (ephemeral) process.env.PAPERCLIP_EPHEMERAL_SERVER = "true"; + else delete process.env.PAPERCLIP_EPHEMERAL_SERVER; + try { + const f = await namespaceHold(`CLAIM-CONTAINMENT-STOP-${ephemeral}`); + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const requestId = randomUUID(); + await claimCancellationRequest(f.db, f.runId, fixture.current.companyId, requestId, f.userId); + const cancellation = cancelHeartbeatNativeRun({ db: f.db, runId: f.runId, reason: "Operator requested Stop", + runtimeMode: "native", cancellationRequestId: requestId }); + if (ephemeral) { + // The actual frontend waits its existing20s for the real owner's ACK; + // this isolated foreign-owner fixture has no local delivery handle. + await expect(cancellation).rejects.toThrow("native_cancellation_pending_recovery"); + } else { + const acknowledgement = await cancellation; + expect(acknowledgement).toMatchObject({ dispatched: false }); + expect(acknowledgement.auditId).toEqual(expect.any(String)); + } + const [pending] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + expect(pending.status).toBe("running"); + expect(pending.resultJson?.startupCancellation).toMatchObject({ cancellationRequestId: requestId, + requestedBy: { type: "board", userId: f.userId } }); + const intent = pending.resultJson?.nativeCancellation; + expect(intent).toMatchObject({ dispatchState: ephemeral ? "pending" : "acknowledged", dispatched: false }); + if (!intent || typeof intent !== "object" || Array.isArray(intent) || !("intentAuditId" in intent)) + throw new Error("The normal cancellation owner must persist an audited intent"); + expect(intent.intentAuditId).toEqual(expect.any(String)); + if (!ephemeral) { + if (!("acknowledgementAuditId" in intent)) throw new Error("The local no-op acknowledgement must have its own audit"); + expect(intent.acknowledgementAuditId).toEqual(expect.any(String)); + expect(intent.acknowledgementAuditId).not.toBe(intent.intentAuditId); + } + if (ephemeral) { + const response = await request(boardRoutes(f)).post(`/api/issues/${f.issueId}/recovery-actions/resolve`).send(confirmation(f)); + expect(response.status, JSON.stringify(response.body)).toBe(409); + } + expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(before); + expect((await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.action.id)))[0].status).toBe("active"); + } finally { + if (previous === undefined) delete process.env.PAPERCLIP_EPHEMERAL_SERVER; + else process.env.PAPERCLIP_EPHEMERAL_SERVER = previous; + } + }, 60_000); + + it("keeps the authority digest stable across PostgreSQL session time zones", async () => { + const f = await namespaceHold("CLAIM-CONTAINMENT-TIMEZONE"); + await f.db.transaction(async tx => { + await tx.execute(sql`set local time zone 'Pacific/Honolulu'`); + const [row] = await tx.select({ owner: nativeRunFinalizations, + expiryEpoch: sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` }) + .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(nativeControllerAuthorityDigest(row.owner, row.expiryEpoch)).toBe(f.digest); + }); + }, 60_000); + + it("keeps plugin-runner leases without a checkpoint in their normal Stop recovery", async () => { + const f = await seedClaim("CLAIM-PLUGIN-NO-CHECKPOINT", "expired", "foreign-process-space"); + await f.db.insert(environmentLeases).values({ companyId: fixture.current.companyId, issueId: f.issueId, + heartbeatRunId: f.runId, provider: "plugin", status: "active", metadata: { runnerTaskLease: true } }); + const probe = observeOwnerPid(); + try { + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "remote_controller_process_space_unverified" }]); + expect(probe.observed).toEqual([]); + const [action] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(action.evidence.controllerAuthority).toBeUndefined(); + expect(action.nextAction).toContain("ordinary Stop"); + } finally { probe.restore(); } + }, 60_000); + + it("retires exact namespace attention when real current identity becomes readable and normal recovery claims", async () => { + const actual = await controllerProcess.currentControllerProcess(); + expect(actual.hostHash).not.toBe("unknown"); + const f = await seedClaim("CLAIM-SPACE-READABLE-AGAIN", "expired", actual.hostHash); + // Degrade evidence for the first scan; never fabricate a stronger hash. + const unreadable = vi.spyOn(controllerProcess, "currentProcessSpaceHash").mockReturnValue("unknown"); + try { + expect(await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId] })) + .toEqual([{ kind: "awaiting_evidence", runId: f.runId, reason: "controller_process_space_unverified" }]); + } finally { unreadable.mockRestore(); } + const [attention] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + const probe = observeOwnerPid(); + const onClaim = vi.fn(); + try { + await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], onClaim }); + expect(onClaim).toHaveBeenCalledOnce(); + expect(probe.observed).toEqual([priorPid]); + const [settled] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, attention.id)); + expect(settled).toMatchObject({ status: "resolved", outcome: "restored", evidence: attention.evidence }); + expect(settled.evidence.executionReconciliation).toBeUndefined(); + } finally { probe.restore(); } + }, 60_000); + }); + // Phase 2: the accepted-result lease settles on the shared ownership path // (proveOwnerGone + registry settle), never by lease age. const deadController = (hostHash: string | null) => ({ bootId: randomUUID(), pid: 2_000_000_001, @@ -443,12 +1307,15 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut expect(await legacyProofs(f)).toHaveLength(0); }, 90_000); - it("records each use of the one-release fallback for a controller saved before its PID space", async () => { + it("keeps an accepted Native result without a recorded PID space out of the generic single-host fallback", async () => { const f = await seedAcceptedResult("ACCEPTED-LEGACY-DEAD", (runner) => `${runner}:${randomUUID()}`); await killedController(f, deadController(null)); - const { run } = await restart(f); - expect(run.status).toBe("succeeded"); - expect(await legacyProofs(f)).toHaveLength(1); + const { run, owner } = await restart(f); + expect({ status: run.status, phase: owner.phase, leaseExpiresAt: owner.leaseExpiresAt }) + .toEqual({ status: "running", phase: "workspace_finalizing", leaseExpiresAt: f.leaseExpiresAt }); + expect(await legacyProofs(f)).toHaveLength(0); + expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))) + .toEqual([expect.objectContaining({ cause: "native_controller_host_unverified", status: "active" })]); }, 90_000); it("keeps a result leased to a dead PID recorded in another PID space: another host is never proof", async () => { @@ -464,7 +1331,7 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut await killedController(f, deadController(null)); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); try { - const report = await runOwnershipReconciliationPass(f.db, { state: createOwnershipReportState() }); + const report = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state: createOwnershipReportState() }); expect(report.kinds.find(kind => kind.kind === "native_accepted_result_lease")) .toMatchObject({ settled: 0, reasons: { process_space_unrecorded: 1 } }); } finally { vi.unstubAllEnvs(); } @@ -475,11 +1342,39 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut .toEqual([["native_controller_host_unverified", "board", "active"]]); }, 60_000); - it("finalizes the result once the board attests the unrecorded controller stopped", async () => { + it("finalizes the accepted result after the board confirms its actual unrecorded-controller action", async () => { const f = await seedAcceptedResult("ACCEPTED-OPERATOR-ATTESTED", (runner) => `${runner}:${randomUUID()}`); - await killedController(f, deadController("operator_attested")); - const { run } = await restart(f); - expect(run.status).toBe("succeeded"); + await killedController(f, deadController(null)); + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + try { + const [before] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + const report = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state: createOwnershipReportState() }); + expect(report.kinds.find(kind => kind.kind === "native_accepted_result_lease")) + .toMatchObject({ settled: 0, reasons: { process_space_unrecorded: 1 } }); + const [action] = await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(action).toMatchObject({ cause: "native_controller_host_unverified", status: "active" }); + await expect(confirmNativeControllerStopped({ db: f.db, companyId: fixture.current.companyId, + issueId: f.issueId, runId: f.runId, actionId: action.id, actorId: "board-user", + stopEvidence: "The original server container is stopped; the accepted provider result is retained." })) + .resolves.toEqual({ runId: f.runId, status: "queued" }); + const [attested] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + expect(attested).toMatchObject({ phase: before.phase, resultId: before.resultId, leaseOwner: before.leaseOwner, + leaseExpiresAt: before.leaseExpiresAt, controllerGeneration: before.controllerGeneration, + attempt: before.attempt, controllerHostHash: "operator_attested" }); + expect(await f.db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, f.issueId), + eq(issueRecoveryActions.status, "active")))).toEqual([]); + const { run, owner } = await restart(f); + expect({ status: run.status, phase: owner.phase, resultId: owner.resultId }) + .toEqual({ status: "succeeded", phase: "committed", resultId: before.resultId }); + const second = await restart(f); + expect(second.owner.resultId).toBe(before.resultId); + expect(await f.db.select({ id: nativeRunResults.id }).from(nativeRunResults) + .where(eq(nativeRunResults.runId, f.runId))).toHaveLength(1); + expect(await f.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1); + expect((await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0]) + .toMatchObject({ status: "resolved", outcome: "restored" }); + } finally { vi.unstubAllEnvs(); } }, 90_000); it("settles a dead controller's result lease in the reconciler pass when restart recovery missed it", async () => { @@ -488,18 +1383,18 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut // before the row, or failed). The next reconciler pass settles it. const f = await seedAcceptedResult("ACCEPTED-RECONCILER-PASS", (runner) => `${runner}:${randomUUID()}`); await killedController(f, deadController(currentProcessSpaceHash())); - const readOnly = await runOwnershipReconciliationPass(f.db, { state: createOwnershipReportState(), record: false }); + const readOnly = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state: createOwnershipReportState(), record: false }); expect(readOnly.kinds.find(kind => kind.kind === "native_accepted_result_lease")) .toMatchObject({ settled: 0, verdicts: { gone: 1 } }); const [held] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); expect(held.leaseExpiresAt).toEqual(f.leaseExpiresAt); const state = createOwnershipReportState(); - const first = await runOwnershipReconciliationPass(f.db, { state }); + const first = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state }); expect(first.kinds.find(kind => kind.kind === "native_accepted_result_lease")).toMatchObject({ settled: 1 }); const [released] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); expect(released.leaseExpiresAt!.getTime()).toBeLessThanOrEqual(Date.now()); // Settled records are no longer in progress, so rule I2 never fires. - const second = await runOwnershipReconciliationPass(f.db, { state }); + const second = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state }); expect(second.violations.filter(violation => violation.kind === "native_accepted_result_lease")).toEqual([]); }, 60_000); @@ -508,7 +1403,7 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut const self = await currentNativeControllerIdentity(); await killedController(f, { bootId: randomUUID(), pid: process.pid, processStartedAt: self.processStartedAt, hostHash: currentProcessSpaceHash() }); - const report = await runOwnershipReconciliationPass(f.db, { state: createOwnershipReportState() }); + const report = await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state: createOwnershipReportState() }); expect(report.kinds.find(kind => kind.kind === "native_accepted_result_lease")) .toMatchObject({ settled: 0, verdicts: { alive: 1 } }); const [owner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); @@ -524,7 +1419,7 @@ describeEmbeddedPostgres("heartbeat native recovery interrupted by graceful shut try { const kind = OWNABLE_KINDS.find(candidate => candidate.id === "native_accepted_result_lease")!; expect(await kind.listForRuns!(f.db, [f.runId])).toEqual([]); - await runOwnershipReconciliationPass(f.db, { state: createOwnershipReportState() }); + await runOwnershipReconciliationPass(f.db, { kinds: acceptedRunKinds(f), state: createOwnershipReportState() }); } finally { vi.unstubAllEnvs(); } expect(await f.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId))).toEqual([]); }, 60_000); diff --git a/server/src/__tests__/heartbeat-observed-workspace-preservation.test.ts b/server/src/__tests__/heartbeat-observed-workspace-preservation.test.ts index b5bd1bf553c..ea8f5b66267 100644 --- a/server/src/__tests__/heartbeat-observed-workspace-preservation.test.ts +++ b/server/src/__tests__/heartbeat-observed-workspace-preservation.test.ts @@ -1,3 +1,4 @@ +import { currentControllerProcess } from "../services/controller-process-identity.js"; import { execFile } from "node:child_process"; import { randomUUID } from "node:crypto"; import { cp, mkdir, readFile, realpath, writeFile } from "node:fs/promises"; @@ -313,8 +314,15 @@ describeEmbeddedPostgres("stopped observed native workspace preservation", () => it.each(["before_pin", "after_pin"] as const)("serializes the real ephemeral checkpoint claim %s", async order => { const f = await seedSource(`CLAIM-${order}`); - if (order === "before_pin") await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId)); - else expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true); + if (order === "before_pin") { + // This branch exercises a same-host checkpoint claim, not a legacy or + // foreign namespace. Keep every other historical preservation fixture unchanged. + const controller = await currentControllerProcess(); + expect(controller.hostHash).not.toBe("unknown"); + await f.db.update(nativeRunFinalizations).set({ controllerHostHash: controller.hostHash }) + .where(eq(nativeRunFinalizations.runId, f.runId)); + await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId)); + } else expect(await pinFailedObservedWorkspace(f.db, companyId, f.runId)).toBe(true); const dispositions = await claimNativeRestartRecoveries({ db: f.db, restartKind: "hard", runIds: [f.runId], controller: fixture.current.successor }); if (order === "before_pin") { expect(dispositions).toMatchObject([{ kind: "reattach_remote_runner", runId: f.runId }]); diff --git a/server/src/__tests__/helpers/native-restart-fixture.ts b/server/src/__tests__/helpers/native-restart-fixture.ts index 2b6ced75946..6c3b382e959 100644 --- a/server/src/__tests__/helpers/native-restart-fixture.ts +++ b/server/src/__tests__/helpers/native-restart-fixture.ts @@ -157,8 +157,10 @@ export function nativeRestartFixture(options: { dedicatedDatabaseUrl?: string } async function seedRun( label: string, existingDb?: ReturnType, + seedOptions: { agentId?: string } = {}, ) { const db = existingDb ?? createDb(temporary.connectionString); + const runAgentId = seedOptions.agentId ?? agentId; const issueId = randomUUID(); const runId = randomUUID(); await db.insert(issues).values({ @@ -169,14 +171,14 @@ export function nativeRestartFixture(options: { dedicatedDatabaseUrl?: string } status: "in_progress", priority: "medium", workMode: "standard", - assigneeAgentId: agentId, + assigneeAgentId: runAgentId, }); const [run] = await db .insert(heartbeatRuns) .values({ id: runId, companyId, - agentId, + agentId: runAgentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, diff --git a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts index 8529d65d0e1..804847e4429 100644 --- a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts +++ b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts @@ -3747,4 +3747,20 @@ describe("agent issue mutation checkout ownership", () => { expect(mockIssueService.update).not.toHaveBeenCalled(); }); }); + + it.each(["owner-agent", "board-without-runtime-access"] as const)("denies original-host containment for %s before settlement", async actorKind => { + mockIssueService.getById.mockResolvedValue(makeIssue({ status: "in_progress", assigneeAgentId: ownerAgentId })); + const actor = actorKind === "owner-agent" ? ownerActor() + : { type: "board", userId: "board-user", source: "session", companyIds: [companyId] }; + const response = await request(await createApp(actor)).post(`/api/issues/${issueId}/recovery-actions/resolve`).send({ + actionId: recoveryActionId, outcome: "restored", sourceIssueStatus: "in_progress", + executionReconciliation: { runId: ownerRunId, controllerStopped: true, providerStopped: true, + controllerAuthorityDigest: "a".repeat(64), actionOutcome: "mixed", + outcomeEvidence: "Original-host containment and external action outcomes were reviewed." }, + }); + expect(response.status, JSON.stringify(response.body)).toBe(403); + if (actorKind === "owner-agent") expect(response.body.error).toBe("Board access required"); + expect(mockIssueRecoveryActionService.resolveActiveForIssue).not.toHaveBeenCalled(); + expect(mockIssueService.update).not.toHaveBeenCalled(); + }); }); diff --git a/server/src/__tests__/ownership-reconciliation-report.test.ts b/server/src/__tests__/ownership-reconciliation-report.test.ts index 82489deaa92..58bf048e852 100644 --- a/server/src/__tests__/ownership-reconciliation-report.test.ts +++ b/server/src/__tests__/ownership-reconciliation-report.test.ts @@ -207,7 +207,7 @@ describe("report-only ownership reconciliation", () => { }); }); - it("proves a foreign-boot coordinator gone when its PID-space stamp matches and its PID is dead", async () => { + it("proves a recorded same-space Native controller gone while keeping a legacy NULL controller unprovable", async () => { const issueId = randomUUID(); const runId = randomUUID(); const legacyRunId = randomUUID(); const legacyIssueId = randomUUID(); await db.insert(issues).values([{ id: issueId, companyId, title: "Dead owner" }, { id: legacyIssueId, companyId, title: "Legacy" }]); await db.insert(heartbeatRuns).values([ @@ -220,13 +220,11 @@ describe("report-only ownership reconciliation", () => { controllerProcessStartedAt: new Date("2026-11-07T09:00:00.000Z"), controllerGeneration: 1 }; await db.insert(nativeRunFinalizations).values([ { ...claim, runId, issueId, controllerHostHash: currentProcessSpaceHash() }, - // Written before the PID space was stamped: the one-release local proof - // on a single-host server (#19883), never on an ephemeral one. { ...claim, runId: legacyRunId, issueId: legacyIssueId, controllerHostHash: null }, ]); const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false }); expect(kindReport(report, "native_executor_claim")).toMatchObject({ inProgress: 2, - verdicts: { gone: 2, unprovable: 0 }, reasons: { pid_dead: 2 } }); + verdicts: { gone: 1, unprovable: 1 }, reasons: { pid_dead: 1, process_space_unrecorded: 1 } }); vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); try { const ephemeral = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false }); diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 8ffb36b5765..3c118e4a9f2 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -10297,6 +10297,12 @@ export function issueRoutes( const decision = await access.decide({ actor: req.actor, action: "runtime:manage", resource: { type: "company", companyId: existing.companyId } }); if (!decision.allowed) throw forbidden(decision.explanation, authorizationDeniedDetails(decision)); } + if (executionReconciliation?.controllerStopped !== undefined || executionReconciliation?.controllerAuthorityDigest !== undefined) { + assertBoard(req); + const permission = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", + resource: { type: "company", companyId: existing.companyId } }); + if (!permission.allowed) throw forbidden(permission.explanation, authorizationDeniedDetails(permission)); + } const actor = getActorInfo(req); const actionStatus = outcome === "cancelled" ? "cancelled" : "resolved"; const postCommitActivityPublications: ActivityPublication[] = []; @@ -10419,6 +10425,11 @@ export function issueRoutes( { source: "recovery_action_resolution" }, ); + if (activeRecoveryAction.kind === "active_run_watchdog" && activeRecoveryAction.ownerType === "board" + && activeRecoveryAction.cause === "native_restart_recovery_blocked" + && activeRecoveryAction.evidence.reason === "controller_process_space_unverified" + && (!executionReconciliation?.controllerStopped || !executionReconciliation.controllerAuthorityDigest)) + throw conflict("Confirm the original controller and provider stopped and review their action outcomes before resolving this hold."); const workspaceRetry = outcome === "restored" && sourceIssueStatus === "todo" ? await prepareRestrictedWorkspaceRetry(tx as unknown as Db, lockedIssue, activeRecoveryAction) : null; @@ -10548,7 +10559,21 @@ export function issueRoutes( } } - if ( + const controllerContainment = executionReconciliation?.controllerStopped === true + || executionReconciliation?.controllerAuthorityDigest !== undefined; + if (controllerContainment) { + assertBoard(req); + const permission = await access.decide({ actor: req.actor, action: "runtime:confirm_owner_stop", + resource: { type: "company", companyId: lockedIssue.companyId } }); + if (!permission.allowed) throw forbidden(permission.explanation, authorizationDeniedDetails(permission)); + if (outcome !== "restored" || sourceIssueStatus !== lockedIssue.status || !actionId + || activeRecoveryAction.id !== actionId || workspaceBaseRef || workspaceRetry) + throw conflict("Containment only records the original execution. Keep the task unchanged, then use Retry separately."); + await validateExecutionReconciliation({ db: tx as unknown as Db, companyId: lockedIssue.companyId, + issueId: lockedIssue.id, agentId: lockedIssue.assigneeAgentId, + sourceRunId: activeRecoveryAction.evidence.runId, decision: executionReconciliation, + controllerContainment: { actionId, actorId: actor.actorId } }); + } else if ( sourceIssueStatus === "todo" && requiresExecutionReconciliation(activeRecoveryAction.cause) ) { @@ -10570,7 +10595,7 @@ export function issueRoutes( } let chatRetry: { actionId: string; issueId: string } | null = null; - if (outcome === "restored" && sourceIssueStatus === "todo") { + if (!controllerContainment && outcome === "restored" && sourceIssueStatus === "todo") { const [chatBinding] = await tx .select({ id: chatConversations.id }) .from(chatConversations) @@ -10625,6 +10650,7 @@ export function issueRoutes( chatRetry ? { kind: "chat_failed_run_retry", actionId: chatRetry.actionId } : undefined, + controllerContainment ? { controllerContainmentOnly: true } : undefined, ); } let issue = lockedIssue; @@ -10780,7 +10806,7 @@ export function issueRoutes( issue = updatedIssue; } - const recordedOutcome = + const recordedOutcome = controllerContainment ? "restored" : outcome === "restored" && issue.status === "todo" && activeRecoveryAction.returnOwnerAgentId != null && diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 04eef87a7d0..573425394ab 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,3 +1,4 @@ +import { hasCommittedNativeControllerContainment } from "../native-runtime/native-failed-run-owner-release.js"; import { notifyStrandedAgentStatus } from "../live-run-recovery-notifications.js"; import { notifyDeferredExecutionWork, notifyTaskRetrySettlement } from "../deferred-execution-notifications.js"; import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; @@ -382,6 +383,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ) .for("share", { noWait: true }); + if (coordinator && await hasCommittedNativeControllerContainment(tx, run, issueId)) { + return { coordinator, leases: [], retiredOwner: null, operatorContained: true }; + } // A queued request can fail before its first provider event because the // predecessor's exact retained checkpoint has not yet been suspended. // A later authenticated cleanup receipt can authorize retrying that @@ -690,8 +694,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti if (run.runtimeMode !== "native") throw failedChatRetryDenied(); } if (run.runtimeMode !== "native") return; - const { coordinator, leases, retiredOwner } = - await failedNativeRetryCoordinator(tx, run, source.issueId); + const state = await failedNativeRetryCoordinator(tx, run, source.issueId); + if ("operatorContained" in state && state.operatorContained) return; + const { coordinator, leases, retiredOwner } = state; if (retiredOwner) { if ( !nativePreProviderRetryAfterCleanupStateIsSafe({ diff --git a/server/src/services/chat-channels/task-retry-authority.ts b/server/src/services/chat-channels/task-retry-authority.ts index c2018c6e5ff..65ef377bc7b 100644 --- a/server/src/services/chat-channels/task-retry-authority.ts +++ b/server/src/services/chat-channels/task-retry-authority.ts @@ -1,3 +1,4 @@ +import { hasCommittedNativeControllerContainment } from "../native-runtime/native-failed-run-owner-release.js"; import { and, eq, sql } from "drizzle-orm"; import { agentWakeupRequests, environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, issues, type Db } from "@paperclipai/db"; import { buildExecutionContinuation } from "../execution-continuation.js"; @@ -74,12 +75,11 @@ async function taskChatRetryExecutionContained(db: Db, run: typeof heartbeatRuns const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), )).for("share", { noWait: true }); - // A verified stop counts, including one whose destroy is queued as stopped - // cleanup (a recovered or retained orphan sandbox awaiting teardown). if (leases.some(lease => lease.provider !== "local")) { return leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease)); } - return await recordedLocalExecutionStopped(db, run) || + return await hasCommittedNativeControllerContainment(db, run, run.nativeIssueId ?? "") || + await recordedLocalExecutionStopped(db, run) || await hasNativeLocalProcessStop(db, run.companyId, run.id) || await hasHistoricalSuspendedNativeSession(db, run); } diff --git a/server/src/services/controller-process-identity.ts b/server/src/services/controller-process-identity.ts index 9986100214b..338c7485b60 100644 --- a/server/src/services/controller-process-identity.ts +++ b/server/src/services/controller-process-identity.ts @@ -1,6 +1,7 @@ import { createHash, randomUUID } from "node:crypto"; +import { execFileSync } from "node:child_process"; import { readFileSync, readlinkSync } from "node:fs"; -import { hostname, uptime } from "node:os"; +import { hostname } from "node:os"; import { readProcessStartedAt } from "./hot-restart.js"; import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js"; @@ -63,10 +64,17 @@ function processSpaceHash(): string { const parts = [hostname()]; if (process.platform === "linux") { parts.push(readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim(), readlinkSync("/proc/self/ns/pid")); + } else if (process.platform === "darwin") { + // One bounded, read-only kernel identity observation per module. Rounded + // boot-time estimates can collide across hosts and cannot authorize PID + // comparisons. Old weak hashes simply stop matching this stronger proof. + const bootSession = execFileSync("/usr/sbin/sysctl", ["-n", "kern.bootsessionuuid"], { + encoding: "utf8", timeout: 1_500, maxBuffer: 4_096, windowsHide: true, + }).trim().toLowerCase(); + if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(bootSession)) return UNKNOWN_PROCESS_SPACE; + parts.push(bootSession); } else { - // No PID namespaces here. The boot time keeps a rebooted host distinct; - // rounding can only make a record unmatched, which fails closed. - parts.push(String(Math.round((Date.now() - uptime() * 1000) / 600_000))); + return UNKNOWN_PROCESS_SPACE; } if (parts.some(part => !part)) return UNKNOWN_PROCESS_SPACE; return createHash("sha256").update(parts.join("\0")).digest("hex").slice(0, 32); diff --git a/server/src/services/controller-process-space-identity.test.ts b/server/src/services/controller-process-space-identity.test.ts new file mode 100644 index 00000000000..bbf3b9046f4 --- /dev/null +++ b/server/src/services/controller-process-space-identity.test.ts @@ -0,0 +1,71 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const state = vi.hoisted(() => ({ read: vi.fn(), link: vi.fn(), sysctl: vi.fn(), selfBirth: vi.fn() })); +vi.mock("node:fs", async original => ({ ...await original(), + readFileSync: state.read, readlinkSync: state.link })); +vi.mock("node:os", async original => ({ ...await original(), hostname: () => "same-host" })); +vi.mock("node:child_process", async original => ({ ...await original(), execFileSync: state.sysctl })); +vi.mock("./hot-restart.js", () => ({ readProcessStartedAt: state.selfBirth })); + +const descriptor = Object.getOwnPropertyDescriptor(process, "platform")!; +const firstBoot = "00000000-0000-4000-8000-000000000001"; +const secondBoot = "00000000-0000-4000-8000-000000000002"; +function platform(value: string) { Object.defineProperty(process, "platform", { ...descriptor, value }); } +async function fresh() { vi.resetModules(); return import("./controller-process-identity.js"); } + +beforeEach(() => { + vi.resetAllMocks(); + state.read.mockReturnValue(firstBoot + "\n"); state.link.mockReturnValue("pid:[42]"); + state.sysctl.mockReturnValue(firstBoot.toUpperCase() + "\n"); + state.selfBirth.mockResolvedValue("2026-11-07T00:00:00.000Z"); +}); +afterEach(() => { Object.defineProperty(process, "platform", descriptor); vi.restoreAllMocks(); }); + +describe("controller process-space identity", () => { + it("keeps Linux boot and PID-namespace isolation without launching a command", async () => { + platform("linux"); + const original = await (await fresh()).currentControllerProcess(); + expect(original.hostHash).toMatch(/^[0-9a-f]{32}$/); + expect(state.read).toHaveBeenCalledWith("/proc/sys/kernel/random/boot_id", "utf8"); + expect(state.link).toHaveBeenCalledWith("/proc/self/ns/pid"); + expect(state.sysctl).not.toHaveBeenCalled(); + state.link.mockReturnValue("pid:[43]"); + expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(original.hostHash); + state.link.mockReturnValue("pid:[42]"); state.read.mockReturnValue(secondBoot); + expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(original.hostHash); + }); + + it("uses Darwin kernel boot UUID once per module, never a rounded clock identity", async () => { + platform("darwin"); + const owner = await fresh(); + const first = await owner.currentControllerProcess(); + expect(first.hostHash).toMatch(/^[0-9a-f]{32}$/); + expect(state.sysctl).toHaveBeenCalledExactlyOnceWith("/usr/sbin/sysctl", ["-n", "kern.bootsessionuuid"], + { encoding: "utf8", timeout: 1_500, maxBuffer: 4_096, windowsHide: true }); + expect(state.read).not.toHaveBeenCalled(); expect(state.link).not.toHaveBeenCalled(); + state.sysctl.mockReturnValue(secondBoot); + expect((await owner.currentControllerProcess()).hostHash).toBe(first.hostHash); + expect(state.sysctl).toHaveBeenCalledTimes(1); + expect((await (await fresh()).currentControllerProcess()).hostHash).not.toBe(first.hostHash); + expect(state.sysctl).toHaveBeenCalledTimes(2); + expect(JSON.stringify(first)).not.toContain(firstBoot); + // Old weak/foreign receipts retain their original deadline: no local probe. + const kill = vi.spyOn(process, "kill"); + expect(await owner.controllerProcessExited({ bootId: "old", pid: 2_000_000_001, + hostHash: "old-rounded-clock-hash", processStartedAt: null })).toBe(false); + expect(kill).not.toHaveBeenCalled(); + }); + + it.each(["unreadable", "malformed", "unsupported"] as const)("fails closed without an owner PID probe for %s identity", async mode => { + platform(mode === "unsupported" ? "win32" : "darwin"); + if (mode === "unreadable") state.sysctl.mockImplementation(() => { throw new Error("sysctl unavailable"); }); + if (mode === "malformed") state.sysctl.mockReturnValue("not-a-kernel-uuid"); + const owner = await fresh(); + expect((await owner.currentControllerProcess()).hostHash).toBe("unknown"); + const kill = vi.spyOn(process, "kill"); + expect(await owner.controllerProcessExited({ bootId: "old", pid: 2_000_000_001, + hostHash: "unknown", processStartedAt: null })).toBe(false); + expect(kill).not.toHaveBeenCalled(); + if (mode === "unsupported") expect(state.sysctl).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/execution-projection.test.ts b/server/src/services/execution-projection.test.ts index 847a4c0fcd7..e5c6eb289ee 100644 --- a/server/src/services/execution-projection.test.ts +++ b/server/src/services/execution-projection.test.ts @@ -39,6 +39,24 @@ const project = ( ) => projectExecution(r, c, pending, undefined, now); describe("execution truth projection", () => { + it.each(["contained", "ordinary", "wrong-source", "continued"] as const)("projects controller containment without inventing queued work: %s", state => { + const stopped = run({ id: "old", status: "failed" }); + const action = { status: "resolved" as const, cause: "native_restart_recovery_blocked", nextAction: "Inspect the original owner", + evidence: { reason: "controller_process_space_unverified", continuationDelivery: "not_requested", + executionReconciliation: { runId: state === "wrong-source" ? "another-run" : "old", providerStopped: true, + ...(state === "ordinary" ? {} : { controllerStopped: true, controllerAuthorityDigest: "a".repeat(64) }) }, + ...(state === "continued" ? { explicitUserContinuation: { previousRunId: "old", runId: "new" } } : {}), + } }; + const projected = projectExecution(stopped, coordinator({ phase: "terminal_failure" }), [], action, now); + expect(projected.phase).toBe(state === "contained" ? "failed" : state === "continued" ? "completed" : "queued"); + if (state === "contained") { + expect(projected.label).toBe("Execution contained"); + expect(projected.nextAction).toContain("Use Retry"); + expect(projected.successorRunId).toBeNull(); + } + if (state === "continued") expect(projected.successorRunId).toBe("new"); + }); + it("offers Stop for every live run, including a long-silent native turn", () => { const silentSince = new Date(now.getTime() - 35 * 60_000); const silent = run({ lastUsefulActionAt: silentSince, startedAt: silentSince }); diff --git a/server/src/services/execution-projection.ts b/server/src/services/execution-projection.ts index b0a21ac3508..ab3de935360 100644 --- a/server/src/services/execution-projection.ts +++ b/server/src/services/execution-projection.ts @@ -334,6 +334,27 @@ export function projectExecution( "The task was closed or reassigned before delivery. Inspect the current task owner and status."; return set("failed", "Continuation cancelled"); } + const reconciliation = recoveryAction.evidence.executionReconciliation; + if (recoveryAction.cause === "native_restart_recovery_blocked" + && recoveryAction.evidence.reason === "controller_process_space_unverified" + && recoveryAction.evidence.continuationDelivery === "not_requested" + && reconciliation && typeof reconciliation === "object" && !Array.isArray(reconciliation) + && "runId" in reconciliation && reconciliation.runId === run.id + && "controllerStopped" in reconciliation && reconciliation.controllerStopped === true + && "providerStopped" in reconciliation && reconciliation.providerStopped === true + && "controllerAuthorityDigest" in reconciliation && typeof reconciliation.controllerAuthorityDigest === "string" && /^[0-9a-f]{64}$/.test(reconciliation.controllerAuthorityDigest)) { + const continuation = recoveryAction.evidence.explicitUserContinuation as + { previousRunId?: unknown; runId?: unknown } | undefined; + const explicitSuccessor = text(continuation?.runId); + if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) { + projection.successorRunId = explicitSuccessor; + projection.nextAction = null; + return set("completed", "Continued in another run"); + } + projection.cause = null; + projection.nextAction = "The original execution was contained and its action outcomes reviewed. Use Retry to request another attempt."; + return set("failed", "Execution contained"); + } const deliveredRunId = text(recoveryAction.evidence.continuationRunId); projection.successorRunId = deliveredRunId; projection.cause = null; diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index 7c1ed06d034..a5e23dab54b 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -1,3 +1,4 @@ +import { settleOperatorContainedNativeOwner } from "./native-runtime/native-failed-run-owner-release.js"; import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -14,6 +15,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { logger } from "../middleware/logger.js"; import { and, eq, exists, inArray, isNull, not, or, sql } from "drizzle-orm"; import { + agentWakeupRequests, chatActions, environmentLeases, heartbeatRuns, @@ -53,7 +55,7 @@ export function executionRecoverySettlementNote(input: { : "Automatic recovery stopped. Recorded work is preserved; actions with unverified outcomes will not be repeated."; } -export async function validateExecutionReconciliation(input: { +type ExecutionReconciliationInput = { db: Db; companyId: string; issueId: string; @@ -62,13 +64,37 @@ export async function validateExecutionReconciliation(input: { decision: ExecutionReconciliation | undefined; /** Board-only source repair: never authorize a continuation of this run. */ workspaceRepairOnly?: boolean; -}) { + controllerContainment?: { actionId: string; actorId: string }; +}; + +export async function validateExecutionReconciliation(input: ExecutionReconciliationInput) { + if (input.controllerContainment) { + // Validation and contained-owner settlement must roll back together even + // when a caller is not already inside the board route's transaction. + return input.db.transaction(tx => validateExecutionReconciliationInScope({ ...input, db: tx as unknown as Db })); + } + return validateExecutionReconciliationInScope(input); +} + +async function validateExecutionReconciliationInScope(input: ExecutionReconciliationInput) { const { db, companyId, issueId, agentId, decision } = input; if (!decision || decision.runId !== input.sourceRunId || (!agentId && !input.workspaceRepairOnly)) { throw conflict( "Reconcile the recorded execution and its action outcomes before continuing this task.", ); } + let operatorContained = false; + if (decision.controllerStopped !== undefined || decision.controllerAuthorityDigest !== undefined) { + if (!decision.controllerStopped || !decision.controllerAuthorityDigest || !input.controllerContainment + || input.workspaceRepairOnly) throw conflict("Original-host controller containment requires its exact board recovery action."); + operatorContained = await settleOperatorContainedNativeOwner(db, { + companyId, issueId, runId: decision.runId, actionId: input.controllerContainment.actionId, + actorId: input.controllerContainment.actorId, authorityDigest: decision.controllerAuthorityDigest, + actionOutcome: decision.actionOutcome, outcomeEvidence: decision.outcomeEvidence, + }); + if (!operatorContained) throw conflict("The recorded controller authority changed, is still leased, or its cleanup is unsettled. Refresh before recording containment.", + { code: "native_controller_containment_stale" }); + } const [run] = await db .select() .from(heartbeatRuns) @@ -110,7 +136,7 @@ export async function validateExecutionReconciliation(input: { (!decision.workspaceRepairEvidence || decision.workspaceRepairEvidence.trim().length < 20)) { throw conflict("Verify safe workspace staging or repair and record workspaceRepairEvidence before continuing this run."); } - let checkLocalProcesses = true; + let checkLocalProcesses = !operatorContained; if (retainedWorkspace) { const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run.id), @@ -128,7 +154,7 @@ export async function validateExecutionReconciliation(input: { : !hasRemoteTerminationReceipt(lease))) { throw conflict("The retained workspace environment has not confirmed that it stopped. Wait for cleanup before recording workspace repair."); } - checkLocalProcesses = leases.some(lease => lease.provider === "local"); + checkLocalProcesses = !operatorContained && leases.some(lease => lease.provider === "local"); } // A retained remote source's PIDs belong to its sandbox, not this server. // Its exact lease receipts above replace host-local process probes. @@ -199,8 +225,10 @@ export async function markExecutionReconciliation( decision: ExecutionReconciliation, actorId: string, deliveryOwner?: { kind: "chat_failed_run_retry"; actionId: string }, - options?: { workspaceRepairOnly?: boolean }, + options?: { workspaceRepairOnly?: boolean; controllerContainmentOnly?: boolean }, ) { + if (options?.controllerContainmentOnly && (!decision.controllerStopped || !decision.controllerAuthorityDigest || deliveryOwner)) + throw conflict("Controller containment records no automatic continuation."); if (options?.workspaceRepairOnly && (deliveryOwner || !hasRequiredWorkspaceRecovery(action.evidence))) { throw conflict("Workspace repair must target the retained source without a continuation owner."); } @@ -252,7 +280,7 @@ export async function markExecutionReconciliation( actorId, recordedAt: new Date().toISOString(), }, - continuationDelivery: options?.workspaceRepairOnly ? "not_requested" : deliveryOwner ? "delegated" : "pending", + continuationDelivery: options?.workspaceRepairOnly || options?.controllerContainmentOnly ? "not_requested" : deliveryOwner ? "delegated" : "pending", ...(deliveryOwner ? { continuationDeliveryOwner: deliveryOwner } : {}), }, }) @@ -267,6 +295,20 @@ export async function markExecutionReconciliation( kind: issueRecoveryActions.kind, evidence: issueRecoveryActions.evidence, }); + if (options?.controllerContainmentOnly) { + // A Retry clicked before the operator reviewed the effects must not turn + // into an automatic replay after containment. Retain only that exact, + // unadmitted source receipt as skipped; newer/accepted intents are untouched. + await tx.update(agentWakeupRequests).set({ status: "skipped", finishedAt: new Date(), updatedAt: new Date(), + error: "controller_containment_requires_new_retry" }).where(and( + eq(agentWakeupRequests.companyId, action.companyId), isNull(agentWakeupRequests.runId), + inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]), + sql`${agentWakeupRequests.payload}->>'issueId' = ${action.sourceIssueId}`, + sql`${agentWakeupRequests.payload}->'taskRetry'->>'schema' = 'paperclip.task-retry.v1'`, + sql`${agentWakeupRequests.payload}->'taskRetry'->>'sourceRunId' = ${decision.runId}`, + )); + return; + } if ( !source || options?.workspaceRepairOnly || source.cause !== "legacy_execution_requires_reconciliation" || diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index df53a07525e..f98c639e8dd 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -6,7 +6,7 @@ import { writeExecutionControl } from "./execution-control-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { and, desc, eq, inArray, isNull, isNotNull, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { heartbeatRuns, issueRecoveryActions, workspaceOperations } from "@paperclipai/db"; +import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db"; import { isUuidLike } from "@paperclipai/shared"; import type { IssueRecoveryAction, @@ -623,6 +623,15 @@ export function issueRecoveryActionService(db: DbOrTransaction) { const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) .where(and(...predicates)).limit(1); if (!pending.length) return []; + // Resolving another attention releases this issue's preserved Native + // waiter to one normal inspection. No recurring monitor or new turn. + const [waiter] = await tx.select({ id: nativeRunFinalizations.runId }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.sourceIssueId), + eq(nativeRunFinalizations.phase, "observed"), isNull(nativeRunFinalizations.resultId), + isNotNull(nativeRunFinalizations.leaseOwner), inArray(heartbeatRuns.status, ["failed", "timed_out", "interrupted"]), + )).limit(1); + if (waiter) await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionFinalization); await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); diff --git a/server/src/services/native-runtime/native-accepted-result-lease-settlement.ts b/server/src/services/native-runtime/native-accepted-result-lease-settlement.ts index fad566bd33b..659fa94780f 100644 --- a/server/src/services/native-runtime/native-accepted-result-lease-settlement.ts +++ b/server/src/services/native-runtime/native-accepted-result-lease-settlement.ts @@ -1,4 +1,4 @@ -import { and, eq, gt, inArray, isNull, notLike, sql } from "drizzle-orm"; +import { and, eq, gt, inArray, isNotNull, isNull, notInArray, notLike, sql } from "drizzle-orm"; import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; @@ -6,8 +6,37 @@ import { notifyNativeResultCommit } from "../native-result-commit-notifications. import type { OwnerProof } from "../owner-proof.js"; import type { InProgressRecord, SettleOutcome } from "../recovery/ownership-registry.js"; import { requestNativeControllerStopConfirmation } from "./native-controller-host-recovery.js"; +import { nativeControllerOwnerStamp } from "./native-owner-stamps.js"; +import { readNativeRemoteExecutionOwnershipForRuns } from "./native-runner-ownership.js"; import { recordLegacyHostFallbackProof } from "./native-restart-recovery.js"; +export async function classifyNativeAcceptedResultLeaseOwners(db: Db, records: readonly InProgressRecord[]): Promise { + if (!records.length) return []; + const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }) + .from(nativeRunFinalizations).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId))) + .where(and(inArray(nativeRunFinalizations.runId, records.map(record => record.recordId)), + isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, ["committed", "terminal_failure"]), + isNotNull(nativeRunFinalizations.leaseOwner), notLike(nativeRunFinalizations.leaseOwner, "native-finalizer:%"), + notLike(nativeRunFinalizations.leaseOwner, "native-cleanup:%"), sql`${nativeRunFinalizations.leaseExpiresAt} > now()`)); + const ownership = await readNativeRemoteExecutionOwnershipForRuns(db, + rows.map(row => ({ run: row.run, issueId: row.coordinator.issueId }))); + const byRun = new Map(rows.map((row, index) => [row.run.id, { ...row, remote: ownership[index]!.remoteProcess }])); + return records.flatMap(record => { + const row = byRun.get(record.recordId); + if (!row || row.run.companyId !== record.companyId || row.coordinator.issueId !== record.issueId) + return []; + const current = { recordId: row.run.id, companyId: row.run.companyId, runId: row.run.id, + issueId: row.coordinator.issueId, updatedAt: row.coordinator.updatedAt }; + if (!row.remote) return [{ ...current, + owner: { kind: "process", stamp: nativeControllerOwnerStamp(row.coordinator, "native_accepted_result_lease") } }]; + return [{ ...current, owner: { kind: "fence", stamp: { kind: "remote", scope: "native_accepted_result_lease", + generation: row.coordinator.controllerGeneration, fenceToken: row.coordinator.leaseOwner }, + fence: { currentFenceToken: row.coordinator.leaseOwner, currentGeneration: row.coordinator.controllerGeneration, + leaseExpiresAt: row.coordinator.leaseExpiresAt } } }]; + }); +} + /** * Settle of the `native_accepted_result_lease` ownership kind (plan Phase 2). * @@ -36,6 +65,7 @@ export async function settleNativeAcceptedResultLease(db: Db, record: InProgress issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, leaseOwner: nativeRunFinalizations.leaseOwner, controllerBootId: nativeRunFinalizations.controllerBootId, controllerPid: nativeRunFinalizations.controllerPid, controllerHostHash: nativeRunFinalizations.controllerHostHash, + controllerGeneration: nativeRunFinalizations.controllerGeneration, controllerProcessStartedAt: nativeRunFinalizations.controllerProcessStartedAt, remote: sql`(${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync' is not null or ${heartbeatRuns.runnerProfileJson}->'remoteWorkspace' is not null)`, @@ -55,7 +85,8 @@ export async function settleNativeAcceptedResultLease(db: Db, record: InProgress // Remote copyback is fenced by its own controller lease, never by a local proof. if (row.remote) return { settled: false, reason: "remote_lease_fenced" }; // The proof judged this exact controller; a different one holds it now. - if (row.controllerBootId !== stamp.bootId || row.controllerPid !== stamp.pid + if (row.leaseOwner !== stamp.fenceToken || row.controllerGeneration !== stamp.generation + || row.controllerBootId !== stamp.bootId || row.controllerPid !== stamp.pid || (row.controllerHostHash ?? null) !== (stamp.processSpace ?? null) || (row.controllerProcessStartedAt?.toISOString() ?? null) !== (stamp.processStartedAt ?? null)) { return { settled: false, reason: "owner_changed_since_proof" }; diff --git a/server/src/services/native-runtime/native-controller-host-recovery.test.ts b/server/src/services/native-runtime/native-controller-host-recovery.test.ts index dec4b4ffdcf..ecab8c87b98 100644 --- a/server/src/services/native-runtime/native-controller-host-recovery.test.ts +++ b/server/src/services/native-runtime/native-controller-host-recovery.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { and, eq } from "drizzle-orm"; import { - activityLog, agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, + activityLog, agents, companies, createDb, environmentLeases, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -29,6 +29,8 @@ describe("native controller rows saved before their host was recorded", () => { await db.insert(issues).values({ id: issueId, companyId, title: "Native task", status: "in_progress", assigneeAgentId: agentId }); await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, status: "running", runtimeMode: "native", contextSnapshot: { issueId } }); + // Native admission binds its running source before exposing recovery. + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); // The shape the previous release saved: a local controller and no host. const [coordinator] = await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1, leaseOwner: "old-controller", leaseExpiresAt: new Date(0), controllerBootId: randomUUID(), @@ -77,6 +79,66 @@ describe("native controller rows saved before their host was recorded", () => { await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: actions[0]!.id, runId, stopEvidence, actorId: "board-user" })).rejects.toMatchObject({ status: 409 }); }); + it("keeps an observed no-checkpoint source recoverable after a controller-only confirmation", async () => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + const { companyId, agentId, issueId, runId, coordinator } = await seed(); + // Historical initial state: the previous release published its real + // controller-only action for an observed run with no checkpoint. There + // is no accepted provider result and no fabricated terminal outcome. + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + await db.update(nativeRunFinalizations).set({ recoveryState: "awaiting_evidence" }) + .where(eq(nativeRunFinalizations.runId, runId)); + await requestNativeControllerStopConfirmation(db, coordinator); + const [originalAction] = await db.select().from(issueRecoveryActions) + .where(eq(issueRecoveryActions.sourceIssueId, issueId)); + await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: originalAction!.id, runId, + stopEvidence: "The original server container is stopped; provider effects are not yet verified.", + actorId: "board-user" })).resolves.toEqual({ runId, status: "queued" }); + const [original] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, originalAction!.id)); + expect(original).toMatchObject({ status: "resolved", evidence: { controllerStop: { actorId: "board-user" } } }); + const [held] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + expect(held).toMatchObject({ phase: "observed", resultId: null, leaseOwner: coordinator.leaseOwner, + controllerGeneration: coordinator.controllerGeneration, attempt: coordinator.attempt }); + const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(source).toMatchObject({ status: "running", resultJson: null }); + const active = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.status, "active"))); + expect(active).toEqual([expect.objectContaining({ cause: "native_restart_recovery_blocked", ownerType: "board", + returnOwnerAgentId: agentId, attemptCount: 0, wakePolicy: null, monitorPolicy: null, + evidence: expect.objectContaining({ runId, reason: "controller_process_space_unverified", + controllerAuthority: expect.objectContaining({ schema: "paperclip.native-controller-authority.v1", + digest: expect.stringMatching(/^[a-f0-9]{64}$/) }) }) })]); + expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, runId))) + .toEqual([]); + }); + + it.each(["newer-owner", "newer-source"] as const)( + "keeps a stale controller-only confirmation from changing %s", async kind => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + const { companyId, agentId, issueId, runId, coordinator } = await seed(); + await requestNativeControllerStopConfirmation(db, coordinator); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + if (kind === "newer-owner") { + await db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), + controllerGeneration: coordinator.controllerGeneration + 1, leaseOwner: "new-controller" }) + .where(eq(nativeRunFinalizations.runId, runId)); + } else { + const nextRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issueId, + runtimeMode: "native", status: "queued", contextSnapshot: { issueId } }); + await db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, issueId)); + } + const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + await expect(confirmNativeControllerStopped({ db, companyId, issueId, actionId: action!.id, runId, + stopEvidence: "The previous server stopped, but this dialog predates the current owner.", actorId: "board-user" })) + .rejects.toMatchObject({ status: 409 }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]).toEqual(before); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action!.id)))[0]) + .toMatchObject({ status: "active" }); + }, + ); + + it("binds stop confirmation to an exact foreign-host controller and rejects changed ownership", async () => { const { companyId, issueId, runId, coordinator } = await seed(); @@ -101,4 +163,28 @@ describe("native controller rows saved before their host was recorded", () => { .toEqual({ allowed: true, reason: "operator_attested_controller_stopped" }); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.status).toBe("running"); }); + it.each(["same-task", "different-task"] as const)("binds controller follow-up to the %s runner lease", async scope => { + const { companyId, agentId, issueId, runId, coordinator } = await seed(); + let leaseIssueId = issueId; + if (scope === "different-task") { + leaseIssueId = randomUUID(); + await db.insert(issues).values({ id: leaseIssueId, companyId, title: "Another task", status: "in_progress", assigneeAgentId: agentId }); + } + await db.insert(environmentLeases).values({ companyId, issueId: leaseIssueId, heartbeatRunId: runId, + provider: "plugin", status: "active", metadata: { runnerTaskLease: true } }); + await requestNativeControllerStopConfirmation(db, coordinator); + const [original] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + await expect(confirmNativeControllerStopped({ db, companyId, issueId, runId, actionId: original.id, + actorId: "board-user", stopEvidence: "The original controller is stopped; provider containment remains unverified." })) + .resolves.toEqual({ runId, status: "queued" }); + const [attention] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.status, "active"))); + expect(attention.evidence.reason).toBe(scope === "same-task" + ? "remote_controller_process_space_unverified" : "controller_process_space_unverified"); + if (scope === "same-task") expect(attention.evidence.controllerAuthority).toBeUndefined(); + else expect(attention.evidence.controllerAuthority).toMatchObject({ schema: "paperclip.native-controller-authority.v1" }); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]) + .toMatchObject({ phase: "observed", resultId: null, leaseOwner: coordinator.leaseOwner }); + }); + }); diff --git a/server/src/services/native-runtime/native-controller-host-recovery.ts b/server/src/services/native-runtime/native-controller-host-recovery.ts index ae424e1741e..eb0a1602ef3 100644 --- a/server/src/services/native-runtime/native-controller-host-recovery.ts +++ b/server/src/services/native-runtime/native-controller-host-recovery.ts @@ -1,9 +1,12 @@ -import { and, eq, isNull } from "drizzle-orm"; -import { heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { readNativeRemoteExecutionOwnership } from "./native-runner-ownership.js"; +import { and, eq, isNull, ne, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { confirmNativeControllerStoppedSchema } from "@paperclipai/shared"; import { conflict } from "../../errors.js"; import { persistActivity, publishActivity } from "../activity-log.js"; import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; +import { surfaceNativeControllerNamespaceAttention } from "./native-restart-recovery.js"; +import { readPendingNativeRunStop } from "./native-cancellation-request.js"; export const NATIVE_CONTROLLER_HOST_UNVERIFIED = "native_controller_host_unverified"; /** controller_host_hash value written once the board confirms the stop. */ @@ -27,7 +30,7 @@ export async function requestNativeControllerStopConfirmation(tx: Db, coordinato processStartedAt: coordinator.controllerProcessStartedAt?.toISOString() ?? null } }, nextAction: "Paperclip cannot verify that this run's previous server stopped from the current host. " + "Confirm that the previous server process has stopped (for example, its " - + "container was replaced), and the run resumes from its saved state.", + + "container was replaced), then recovery checks the saved run. Controller confirmation does not confirm that its provider or tools stopped.", }).onConflictDoNothing(); } @@ -41,35 +44,78 @@ export async function confirmNativeControllerStopped(input: { }); const committed = await input.db.transaction(async transaction => { const tx = transaction as unknown as Db; + // Match Native admission and containment lock order. This records only + // the previous controller stop, not provider containment or a new turn. + const [task] = await tx.select().from(issues).where(and( + eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); + const [run] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))).for("update"); const [action] = await tx.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, input.companyId), eq(issueRecoveryActions.sourceIssueId, input.issueId), eq(issueRecoveryActions.id, input.actionId))).for("update"); - const [run] = await tx.select({ agentId: heartbeatRuns.agentId }).from(heartbeatRuns).where(and( - eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); const recorded = (action?.evidence?.controller ?? {}) as Record; - if (!coordinator || !action || !run || action.cause !== NATIVE_CONTROLLER_HOST_UNVERIFIED || action.ownerType !== "board" + if (!task || !coordinator || !action || !run || action.cause !== NATIVE_CONTROLLER_HOST_UNVERIFIED || action.ownerType !== "board" || !["active", "escalated"].includes(action.status) || action.evidence?.runId !== input.runId || coordinator.controllerHostHash !== (recorded.hostHash ?? null) || coordinator.controllerPid !== recorded.pid || coordinator.controllerBootId !== recorded.bootId - || (coordinator.controllerProcessStartedAt?.toISOString() ?? null) !== recorded.processStartedAt) throw changed(); + || (coordinator.controllerProcessStartedAt?.toISOString() ?? null) !== recorded.processStartedAt + || run.runtimeMode !== "native" + || !(coordinator.resultId === null ? run.status === "running" : ["running", "cancelled"].includes(run.status)) + || (coordinator.resultId === null ? !["observed", "retryable_failure"].includes(coordinator.phase) + : ["committed", "terminal_failure"].includes(coordinator.phase)) + || task.assigneeAgentId !== run.agentId || task.executionRunId !== run.id + || (task.checkoutRunId !== null && task.checkoutRunId !== run.id) + || (run.nativeIssueId ?? run.contextSnapshot?.issueId) !== task.id + || ["done", "cancelled"].includes(task.status) + || !coordinator.leaseOwner || !coordinator.leaseExpiresAt + || coordinator.leaseOwner.startsWith("native-finalizer:") || coordinator.leaseOwner.startsWith("native-cleanup:") + || await readPendingNativeRunStop(tx, run)) throw changed(); + const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), ne(heartbeatRuns.id, run.id), + or(eq(heartbeatRuns.nativeIssueId, task.id), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${task.id}`, + sql`exists (select 1 from ${nativeRunFinalizations} newer_owner + where newer_owner.run_id = ${heartbeatRuns.id} and newer_owner.company_id = ${run.companyId} + and newer_owner.issue_id = ${task.id})`), + sql`${heartbeatRuns.createdAt} >= (select source.created_at from ${heartbeatRuns} source + where source.id = ${run.id} and source.company_id = ${run.companyId})`, + )).limit(1); + if (newer) throw changed(); + // Use the DB's full precision, not a Date or session-dependent rendering. + const [expiry] = await tx.select({ epoch: sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` }) + .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); + if (!expiry?.epoch) throw changed(); // Work intent before the write it announces (sleep-aware background work). await notifyLiveRunRecovery(tx, { companyId: input.companyId, runId: input.runId, agentId: run.agentId }); const now = new Date(); - const updated = await tx.update(nativeRunFinalizations).set({ controllerHostHash: OPERATOR_ATTESTED_CONTROLLER_HOST, updatedAt: now }) + const [updated] = await tx.update(nativeRunFinalizations).set({ controllerHostHash: OPERATOR_ATTESTED_CONTROLLER_HOST, updatedAt: now }) .where(and(eq(nativeRunFinalizations.runId, input.runId), coordinator.controllerHostHash === null ? isNull(nativeRunFinalizations.controllerHostHash) : eq(nativeRunFinalizations.controllerHostHash, coordinator.controllerHostHash), eq(nativeRunFinalizations.controllerPid, coordinator.controllerPid!), - eq(nativeRunFinalizations.controllerBootId, coordinator.controllerBootId!))) - .returning({ runId: nativeRunFinalizations.runId }); - if (!updated.length) throw changed(); + eq(nativeRunFinalizations.controllerBootId, coordinator.controllerBootId!), + eq(nativeRunFinalizations.leaseOwner, coordinator.leaseOwner), + eq(nativeRunFinalizations.controllerGeneration, coordinator.controllerGeneration), + eq(nativeRunFinalizations.attempt, coordinator.attempt), + eq(nativeRunFinalizations.phase, coordinator.phase), + coordinator.resultId === null ? isNull(nativeRunFinalizations.resultId) + : eq(nativeRunFinalizations.resultId, coordinator.resultId), + sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text = ${expiry.epoch}`)) + .returning(); + if (!updated) throw changed(); const proof = { controller: recorded, actorId: input.actorId, stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() }; await tx.update(issueRecoveryActions).set({ status: "resolved", outcome: "restored", resolvedAt: now, updatedAt: now, evidence: { ...action.evidence, controllerStop: proof }, - nextAction: "The previous server was confirmed stopped. Recovery resumes the run from its saved state.", + nextAction: "The previous server was confirmed stopped. Recovery will inspect the saved run. Separate provider containment may still be required.", }).where(eq(issueRecoveryActions.id, action.id)); + // The controller-only action is now resolved. Publish its responsible + // next owner in this same transaction: an observed/no-checkpoint run is + // not necessarily admitted by any deferred recovery scanner. + if (coordinator.resultId === null) { + const { remoteProcess } = await readNativeRemoteExecutionOwnership(tx, { run, issueId: task.id }); + await surfaceNativeControllerNamespaceAttention(tx, { run, coordinator: updated, expiryEpoch: expiry.epoch, remoteProcess }); + } const publication = await persistActivity(tx, { companyId: input.companyId, actorType: "user", actorId: input.actorId, action: "issue.native_controller_stop_confirmed", entityType: "issue", entityId: input.issueId, runId: input.runId, details: { recoveryActionId: input.actionId, ...proof } }); diff --git a/server/src/services/native-runtime/native-failed-run-owner-release.test.ts b/server/src/services/native-runtime/native-failed-run-owner-release.test.ts index 2bf19985fdc..146aa9585b8 100644 --- a/server/src/services/native-runtime/native-failed-run-owner-release.test.ts +++ b/server/src/services/native-runtime/native-failed-run-owner-release.test.ts @@ -1,7 +1,7 @@ import { spawn } from "node:child_process"; import { randomUUID } from "node:crypto"; import { once } from "node:events"; -import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { and, eq } from "drizzle-orm"; import { activityLog, @@ -12,10 +12,12 @@ import { environmentLeases, heartbeatRuns, issues, + issueRecoveryActions, nativeRunFinalizations, subscribeDatabaseWork, } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { agentExecutionsHaveStopped } from "../agent-execution-stop.js"; import { readProcessStartedAt } from "../hot-restart.js"; import { currentProcessSpaceHash } from "../controller-process-identity.js"; @@ -31,6 +33,8 @@ import { settleUnstartedNativeSessionResume, } from "./native-failed-run-owner-release.js"; +import * as controllerProcess from "../controller-process-identity.js"; + const LEASE_FAILURE = 'Failed to acquire lease for environment "Managed computer" (sandbox): Sandbox creation has failed'; describe("failed native run owner release", () => { @@ -63,9 +67,10 @@ describe("failed native run owner release", () => { errorCode?: string; error?: string; coordinator: Partial; + runId?: string; }) { const issueId = randomUUID(); - const runId = randomUUID(); + const runId = input.runId ?? randomUUID(); await db.insert(issues).values({ id: issueId, companyId, title: "Resume the native run", status: "in_progress", assigneeAgentId: agentId, workMode: "standard" }); await db.insert(heartbeatRuns).values({ @@ -80,8 +85,10 @@ describe("failed native run owner release", () => { providerSessionId: "provider-session", activeTurnId: "active-turn" }, }, }); + const current = await controllerProcess.currentControllerProcess(); + expect(current.hostHash).not.toBe("unknown"); await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1, - ...input.coordinator } as typeof nativeRunFinalizations.$inferInsert); + controllerHostHash: current.hostHash, ...input.coordinator } as typeof nativeRunFinalizations.$inferInsert); return { runId, issueId }; } @@ -166,6 +173,194 @@ describe("failed native run owner release", () => { expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); }); + it.each(["null", "foreign", "unknown-current"] as const)("keeps a failed namespace-held source out of automatic expired-owner release: %s", async mode => { + const actual = await controllerProcess.currentControllerProcess(); + expect(actual.hostHash).not.toBe("unknown"); + const recordedPid = 2_000_000_001; + const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed", + coordinator: { leaseOwner: `foreign-controller:${randomUUID()}`, leaseExpiresAt: new Date(0), + controllerBootId: randomUUID(), controllerPid: recordedPid, + controllerHostHash: mode === "null" ? null : mode === "foreign" ? `foreign:${actual.hostHash}` : actual.hostHash, + recoveryState: "awaiting_evidence" } }); + await db.update(heartbeatRuns).set({ processPid: recordedPid, processGroupId: recordedPid }) + .where(eq(heartbeatRuns.id, runId)); + const before = await coordinator(runId); + const originalKill = process.kill.bind(process); + const probes: number[] = []; + const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => { + if (Math.abs(pid) === recordedPid && signal === 0) { + probes.push(pid); + throw Object.assign(new Error("controlled local PID absent; not foreign stop proof"), { code: "ESRCH" }); + } + return originalKill(pid, signal); + }); + const identity = mode === "unknown-current" ? vi.spyOn(controllerProcess, "currentProcessSpaceHash") + .mockReturnValue("unknown") : null; + try { + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + expect(probes).toEqual([]); + expect(await coordinator(runId)).toEqual(before); + const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId)); + expect(attention).toMatchObject({ cause: "native_restart_recovery_blocked", attemptCount: 0, wakePolicy: null, monitorPolicy: null, + evidence: { runId, reason: "controller_process_space_unverified" } }); + for (let inspection = 0; inspection < 3; inspection++) { + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await executionControlSchedule(db).finalization()).toBeNull(); + } + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, before.issueId))).toEqual([attention]); + expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId), + eq(activityLog.action, "native.failed_run_owner_released")))).toHaveLength(0); + } finally { identity?.mockRestore(); probe.mockRestore(); } + }); + + it.each(["timed_out", "interrupted"] as const)("publishes quiet operator attention after an already %s owner expires", async status => { + const { runId, issueId } = await nativeRun({ status, coordinator: { leaseOwner: `foreign:${randomUUID()}`, + controllerBootId: randomUUID(), controllerHostHash: null, leaseExpiresAt: new Date(0) } }); + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + const [action] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + expect(action).toMatchObject({ ownerType: "board", cause: "native_restart_recovery_blocked", attemptCount: 0, + evidence: { runId, reason: "controller_process_space_unverified", controllerAuthority: { digest: expect.any(String) } } }); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await coordinator(runId)).toMatchObject({ phase: "observed", leaseOwner: expect.any(String) }); + }); + + it("keeps empty deadline and release scans from stamping controller ownership", async () => { + const stamped = vi.fn(); + const unsubscribe = controllerProcess.onControllerStamp(stamped); + try { + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(stamped).not.toHaveBeenCalled(); + } finally { unsubscribe(); } + }); + + it("retains a foreign owner's original deadline until its returned unstarted claim settles", async () => { + const leaseExpiresAt = new Date(Date.now() + 60_000); + const leaseOwner = `default:resume:${randomUUID()}`; + // This dispatched claim returned before the executor took an attempt. Its + // exact returned token/attempt, not the PID or lease age, authorizes release. + const { runId, issueId } = await nativeRun({ status: "failed", errorCode: "setup_failed", + coordinator: { leaseOwner, leaseExpiresAt, controllerBootId: randomUUID(), + controllerHostHash: `foreign:${currentProcessSpaceHash()}` } }); + const before = await coordinator(runId); + expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime()); + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + expect(await coordinator(runId)).toEqual(before); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]); + expect(await settleUnstartedNativeSessionResume(db, { runId, leaseOwner, attempt: before.attempt })).toBe(true); + expect(await coordinator(runId)).toMatchObject({ phase: "terminal_failure", leaseOwner: null }); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]); + }); + + it("surfaces a foreign owner only at its original expiry without stamping or probing it", async () => { + const leaseExpiresAt = new Date(Date.now() + 60_000); + const recordedPid = 2_000_000_001; + const { runId, issueId } = await nativeRun({ status: "failed", coordinator: { + leaseOwner: `foreign:${randomUUID()}`, leaseExpiresAt, controllerBootId: randomUUID(), + controllerPid: recordedPid, controllerHostHash: null } }); + const before = await coordinator(runId); + const stamped = vi.fn(); + const unsubscribe = controllerProcess.onControllerStamp(stamped); + const kill = process.kill.bind(process); + const probes: number[] = []; + const probe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => { + if (Math.abs(pid) === recordedPid && signal === 0) { + probes.push(pid); + throw Object.assign(new Error("foreign PID is not local stop evidence"), { code: "ESRCH" }); + } + return kill(pid, signal); + }); + try { + expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(leaseExpiresAt.getTime()); + expect(await releaseAbandonedNativeOwners(db, new Date(leaseExpiresAt.getTime() - 1))).toBe(0); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([]); + expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0); + const [attention] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + expect(attention).toMatchObject({ status: "active", ownerType: "board", attemptCount: 0, + wakePolicy: null, monitorPolicy: null, evidence: { runId, reason: "controller_process_space_unverified" } }); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await releaseAbandonedNativeOwners(db, leaseExpiresAt)).toBe(0); + expect(await coordinator(runId)).toEqual(before); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toEqual([attention]); + expect(probes).toEqual([]); + expect(stamped).not.toHaveBeenCalled(); + } finally { probe.mockRestore(); unsubscribe(); } + }); + + it("finishes each requested namespace inspection pass and preserves the original local deadline", async () => { + const foreign = []; + for (let index = 0; index < 5; index++) foreign.push(await nativeRun({ status: "failed", coordinator: { + leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } })); + const localDeadline = new Date(Date.now() + 600_000); + await nativeRun({ status: "failed", coordinator: { leaseOwner: `local:${randomUUID()}`, leaseExpiresAt: localDeadline } }); + for (let page = 0; page < 2; page++) { + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime()); + } + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime()); + const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId)); + expect(rows).toHaveLength(foreign.length); + expect(rows.every(row => row.attemptCount === 0 && row.wakePolicy === null && row.monitorPolicy === null)).toBe(true); + // Deadline reads do not request another pass or turn surfaced holds into timers. + for (let read = 0; read < 2; read++) { + expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime()); + } + // A later explicit mutation inspection must recheck all retained authorities. + // Finish its known three-page pass before expecting the local timer again. + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime()); + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect((await nextAbandonedNativeOwnerReleaseAt(db))!.getTime()).toBeLessThan(localDeadline.getTime()); + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect((await nextAbandonedNativeOwnerReleaseAt(db))?.getTime()).toBe(localDeadline.getTime()); + const reinspected = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, companyId)); + const byId = (a: typeof rows[number], b: typeof rows[number]) => a.id.localeCompare(b.id); + expect(reinspected.sort(byId)).toEqual([...rows].sort(byId)); + }); + + it("keeps a lower-sorting arrival durable when its notification coalesces between inspection pages", async () => { + for (let index = 1; index <= 4; index++) await nativeRun({ status: "failed", + runId: `ffffffff-ffff-4fff-8fff-${String(index).padStart(12, "0")}`, coordinator: { + leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } }); + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + const lower = await nativeRun({ status: "failed", runId: "00000000-0000-4000-8000-000000000001", coordinator: { + leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } }); + // Simulate one coalesced dirty signal by continuing the actual existing + // page, not resetting its cursor or creating a second artificial worker. + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).not.toBeNull(); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toEqual([]); + expect(await releaseAbandonedNativeOwners(db, new Date(), 2)).toBe(0); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, lower.issueId))).toHaveLength(1); + expect(await releaseAbandonedNativeOwners(db, new Date(), 5)).toBe(0); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + }); + + it("surfaces a preserved namespace waiter when an unrelated action is actually resolved", async () => { + const { runId, issueId } = await nativeRun({ status: "failed", coordinator: { + leaseOwner: `foreign:${randomUUID()}`, controllerHostHash: null, leaseExpiresAt: new Date(0) } }); + const service = issueRecoveryActionService(db); + const unrelated = await service.upsertSourceScoped({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog", + cause: "independent-test-recovery", fingerprint: "independent-test-identity", ownerType: "board", + nextAction: "Resolve this independently owned action first.", wakePolicy: null, monitorPolicy: null, attemptCount: 0 }); + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0].id).toBe(unrelated.id); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + expect(await committedWakes(DELIVERY_QUEUES.executionFinalization, () => service.resolveActiveForIssue({ companyId, + sourceIssueId: issueId, actionId: unrelated.id, status: "resolved", outcome: "restored" }))).toBeGreaterThan(0); + expect(await releaseAbandonedNativeOwners(db)).toBe(0); + const rows = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)); + expect(rows).toHaveLength(2); + expect(rows.find(row => row.id === unrelated.id)).toMatchObject({ status: "resolved", fingerprint: "independent-test-identity" }); + expect(rows.find(row => row.id !== unrelated.id)).toMatchObject({ attemptCount: 0, + evidence: { runId, reason: "controller_process_space_unverified" } }); + expect(await nextAbandonedNativeOwnerReleaseAt(db)).toBeNull(); + }); + it("keeps an unexpired owner blocking the stop check and schedules its expiry", async () => { const leaseExpiresAt = new Date(Date.now() + 10 * 60_000); const { runId } = await nativeRun({ status: "failed", errorCode: "setup_failed", error: LEASE_FAILURE, diff --git a/server/src/services/native-runtime/native-failed-run-owner-release.ts b/server/src/services/native-runtime/native-failed-run-owner-release.ts index ef7ddff6456..172d40b84a2 100644 --- a/server/src/services/native-runtime/native-failed-run-owner-release.ts +++ b/server/src/services/native-runtime/native-failed-run-owner-release.ts @@ -1,15 +1,21 @@ -import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; -import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { currentProcessSpaceHash } from "../controller-process-identity.js"; +import type { ExecutionReconciliation } from "@paperclipai/shared"; +import { nativeControllerAuthorityDigest, readNativeRemoteExecutionOwnership, isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js"; +import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; +import { settleNativeTerminalWake } from "./native-run-projection.js"; +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; +import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm"; +import { activityLog, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db"; import { logger } from "../../middleware/logger.js"; import { adapterExecutionControls } from "../adapter-execution-control.js"; import { proveOwnerGone, type LocalOwnerStamp } from "../owner-proof.js"; -import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; +import { logActivity, persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { LEGACY_WORKSPACE_RECOVERY_KEY } from "../workspace-restore-recovery-state.js"; import { activeNativeExecutionRunIds } from "./native-active-runs.js"; -import { isCurrentNativeControllerProcess, recordLegacyHostFallbackProof } from "./native-restart-recovery.js"; +import { currentNativeControllerIdentity, isCurrentNativeControllerProcess, surfaceNativeControllerNamespaceAttention, recordLegacyHostFallbackProof } from "./native-restart-recovery.js"; import { nativeControllerOwnerStamp, runProcessOwnerStamp } from "./native-owner-stamps.js"; /** @@ -37,7 +43,10 @@ type Release = /** The caller dispatched this claim and its execution has returned. */ | { kind: "unstarted_resume"; runId: string; leaseOwner: string; attempt: number } /** A background pass found an expired owner and must prove it stopped. */ - | { kind: "expired_owner"; runId: string; leaseOwner: string; attempt: number; controllerGeneration: number }; + | { kind: "expired_owner"; runId: string; leaseOwner: string; attempt: number; controllerGeneration: number } + | { kind: "operator_containment"; runId: string; leaseOwner: string; attempt: number; + companyId: string; issueId: string; actionId: string; authorityDigest: string; actorId: string; controllerGeneration: number; + actionOutcome: ExecutionReconciliation["actionOutcome"]; outcomeEvidence: string }; /** Environment leases that `agentExecutionsHaveStopped` would still treat as live. */ const unsettledRunEnvironmentLease = sql`exists ( @@ -70,19 +79,44 @@ function abandonedOwnerCondition() { )!; } -/** - * The earliest lease expiry the background pass will need to inspect. Sandbox - * cleanup is excluded, so an unreleased lease is never polled for here. An - * expired row whose process or controller may still be live keeps the shared - * finalization retry cadence until that process exits. - */ +/** Local owners retain their release deadline. Foreign owners retain the same + * original deadline until expiry surfaces attention owned by the board. */ +async function localOwnerSpaceCondition() { + const processSpace = currentProcessSpaceHash(); + const native = await currentNativeControllerIdentity(); + return or( + and(eq(nativeRunFinalizations.controllerBootId, native.bootId), eq(nativeRunFinalizations.controllerPid, native.pid)), + processSpace !== "unknown" ? eq(nativeRunFinalizations.controllerHostHash, processSpace) : sql`false`, + )!; +} + export async function nextAbandonedNativeOwnerReleaseAt(db: Db): Promise { + // A finite remaining inspection page is work; a surfaced foreign owner is not + // a timer. The existing cursor is cleared after the last lookahead page. + if (abandonedOwnerReleaseCursor !== null) return new Date(); + const local = await localOwnerSpaceCondition(); + // Notifications can coalesce while a partial page advances. A lower-sorting + // arrival without any active attention remains durable inspection work, so + // completing the higher pages cannot consume its only wake. + const [unattended] = await db.select({ at: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId))) + .where(and(abandonedOwnerCondition(), sql`not coalesce(${local}, false)`, + sql`${nativeRunFinalizations.leaseOwner} not like 'native-finalizer:%'`, + sql`${nativeRunFinalizations.leaseOwner} not like 'native-cleanup:%'`, + sql`not exists (select 1 from ${issueRecoveryActions} attention + where attention.company_id = ${nativeRunFinalizations.companyId} + and attention.source_issue_id = ${nativeRunFinalizations.issueId} + and attention.status in ('active', 'escalated'))`, + )).orderBy(asc(nativeRunFinalizations.leaseExpiresAt)).limit(1); const [row] = await db.select({ at: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations) .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId))) - .where(abandonedOwnerCondition()) + .where(and(abandonedOwnerCondition(), sql`coalesce(${local}, false)`)) .orderBy(asc(nativeRunFinalizations.leaseExpiresAt)).limit(1); - return row?.at ?? null; + if (!unattended?.at) return row?.at ?? null; + if (!row?.at) return unattended.at; + return unattended.at < row.at ? unattended.at : row.at; } /** A proof that used the one-release fallback for a row saved before its PID @@ -132,20 +166,94 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, release.runId)); if (!binding) return false; // Same issue-before-coordinator-before-run order as admission and failure settlement. - await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, binding.companyId), + const [task] = await tx.select({ id: issues.id, assigneeAgentId: issues.assigneeAgentId, + status: issues.status, executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues).where(and(eq(issues.companyId, binding.companyId), eq(issues.id, binding.issueId))).for("update"); const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and( eq(nativeRunFinalizations.companyId, binding.companyId), eq(nativeRunFinalizations.runId, release.runId), eq(nativeRunFinalizations.issueId, binding.issueId))).for("update"); const [run] = await tx.select().from(heartbeatRuns).where(and( eq(heartbeatRuns.companyId, binding.companyId), eq(heartbeatRuns.id, release.runId))).for("update"); - if (!coordinator || !run || run.runtimeMode !== "native" || !RELEASABLE_RUN_STATUSES.includes(run.status) - || coordinator.phase !== "observed" || coordinator.resultId !== null + if (!coordinator || !run || run.runtimeMode !== "native" + || !(RELEASABLE_RUN_STATUSES.includes(run.status) || (release.kind === "operator_containment" && run.status === "running")) + || !(coordinator.phase === "observed" || (release.kind === "operator_containment" && coordinator.phase === "retryable_failure")) + || coordinator.resultId !== null || coordinator.leaseOwner !== release.leaseOwner || coordinator.attempt !== release.attempt) return false; - if (release.kind === "expired_owner") { - if (!coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > now - || coordinator.controllerGeneration !== release.controllerGeneration - || run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null) return false; + let containedExpiryEpoch: string | null = null; + if (release.kind === "operator_containment") { + const [action] = await tx.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, release.companyId), eq(issueRecoveryActions.sourceIssueId, release.issueId), + eq(issueRecoveryActions.id, release.actionId))).for("update"); + const [expiry] = await tx.select({ epoch: sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` }) + .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); + const authority = action?.evidence.controllerAuthority; + const recordedDigest = authority && typeof authority === "object" && !Array.isArray(authority) + && (authority as Record).schema === "paperclip.native-controller-authority.v1" + ? (authority as Record).digest : null; + if (!["completed", "not_performed", "mixed"].includes(release.actionOutcome) + || release.outcomeEvidence.trim().length < 20 || release.outcomeEvidence.length > 12_000 + || !task || !action || action.kind !== "active_run_watchdog" || action.ownerType !== "board" + || !["active", "escalated"].includes(action.status) || action.cause !== "native_restart_recovery_blocked" + || action.fingerprint !== `native-controller-space:${run.id}:${release.authorityDigest}` || action.evidence.runId !== run.id + || action.evidence.reason !== "controller_process_space_unverified" + || recordedDigest !== release.authorityDigest || !expiry + || nativeControllerAuthorityDigest(coordinator, expiry.epoch) !== release.authorityDigest + || coordinator.companyId !== release.companyId || coordinator.issueId !== release.issueId + || action.returnOwnerAgentId !== run.agentId || task.assigneeAgentId !== run.agentId + || ["done", "cancelled"].includes(task.status) + || (task.executionRunId !== run.id && !(task.executionRunId === null && RELEASABLE_RUN_STATUSES.includes(run.status))) + || (task.checkoutRunId !== null && task.checkoutRunId !== run.id) + || !coordinator.leaseExpiresAt || coordinator.leaseExpiresAt > now + || coordinator.leaseOwner?.startsWith("native-finalizer:") || coordinator.leaseOwner?.startsWith("native-cleanup:") + || coordinator.failureDetail?.successorRunId || isCurrentNativeControllerProcess(coordinator) + || isNativeRunnerOwnershipHeld(run) || (run.controllerLeaseExpiresAt !== null && run.controllerLeaseExpiresAt > now) + || run.runnerProfileJson?.nativeWorkspaceSync != null || run.runnerProfileJson?.remoteWorkspace != null + || run.runnerProfileJson?.nativeControllerRecovery != null + || run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null + || adapterExecutionControls.has(run.id) || activeNativeExecutionRunIds.has(run.id)) return false; + if (task.executionRunId === null) { + // Issue lock serializes admission. A cleared failure pointer does not + // authorize containment of an older/replaced source. + const [newer] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), + or(eq(heartbeatRuns.nativeIssueId, coordinator.issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${coordinator.issueId}`, + sql`exists (select 1 from ${nativeRunFinalizations} newer_owner + where newer_owner.run_id = ${heartbeatRuns.id} and newer_owner.company_id = ${run.companyId} + and newer_owner.issue_id = ${coordinator.issueId})`), + ne(heartbeatRuns.id, run.id), + sql`${heartbeatRuns.createdAt} >= (select source.created_at from ${heartbeatRuns} source + where source.id = ${run.id} and source.company_id = ${run.companyId})`, + )).limit(1); + if (newer) return false; + } + const [heldElsewhere] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, run.id), or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1); + const [work] = await tx.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, run.companyId), eq(workspaceOperations.heartbeatRunId, run.id), + inArray(workspaceOperations.status, ["queued", "running"]))).limit(1); + if (heldElsewhere || work) return false; + containedExpiryEpoch = expiry.epoch; + // The explicit original-host controller+provider attestation replaces a + // meaningless local PID observation only in this exact authority branch. + } else if (release.kind === "expired_owner") { + if (!coordinator.leaseExpiresAt || coordinator.controllerGeneration !== release.controllerGeneration + || run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null + || coordinator.leaseExpiresAt > now) return false; + // An expired lease does not make foreign PID numbers local evidence. + // Keep this boot's exact owner semantics; every other owner needs the + // same known kernel/PID space before either recorded-process probe. + if (!isCurrentNativeControllerProcess(coordinator)) { + const current = currentProcessSpaceHash(); + if (current === "unknown" || coordinator.controllerHostHash !== current) { + if (!coordinator.leaseOwner?.startsWith("native-finalizer:") && !coordinator.leaseOwner?.startsWith("native-cleanup:")) { + const [expiry] = await tx.select({ epoch: sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text` }) + .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); + const { remoteProcess } = await readNativeRemoteExecutionOwnership(tx, { run, issueId: coordinator.issueId }); + await surfaceNativeControllerNamespaceAttention(tx, { run, coordinator, expiryEpoch: expiry?.epoch ?? null, remoteProcess }); + } + return false; + } + } const [heldElsewhere] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns) .where(and(eq(heartbeatRuns.id, run.id), or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1); if (heldElsewhere || await runProcessMayBeLive(run, now, fallbacks) @@ -159,7 +267,8 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis eq(nativeRunResults.schemaStatus, "accepted"))).limit(1); if (accepted) return false; - const failureCode = coordinator.failureCode ?? run.errorCode ?? FALLBACK_FAILURE_CODE; + const failureCode = release.kind === "operator_containment" ? "native_restart_recovery_blocked" + : coordinator.failureCode ?? run.errorCode ?? FALLBACK_FAILURE_CODE; const scope = { companyId: run.companyId, agentId: run.agentId, runId: run.id, issueId: coordinator.issueId }; await notifyExecutionRunWork(tx, scope); await notifyNativeFinalizationWork(tx, scope); @@ -177,21 +286,49 @@ async function settleFailedRunOwner(db: Db, release: Release, now: Date): Promis ...coordinator.failureDetail, ...(run.error ? { message: run.error.slice(0, 2_000) } : {}), originalFailureCode: run.errorCode ?? failureCode, - releasedOwner: { reason: release.kind, leaseOwner: release.leaseOwner, attempt: release.attempt }, + releasedOwner: release.kind === "operator_containment" + ? { schema: "paperclip.native-controller-containment.v1", reason: release.kind, actionId: release.actionId, + controllerGeneration: coordinator.controllerGeneration, controllerBootId: coordinator.controllerBootId, + controllerHostHash: coordinator.controllerHostHash, authorityDigest: release.authorityDigest, attempt: release.attempt, + actorId: release.actorId, recordedAt: now.toISOString(), controllerStopped: true, providerStopped: true, + actionOutcome: release.actionOutcome, outcomeEvidence: release.outcomeEvidence, + priorProcess: { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt?.toISOString() ?? null } } + : { reason: release.kind, leaseOwner: release.leaseOwner, attempt: release.attempt }, recoveryOwner: { kind: "board" }, nextAction: NEXT_ACTION, }, updatedAt: now, }).where(and( eq(nativeRunFinalizations.runId, run.id), - eq(nativeRunFinalizations.phase, "observed"), + eq(nativeRunFinalizations.phase, release.kind === "operator_containment" ? coordinator.phase : "observed"), isNull(nativeRunFinalizations.resultId), eq(nativeRunFinalizations.leaseOwner, release.leaseOwner), eq(nativeRunFinalizations.attempt, release.attempt), + release.kind === "operator_containment" ? and( + eq(nativeRunFinalizations.controllerGeneration, release.controllerGeneration), + sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text = ${containedExpiryEpoch}`, + coordinator.controllerHostHash === null ? isNull(nativeRunFinalizations.controllerHostHash) + : eq(nativeRunFinalizations.controllerHostHash, coordinator.controllerHostHash), + ) : undefined, )).returning({ runId: nativeRunFinalizations.runId }); if (!updated) return false; - await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, updatedAt: now }) - .where(eq(heartbeatRuns.id, run.id)); + const [settledRun] = await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, updatedAt: now, + ...(release.kind === "operator_containment" ? { status: "failed", finishedAt: now, + errorCode: failureCode, error: "The original controller and provider were contained by a board operator. Review the recorded action outcomes before explicitly retrying.", + processPid: null, processGroupId: null, processStartedAt: null, executionControlDeadlineAt: null } : {}), + }).where(eq(heartbeatRuns.id, run.id)).returning(); + if (release.kind === "operator_containment" && settledRun) { + await preserveNativeFailedWorkspace(tx, settledRun); + await settleNativeTerminalWake(tx, settledRun); + await notifyNativeStatusReassessment(tx, scope); + // Durable activity only: this call may be a savepoint of board resolution, + // so publishing a live event here would precede the outer commit. + await persistActivity(tx, { companyId: run.companyId, actorType: "user", actorId: release.actorId, + action: "native.controller_containment_recorded", entityType: "heartbeat_run", entityId: run.id, + agentId: run.agentId, runId: run.id, details: { recoveryActionId: release.actionId, + authorityDigest: release.authorityDigest, replay: "not_authorized" } }); + return true; + } for (const use of fallbacks) { await recordLegacyHostFallbackProof(tx, run, { allowed: true, reason: use.reason, legacyHostFallback: true }, use.source); } @@ -253,10 +390,10 @@ export async function releaseAbandonedNativeOwners(db: Db, now = new Date(), pag .where(and(abandonedOwnerCondition(), lte(nativeRunFinalizations.leaseExpiresAt, now), after ? gt(nativeRunFinalizations.runId, after) : undefined)) .orderBy(asc(nativeRunFinalizations.runId)) - .limit(pageSize); - let candidates = await page(abandonedOwnerReleaseCursor); - if (candidates.length === 0 && abandonedOwnerReleaseCursor !== null) candidates = await page(null); - abandonedOwnerReleaseCursor = candidates.length === pageSize ? candidates[candidates.length - 1]!.runId : null; + .limit(pageSize + 1); + const rows = await page(abandonedOwnerReleaseCursor); + const candidates = rows.slice(0, pageSize); + abandonedOwnerReleaseCursor = rows.length > pageSize ? candidates[candidates.length - 1]!.runId : null; let released = 0; for (const candidate of candidates) { try { @@ -270,3 +407,105 @@ export async function releaseAbandonedNativeOwners(db: Db, now = new Date(), pag } return released; } + +/** This entry is called only by the board-authorized reconciliation owner. + * Its opaque authority is reloaded and checked under the normal settlement + * locks; the operator receipt never becomes a fabricated provider result. */ +export async function settleOperatorContainedNativeOwner(db: Db, input: { + companyId: string; issueId: string; runId: string; actionId: string; + authorityDigest: string; actorId: string; + actionOutcome: ExecutionReconciliation["actionOutcome"]; outcomeEvidence: string; +}): Promise { + const [owner] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), + eq(nativeRunFinalizations.runId, input.runId))); + if (!owner?.leaseOwner) return false; + return settleFailedRunOwner(db, { kind: "operator_containment", ...input, + leaseOwner: owner.leaseOwner, attempt: owner.attempt, controllerGeneration: owner.controllerGeneration }, new Date()); +} + +/** Committed operator evidence is a separate authority from an observed local + * stop. It permits only a new, independently authorized Retry of this source; + * it never proves a PID was observed dead or admits provider replay. */ +export async function hasCommittedNativeControllerContainment( + db: Pick, + run: Run, + issueId: string, +): Promise { + if (run.runtimeMode !== "native" || run.nativeIssueId !== issueId || run.status !== "failed" + || run.nativePhase !== "terminal_failure" || !run.finishedAt + || run.errorCode !== "native_restart_recovery_blocked" + || run.processPid !== null || run.processGroupId !== null || run.processStartedAt !== null + || adapterExecutionControls.has(run.id) || activeNativeExecutionRunIds.has(run.id) + || (run.controllerLeaseExpiresAt !== null && run.controllerLeaseExpiresAt > new Date()) + || run.runnerProfileJson?.nativeWorkspaceSync != null || run.runnerProfileJson?.remoteWorkspace != null + || run.runnerProfileJson?.nativeControllerRecovery != null + || run.resultJson?.[LEGACY_WORKSPACE_RECOVERY_KEY] != null) return false; + const [task] = await db.select().from(issues).where(and(eq(issues.companyId, run.companyId), + eq(issues.id, issueId))).for("share", { noWait: true }); + if (!task || task.assigneeAgentId !== run.agentId || ["done", "cancelled"].includes(task.status)) return false; + // This is a committed source-containment fact, not admission authority. + // Existing chat/queue owners separately bind current checkout/execution and + // the exact admitted successor; a legitimate successor does not undo the + // source's original-host containment. + const [owner] = await db.select().from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.issueId, issueId), + eq(nativeRunFinalizations.runId, run.id))).for("share", { noWait: true }); + if (!owner || owner.phase !== "terminal_failure" || owner.recoveryState !== "blocked" + || owner.failureCode !== "native_restart_recovery_blocked" || owner.leaseOwner || owner.leaseExpiresAt + || owner.nextAttemptAt || owner.resultId || owner.assessmentId || owner.decisionId + || owner.failureDetail?.successorRunId) return false; + const value = owner.failureDetail?.releasedOwner; + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const receipt = value as Record; + if (receipt.schema !== "paperclip.native-controller-containment.v1" || receipt.reason !== "operator_containment" + || typeof receipt.actionId !== "string" || typeof receipt.authorityDigest !== "string" + || !/^[a-f0-9]{64}$/.test(receipt.authorityDigest) + || receipt.attempt !== owner.attempt || receipt.controllerGeneration !== owner.controllerGeneration + || receipt.controllerBootId !== owner.controllerBootId || receipt.controllerHostHash !== owner.controllerHostHash + || receipt.controllerStopped !== true || receipt.providerStopped !== true + || !["completed", "not_performed", "mixed"].includes(String(receipt.actionOutcome)) + || typeof receipt.outcomeEvidence !== "string" || receipt.outcomeEvidence.trim().length < 20 + || receipt.outcomeEvidence.length > 12_000 || typeof receipt.actorId !== "string" || !receipt.actorId + || typeof receipt.recordedAt !== "string" || !Number.isFinite(Date.parse(receipt.recordedAt))) return false; + const [action] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId), + eq(issueRecoveryActions.id, receipt.actionId))).for("share", { noWait: true }); + const authority = action?.evidence.controllerAuthority; + const decision = action?.evidence.executionReconciliation; + if (!action || action.kind !== "active_run_watchdog" || action.ownerType !== "board" + || action.status !== "resolved" || action.outcome !== "restored" || !action.resolvedAt + || action.returnOwnerAgentId !== run.agentId || action.cause !== "native_restart_recovery_blocked" + || action.fingerprint !== `native-controller-space:${run.id}:${receipt.authorityDigest}` + || action.evidence.runId !== run.id || action.evidence.reason !== "controller_process_space_unverified" + || action.evidence.continuationDelivery !== "not_requested" + || !authority || typeof authority !== "object" || Array.isArray(authority) + || !("schema" in authority) || authority.schema !== "paperclip.native-controller-authority.v1" + || !("digest" in authority) || authority.digest !== receipt.authorityDigest + || !decision || typeof decision !== "object" || Array.isArray(decision)) return false; + const reconciliation = decision as Record; + if (reconciliation.runId !== run.id || reconciliation.controllerStopped !== true || reconciliation.providerStopped !== true + || reconciliation.controllerAuthorityDigest !== receipt.authorityDigest || reconciliation.actorId !== receipt.actorId + || reconciliation.actionOutcome !== receipt.actionOutcome || reconciliation.outcomeEvidence !== receipt.outcomeEvidence) return false; + const [audit] = await db.select({ id: activityLog.id }).from(activityLog).where(and( + eq(activityLog.companyId, run.companyId), eq(activityLog.entityType, "heartbeat_run"), + eq(activityLog.entityId, run.id), eq(activityLog.actorType, "user"), eq(activityLog.actorId, receipt.actorId), + eq(activityLog.action, "native.controller_containment_recorded"), + sql`${activityLog.details}->>'recoveryActionId' = ${action.id}`, + sql`${activityLog.details}->>'authorityDigest' = ${receipt.authorityDigest}`, + sql`${activityLog.details}->>'replay' = 'not_authorized'`)).limit(1); + if (!audit) return false; + const [blocked] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), + or(unsettledRunEnvironmentLease, pendingNativeStop))).limit(1); + const [accepted] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and( + eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id), + eq(nativeRunResults.schemaStatus, "accepted"))).limit(1); + const [work] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, run.companyId), eq(workspaceOperations.heartbeatRunId, run.id), + inArray(workspaceOperations.status, ["queued", "running"]))).limit(1); + const [remote] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), + ne(environmentLeases.provider, "local"))).limit(1); + return !blocked && !accepted && !work && !remote; +} diff --git a/server/src/services/native-runtime/native-owner-stamps.ts b/server/src/services/native-runtime/native-owner-stamps.ts index 8bd2bd9dfdc..50082aba1da 100644 --- a/server/src/services/native-runtime/native-owner-stamps.ts +++ b/server/src/services/native-runtime/native-owner-stamps.ts @@ -11,17 +11,16 @@ type ControllerRow = Pick & { controllerHostHash?: string | null; controllerGeneration?: number | null; + leaseOwner?: string | null; }; -/** NRF records the controller's boot, PID, start time and, since #19868, its - * PID space (`controller_host_hash`). Rows saved before that keep the - * one-release local proof on single-host servers (`processSpaceUnrecorded`, - * recorded as `legacy_host_unknown_dead_pid_proof`, #19883). */ +/** Native controller ownership requires its recorded kernel/PID space before + * a local PID observation. Independent durable stop evidence remains valid. */ export function nativeControllerOwnerStamp(row: ControllerRow, scope: string): LocalOwnerStamp { return { - kind: "local", scope, generation: row.controllerGeneration ?? null, host: null, + kind: "local", scope, generation: row.controllerGeneration ?? null, fenceToken: row.leaseOwner, host: null, processSpace: row.controllerHostHash ?? null, - processSpaceUnrecorded: row.controllerHostHash == null, + processSpaceUnrecorded: row.controllerHostHash == null, requireRecordedProcessSpace: true, bootId: row.controllerBootId, pid: row.controllerPid, processStartedAt: row.controllerProcessStartedAt?.toISOString() ?? null, processGroupId: null, }; diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 5b07bf46788..28a6505dea8 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -34,6 +34,7 @@ import { heartbeatRunEvents, heartbeatRuns, issues, + issueRecoveryActions, nativeRunFinalizations, } from "@paperclipai/db"; import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js"; @@ -49,7 +50,7 @@ import { isUniqueViolation } from "../../db-errors.js"; import { logger } from "../../middleware/logger.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js"; -import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js"; +import { nativeControllerAuthorityDigest, isNativeRunnerOwnershipHeld, readNativeRemoteExecutionOwnership } from "./native-runner-ownership.js"; import { nativeRestartContinuationInFlight, nativeRestartInterruptedTurnId, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { retireSupersededNativeRetry } from "./native-superseded-retry.js"; import { legacyWorkspaceRecoveryNotMigrated } from "../legacy-workspace-recovery-migration.js"; @@ -415,6 +416,7 @@ export async function evaluateNativeControllerTakeover(input: { isProcessAlive?: (pid: number) => boolean; readProcessStartedAt?: (pid: number) => Promise; requireStoppedController?: boolean; + requireRecordedProcessSpace?: boolean; remoteLeaseFenced?: boolean; }): Promise { const { owner, now } = input; @@ -474,12 +476,13 @@ export async function evaluateNativeControllerTakeover(input: { pidExists: input.isProcessAlive ?? base.pidExists, readStartedAt: input.readProcessStartedAt ?? base.readStartedAt, }; - const needsBootProof = owner.controllerHostHash != null - && (owner.controllerHostHash !== probe.processSpace || owner.controllerHostHash === UNKNOWN_PROCESS_SPACE); + const needsBootProof = input.requireRecordedProcessSpace || (owner.controllerHostHash != null + && (owner.controllerHostHash !== probe.processSpace || owner.controllerHostHash === UNKNOWN_PROCESS_SPACE)); const proof = await proveOwnerGone({ kind: "local", scope: "native_controller", generation: null, host: null, processSpace: recordsHost ? owner.controllerHostHash ?? null : probe.processSpace, processSpaceUnrecorded: recordsHost && owner.controllerHostHash == null, + ...(input.requireRecordedProcessSpace ? { requireRecordedProcessSpace: true } : {}), bootId: needsBootProof ? owner.controllerBootId ?? null : null, touchedAt: owner.updatedAt?.toISOString() ?? null, pid: priorPid, processStartedAt: owner.controllerProcessStartedAt?.toISOString() ?? null, processGroupId: null, @@ -927,6 +930,47 @@ function appendBoundedRecoveryHistory(entry: Record) { )`; } +/** Shared by restart inspection and failed-owner cleanup under their existing + * issue/coordinator/run locks. Publishing attention never grants a new turn. */ +export async function surfaceNativeControllerNamespaceAttention(db: Db, input: { + run: Pick; + coordinator: typeof nativeRunFinalizations.$inferSelect; + expiryEpoch: string | null; + remoteProcess: boolean; +}): Promise { + const reason = input.remoteProcess ? "remote_controller_process_space_unverified" : "controller_process_space_unverified"; + const attention = issueRecoveryActionService(db); + const digest = nativeControllerAuthorityDigest(input.coordinator, input.expiryEpoch); + const fingerprint = `native-controller-space:${input.run.id}:${digest}`; + const [existing] = await db.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, input.run.companyId), eq(issueRecoveryActions.sourceIssueId, input.coordinator.issueId), + inArray(issueRecoveryActions.status, ["active", "escalated"]), + )).for("update").limit(1); + // One incident per exact authority. A repeat neither spends a retry + // attempt nor emits a new wake. Renewed/replaced authority gets a new + // incident, preserving the prior confirmation/history. An unrelated + // active recovery owner is never overwritten by this observation. + const sameSource = existing?.cause === "native_restart_recovery_blocked" + && existing.evidence.runId === input.run.id + && ["controller_process_space_unverified", "remote_controller_process_space_unverified"].includes(String(existing.evidence.reason)); + if (!existing || (sameSource && existing.fingerprint !== fingerprint)) { + await attention.upsertSourceScoped({ + companyId: input.run.companyId, sourceIssueId: input.coordinator.issueId, + kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: input.run.agentId, + cause: "native_restart_recovery_blocked", fingerprint, + evidence: { runId: input.run.id, reason, executorStopEvidence: "unverified", + ...(!input.remoteProcess ? { controllerAuthority: { schema: "paperclip.native-controller-authority.v1", + digest, leaseExpiresAt: input.coordinator.leaseExpiresAt!.toISOString() } } : {}) }, + nextAction: input.remoteProcess + ? "Use the task's ordinary Stop action and its environment provider's stop/cleanup recovery. Wait for that exact controller/provider acknowledgement. Local PID checks and local containment confirmation cannot authorize remote recovery." + : "Stop the original controller and provider on their original host. After the saved lease expires, a board operator must record both stops and review action outcomes. Recording containment does not retry the task.", + createOnly: !existing, supersedeOnIdentityChange: true, + attemptCount: 0, maxAttempts: null, timeoutAt: null, lastAttemptAt: null, + wakePolicy: null, monitorPolicy: null, + }); + } +} + export async function claimNativeRestartRecoveries(input: { db: Db; controller?: NativeControllerIdentity; @@ -948,6 +992,7 @@ export async function claimNativeRestartRecoveries(input: { const now = input.now ?? new Date(); const controller = input.controller ?? (await currentNativeControllerIdentity()); + const controllerHostHash = currentProcessSpaceHash(); const candidateQuery = input.db .select({ runId: heartbeatRuns.id, @@ -1022,6 +1067,7 @@ export async function claimNativeRestartRecoveries(input: { .select({ run: heartbeatRuns, coordinator: nativeRunFinalizations, + controllerLeaseExpiryEpoch: sql`extract(epoch from ${nativeRunFinalizations.leaseExpiresAt})::numeric::text`, issueExecutionRunId: issues.executionRunId, issueCheckoutRunId: issues.checkoutRunId, issueAssigneeAgentId: issues.assigneeAgentId, @@ -1075,12 +1121,8 @@ export async function claimNativeRestartRecoveries(input: { const storedRemoteWorkspace = remoteWorkspaceRunReferenceSchema.safeParse(profile.remoteWorkspace); const remoteWorkspace = readNativeWorkspaceSyncReference(profile.nativeWorkspaceSync) ?? (storedRemoteWorkspace.success ? storedRemoteWorkspace.data : null); - const [pluginTask] = await tx.select({ id: environmentLeases.id, - metadata: sql>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`, - }).from(environmentLeases).where(and( - eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.issueId, issue.id), - eq(environmentLeases.heartbeatRunId, row.run.id), sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`, - )).orderBy(desc(environmentLeases.createdAt)).limit(1); + const { pluginTask, remoteProcess } = await readNativeRemoteExecutionOwnership(tx as unknown as Db, + { run: row.run, issueId: issue.id }); const failedRemoteRetry = row.run.status === "failed" && row.coordinator.phase === "retryable_failure" && row.coordinator.failureDetail?.recoveryMode === "exact_checkpoint_resume" && (remoteWorkspace !== null || pluginTask !== undefined); // A same-run bootstrap retry cannot take back an issue lock that moved @@ -1200,22 +1242,51 @@ export async function claimNativeRestartRecoveries(input: { return { kind: "blocked", runId: row.run.id, reason: "native_startup_cleanup_pending" } as const; } const remoteSandbox = profile.nativeWorkspaceSync != null || profile.remoteWorkspace != null; - const remoteProcess = remoteSandbox || !!pluginTask; const durableAuthority = (remoteWorkspace || pluginTask) && await hasNativeRunnerCheckpoint(tx, { companyId: row.run.companyId, issueId: row.coordinator.issueId, runId: row.run.id, }); - const takeover = await evaluateNativeControllerTakeover({ - owner: row.coordinator, - now, - remoteLeaseFenced: Boolean(durableAuthority), - coordinatedPreviousController: - input.coordinatedPreviousController ?? null, - }); + // Lease expiry alone does not prove a local in-flight owner stopped. + // Only this boot's own PID or the same known process space can make a + // local PID observation meaningful. Durable remote leases keep their + // existing released/expired authority instead of using local probes. + const processSpaceUnverified = !durableAuthority + && Boolean(row.coordinator.leaseOwner && row.coordinator.leaseExpiresAt) + && !isCurrentNativeControllerProcess(row.coordinator) + && (controllerHostHash === "unknown" || row.coordinator.controllerHostHash !== controllerHostHash); + const nonExecutionOwner = row.coordinator.leaseOwner?.startsWith("native-finalizer:") + || row.coordinator.leaseOwner?.startsWith("native-cleanup:"); + const evaluatedTakeover = nonExecutionOwner + ? { allowed: false, reason: "native_finalization_or_cleanup_owner" } + : await evaluateNativeControllerTakeover({ + owner: row.coordinator, + now, + requireRecordedProcessSpace: processSpaceUnverified, + remoteLeaseFenced: Boolean(durableAuthority), + coordinatedPreviousController: + input.coordinatedPreviousController ?? null, + }); + const independentControllerWait = !evaluatedTakeover.allowed + && ["controller_lease_held", "controller_lease_grace"].includes(evaluatedTakeover.reason); + // Controller death does not prove a foreign local Runner or provider + // stopped. Remote execution never uses local PID probes and still owes + // its existing checkpoint or never-launched proof below. + const independentlyStoppedRemoteController = remoteProcess && evaluatedTakeover.allowed + && ["controller_stopped", "controller_fenced"].includes(evaluatedTakeover.reason); + const namespaceAttentionNeeded = processSpaceUnverified && !nonExecutionOwner + && !independentControllerWait && !independentlyStoppedRemoteController; + const takeover = namespaceAttentionNeeded + ? { allowed: false, reason: remoteProcess ? "remote_controller_process_space_unverified" : "controller_process_space_unverified" } + : evaluatedTakeover; await recordLegacyHostFallbackProof(tx as unknown as Db, row.run, takeover, "native_restart_recovery"); if (!takeover.allowed && ["controller_host_unrecorded", "controller_host_unprovable"].includes(takeover.reason)) { await requestNativeControllerStopConfirmation(tx as unknown as Db, row.coordinator); } if (!takeover.allowed) { + if (namespaceAttentionNeeded) { + await surfaceNativeControllerNamespaceAttention(tx as unknown as Db, { + run: row.run, coordinator: row.coordinator, expiryEpoch: row.controllerLeaseExpiryEpoch, remoteProcess, + }); + } const previous = row.coordinator.recoveryHistory.at(-1); const waitAlreadyRecorded = previous?.disposition === "awaiting_evidence" && previous.reason === takeover.reason; const event = historyEntry({ @@ -1697,8 +1768,8 @@ export async function claimNativeRestartRecoveries(input: { // The owner row is locked: grant the full lease at this write. leaseExpiresAt: sql`clock_timestamp() + ${durableAuthority ? REMOTE_CONTROLLER_LEASE_MS : 20 * 60_000} * interval '1 millisecond'`, controllerBootId: controller.bootId, + controllerHostHash, controllerPid: controller.pid, - controllerHostHash: controller.hostHash ?? null, controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: generation, recoveryState: @@ -1739,6 +1810,17 @@ export async function claimNativeRestartRecoveries(input: { } as const; } + // A successful ordinary claim has now reconciled this exact earlier + // authority. Retire only its namespace attention; stale/other incidents + // remain owned and no operator confirmation is fabricated. + await issueRecoveryActionService(tx as unknown as Db).resolveActiveForIssue({ + companyId: row.run.companyId, sourceIssueId: row.coordinator.issueId, + kind: "active_run_watchdog", cause: "native_restart_recovery_blocked", + fingerprint: `native-controller-space:${row.run.id}:${nativeControllerAuthorityDigest(row.coordinator, row.controllerLeaseExpiryEpoch)}`, + status: "resolved", outcome: "restored", + resolutionNote: "Normal restart recovery verified the previous authority and committed its exact replacement claim.", + }, tx); + if ( claimKind !== "reattach_existing_runner" && claimKind !== "reattach_remote_runner" diff --git a/server/src/services/native-runtime/native-runner-ownership.ts b/server/src/services/native-runtime/native-runner-ownership.ts index 41d54bb5263..78a191b3026 100644 --- a/server/src/services/native-runtime/native-runner-ownership.ts +++ b/server/src/services/native-runtime/native-runner-ownership.ts @@ -1,8 +1,40 @@ -import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { createHash } from "node:crypto"; +import { environmentLeases, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm"; import { remoteExecutionHasStopped } from "../remote-execution-termination.js"; import { readSystemRecovery } from "../system-recovery-state.js"; +type NativeRemoteExecutionScope = { + run: Pick; + issueId: string; +}; + +export async function readNativeRemoteExecutionOwnershipForRuns(db: Db, inputs: readonly NativeRemoteExecutionScope[]) { + const key = (companyId: string, issueId: string, runId: string) => `${companyId}:${issueId}:${runId}`; + const leases = inputs.length ? await db.select({ id: environmentLeases.id, companyId: environmentLeases.companyId, + issueId: environmentLeases.issueId, runId: environmentLeases.heartbeatRunId, + metadata: sql>`jsonb_build_object('runnerTaskStartup', ${environmentLeases.metadata}->'runnerTaskStartup')`, + }).from(environmentLeases).where(and( + or(...inputs.map(input => and(eq(environmentLeases.companyId, input.run.companyId), + eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.heartbeatRunId, input.run.id)))), + sql`${environmentLeases.metadata}->>'runnerTaskLease' = 'true'`, + )).orderBy(desc(environmentLeases.createdAt)) : []; + const byScope = new Map(); + for (const lease of leases) { + const scope = key(lease.companyId, lease.issueId!, lease.runId!); + if (!byScope.has(scope)) byScope.set(scope, lease); + } + return inputs.map(input => { + const pluginTask = byScope.get(key(input.run.companyId, input.issueId, input.run.id)); + return { pluginTask, remoteProcess: input.run.runnerProfileJson?.nativeWorkspaceSync != null + || input.run.runnerProfileJson?.remoteWorkspace != null || !!pluginTask }; + }); +} + +export async function readNativeRemoteExecutionOwnership(db: Db, input: NativeRemoteExecutionScope) { + return (await readNativeRemoteExecutionOwnershipForRuns(db, [input]))[0]!; +} + export const nativeManualAdmissionOwnerColumns = { runId: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId, issueId: nativeRunFinalizations.issueId, phase: nativeRunFinalizations.phase, @@ -79,3 +111,16 @@ export function nativeRunnerOwnershipNotHeldCondition() { ), false )`; } + +/** Opaque binding for an operator's exact observed controller authority. Never + * expose the token itself. expiryEpoch must come from Postgres numeric epoch + * text, which is timezone independent and retains timestamp microseconds. */ +export function nativeControllerAuthorityDigest(owner: Pick, expiryEpoch: string | null): string { + return createHash("sha256").update(JSON.stringify([ + "paperclip.native-controller-authority.v1", owner.companyId, owner.issueId, owner.runId, + owner.leaseOwner, owner.controllerGeneration, owner.controllerBootId, owner.controllerHostHash, + owner.controllerPid, owner.controllerProcessStartedAt?.toISOString() ?? null, expiryEpoch, + ])).digest("hex"); +} diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 335225b7636..729845d8a14 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -1,3 +1,4 @@ +import { currentControllerProcess, currentProcessSpaceHash } from "../controller-process-identity.js"; import { renewNativeSessionExecutionLease, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js"; import { NativeControllerRecoveryWaitingError } from "./native-controller-recovery.js"; @@ -810,6 +811,15 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); } }, 360_000); + // These historical controllers belong to the same kernel/PID namespace as + // the real local processes this fixture starts. Other-space/legacy NULL + // authority has separate explicit negatives; do not invent a constant hash. + async function knownFixtureControllerHostHash() { + const current = await currentControllerProcess(); + expect(current.hostHash, "Same-host recovery fixture requires actual known process space").not.toBe("unknown"); + return current.hostHash; + } + async function persistRestartEvidence(input: { fixture: Awaited>; runnerPid: number; @@ -863,6 +873,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); leaseOwner: "dead-controller", leaseExpiresAt: new Date(now.getTime() + 60_000), controllerBootId: "dead-controller-boot", + controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: 2_000_000_000, controllerProcessStartedAt: new Date("2026-09-04T11:00:00.000Z"), controllerGeneration: 1, @@ -895,6 +906,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); leaseOwner: "dead-controller", leaseExpiresAt: new Date(now.getTime() + 60_000), controllerBootId: "dead-controller-boot", + controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: 2_000_000_003, controllerProcessStartedAt: new Date("2026-09-04T11:00:00.000Z"), controllerGeneration: 1, @@ -2238,6 +2250,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); leaseOwner: "dead-controller", leaseExpiresAt: new Date(now.getTime() + 60_000), controllerBootId: "dead-controller-boot", + controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: 2_000_000_002, controllerProcessStartedAt: new Date("2026-09-04T10:00:00.000Z"), controllerGeneration: 4, @@ -2280,6 +2293,73 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); ]); }); + it.each(["stopped", "fenced"] as const)("keeps foreign local Runner and provider ownership after its controller is %s", async state => { + const fixture = await seedRun(`FOREIGN-LOCAL-${state}-${randomUUID()}`); + const oldBoot = randomUUID(), foreignSpace = `foreign:${currentProcessSpaceHash()}`; + const runnerPid = 2_000_000_001, runnerGroup = 2_000_000_002, providerPid = 2_000_000_003; + const controllerPid = 2_000_000_004, startedAt = new Date(Date.now() - 600_000); + // Historical in-flight local execution on the other process space. A + // controller-instance stop/fence says nothing about these saved children. + await fixture.db.update(heartbeatRuns).set({ processPid: runnerPid, processGroupId: runnerGroup, + processStartedAt: startedAt, processHostHash: foreignSpace, + runnerProfileJson: { ...fixture.preparedProfile, sessionCheckpoint: { + identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, + sessionId: fixture.native.normalizedSessionId }, + providerSessionId: "foreign-provider-session", activeTurnId: "foreign-in-flight-turn", + process: { providerPid, providerProcessStartedAt: startedAt.toISOString() }, + } }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: oldBoot, controllerPid, + controllerProcessStartedAt: startedAt, controllerHostHash: foreignSpace, + leaseOwner: `foreign-in-flight:${randomUUID()}`, leaseExpiresAt: new Date(0), attempt: 1, controllerGeneration: 1, + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + await fixture.db.insert(controllerInstances).values({ bootId: oldBoot, processSpace: foreignSpace, + pid: controllerPid, leaseExpiresAt: new Date(Date.now() - 1_000), + ...(state === "stopped" ? { stoppedAt: new Date() } : { fencedAt: new Date() }), + }); + registerControllerInstanceDb(fixture.db); + const [before] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const probes: number[] = [], recordedPids = new Set([runnerPid, runnerGroup, providerPid, controllerPid]); + const kill = process.kill.bind(process); + const pidProbe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => { + if (signal === 0 && recordedPids.has(Math.abs(pid))) { + probes.push(pid); + throw Object.assign(new Error("Absent local PID is not foreign execution stop proof"), { code: "ESRCH" }); + } + return kill(pid, signal); + }); + const claimed = vi.fn(), stamped = vi.fn(), unsubscribe = onControllerStamp(stamped); + try { + const dispositions = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId], onClaim: claimed }); + const [after] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + const stopped = await hasNativeLocalProcessStop(fixture.db, companyId, fixture.runId); + expect(probes).toEqual([]); + expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: fixture.runId, + reason: "controller_process_space_unverified" }]); + expect(claimed).not.toHaveBeenCalled(); + expect(stamped).not.toHaveBeenCalled(); + expect(stopped).toBe(false); + expect(after).toMatchObject({ phase: before.phase, attempt: before.attempt, + controllerGeneration: before.controllerGeneration, controllerBootId: before.controllerBootId, + controllerPid: before.controllerPid, controllerHostHash: before.controllerHostHash, + leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, resultId: null }); + expect(run).toMatchObject({ status: "running", processPid: runnerPid, processGroupId: runnerGroup, + processStartedAt: startedAt, processHostHash: foreignSpace }); + expect(run.runnerProfileJson).toEqual({ ...fixture.preparedProfile, sessionCheckpoint: { + identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, + sessionId: fixture.native.normalizedSessionId }, + providerSessionId: "foreign-provider-session", activeTurnId: "foreign-in-flight-turn", + process: { providerPid, providerProcessStartedAt: startedAt.toISOString() }, + } }); + const [attention] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(attention).toMatchObject({ status: "active", cause: "native_restart_recovery_blocked", + evidence: { runId: fixture.runId, reason: "controller_process_space_unverified" } }); + expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toEqual([]); + } finally { unsubscribe(); pidProbe.mockRestore(); resetControllerInstanceLeaseForTests(); } + }); + it("recovers a preparation with no runner PID once its foreign controller stops", async () => { const fixture = await seedRun("REMOTE-PREPARATION-OWNER"); const oldBoot = randomUUID(); @@ -2295,6 +2375,9 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); registerControllerInstanceDb(fixture.db); const stamped = vi.fn(), unsubscribe = onControllerStamp(stamped); try { + expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [randomUUID()] })).toEqual([]); + expect(stamped).not.toHaveBeenCalled(); const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, restartKind: "hard", runIds: [fixture.runId] }); expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "controller_lease_held", retryAt: expect.any(Date) }); @@ -2316,6 +2399,52 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); } finally { unsubscribe(); resetControllerInstanceLeaseForTests(); } }); + it.each(["grace", "fenced"] as const)("uses durable foreign controller %s evidence before classifying its namespace", async state => { + const fixture = await seedRun(`REMOTE-CONTROLLER-${state}-${randomUUID()}`); + const oldBoot = randomUUID(); + await fixture.db.update(heartbeatRuns).set({ + runnerProfileJson: { ...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace() }, + }).where(eq(heartbeatRuns.id, fixture.runId)); + await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: oldBoot, controllerPid: 7, + controllerProcessStartedAt: new Date(Date.now() - 600_000), controllerHostHash: "previous-pid-space", + leaseOwner: "previous-preparation", leaseExpiresAt: new Date(0), attempt: 1, controllerGeneration: 1, + }).where(eq(nativeRunFinalizations.runId, fixture.runId)); + await fixture.db.insert(controllerInstances).values({ bootId: oldBoot, processSpace: "previous-pid-space", + pid: 7, leaseExpiresAt: new Date(Date.now() - 1_000), + ...(state === "fenced" ? { fencedAt: new Date() } : {}), + }); + registerControllerInstanceDb(fixture.db); + const originalKill = process.kill.bind(process); + const pidProbe = vi.spyOn(process, "kill").mockImplementation((pid, signal) => { + if (pid === 7 && signal === 0) throw new Error("A foreign controller PID must not be probed"); + return originalKill(pid, signal); + }); + const stamped = vi.fn(), unsubscribe = onControllerStamp(stamped); + try { + const [before] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + const [disposition] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + const [after] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + if (state === "grace") { + expect(disposition).toMatchObject({ kind: "awaiting_evidence", reason: "controller_lease_grace", retryAt: expect.any(Date) }); + expect(stamped).not.toHaveBeenCalled(); + expect(after).toMatchObject({ leaseOwner: before.leaseOwner, leaseExpiresAt: before.leaseExpiresAt, + controllerBootId: before.controllerBootId, controllerHostHash: before.controllerHostHash, + controllerGeneration: before.controllerGeneration, attempt: before.attempt }); + await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, restartKind: "hard", runIds: [fixture.runId] }); + const [repeated] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(repeated.updatedAt).toEqual(after.updatedAt); + } else { + expect(disposition).toMatchObject({ kind: "bootstrap_incomplete", runId: fixture.runId }); + expect(stamped).toHaveBeenCalledOnce(); + expect(after).toMatchObject({ controllerGeneration: 2, attempt: 1, resultId: null }); + } + expect((await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)))[0]) + .toMatchObject({ status: "running", processPid: null }); + expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))).toEqual([]); + } finally { unsubscribe(); pidProbe.mockRestore(); resetControllerInstanceLeaseForTests(); } + }); + it("recovers an unclaimed remote startup once without requiring a provider checkpoint", async () => { const fixture = await seedRun("REMOTE-UNCLAIMED"); await fixture.db.update(heartbeatRuns).set({ @@ -2473,7 +2602,8 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); const bootId = randomUUID(); const leaseOwner = `${bootId}:1:${randomUUID()}`; await fixture.db.update(nativeRunFinalizations).set({ - attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: bootId, controllerPid: 2_000_000_001, + attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: bootId, + controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: 2_000_000_001, controllerProcessStartedAt: new Date(Date.now() - 60_000), leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), }).where(eq(nativeRunFinalizations.runId, runId)); if (options.checkpoint !== false) { @@ -2708,6 +2838,7 @@ process.on("SIGTERM", () => child.kill("SIGTERM")); await fixture.db.update(nativeRunFinalizations).set({ attempt: options.attempt ?? 1, controllerGeneration: 1, controllerBootId: options.liveController ? current.bootId : randomUUID(), + controllerHostHash: await knownFixtureControllerHostHash(), controllerPid: options.liveController ? current.pid : 2_000_000_001, controllerProcessStartedAt: options.liveController ? current.processStartedAt : new Date(Date.now() - 60_000), leaseOwner: `previous:${randomUUID()}`, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 0b388583fc0..01db246fb72 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1,3 +1,4 @@ +import { currentControllerProcess } from "../controller-process-identity.js"; import { activeNativeExecutionRunIds } from "./native-active-runs.js"; import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js"; import { readIdleLocalWork, markIdleStartupComplete, markIdleIngressTracked } from "../idle-local-work.js"; @@ -9810,6 +9811,7 @@ describe("native session execution lease fencing", () => { expect(profileWrites).toBe(2); expect(nullWrites).toBe(0); const claim = updates.find(update => update.table === nativeRunFinalizations && update.values.phase === "observed" && update.values.leaseOwner)!.values; + expect(claim.controllerHostHash).toBe((await currentControllerProcess()).hostHash); expect(recordAdmission).toHaveBeenCalledExactlyOnceWith(db, { session, identity, waitForClaim: false, admissionEpoch: (profile.nativeProviderPackAdmissionState as Record).admissionEpoch, claim: { leaseOwner: claim.leaseOwner, attempt: claim.attempt, controllerGeneration: claim.controllerGeneration, controllerBootId: claim.controllerBootId } }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 28285036118..e731e4dda80 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { currentControllerProcess } from "../controller-process-identity.js"; import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js"; import { isGovernancePauseStop } from "../governance-pause-stop.js"; import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js"; @@ -9902,6 +9903,7 @@ async function executePaperclipNativeSessionAttempt( const leaseOwner = input.leaseOwner ?? `${effectiveRunnerInstanceId}:${randomUUID()}`; const controller = await currentNativeControllerIdentity(); + const controllerHostHash = (await currentControllerProcess()).hostHash; const executionLeaseTtlMs = input.runnerExecutionTarget?.kind === "remote" ? REMOTE_CONTROLLER_LEASE_MS : NATIVE_SESSION_EXECUTION_LEASE_TTL_MS; let attempt: number; @@ -10087,8 +10089,8 @@ async function executePaperclipNativeSessionAttempt( // Start the full lease after claim locks and lineage reads finish. leaseExpiresAt: sql`clock_timestamp() + ${executionLeaseTtlMs} * interval '1 millisecond'`, controllerBootId: controller.bootId, + controllerHostHash, controllerPid: controller.pid, - controllerHostHash: controller.hostHash ?? null, controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: nextControllerGeneration, failureCode: null, diff --git a/server/src/services/owner-proof.test.ts b/server/src/services/owner-proof.test.ts index 1dd052eee5f..6258a17d639 100644 --- a/server/src/services/owner-proof.test.ts +++ b/server/src/services/owner-proof.test.ts @@ -45,6 +45,32 @@ describe("proveOwnerGone: local owners", () => { expect(await proveOwnerGone(stamp(), probe())).toEqual({ verdict: "gone", reason: "pid_dead" }); }); + it.each([null, "foreign-space", "unknown"])("does not probe a recorded-space-required owner in %s space", async processSpace => { + const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null); + const result = await proveOwnerGone(stamp({ processSpace, processSpaceUnrecorded: processSpace === null, + requireRecordedProcessSpace: true }), probe({ pidExists, readStartedAt })); + expect(result).toEqual({ verdict: "unprovable", reason: processSpace === null ? "process_space_unrecorded" + : processSpace === "unknown" ? "unknown_process_space" : "different_process_space" }); + expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled(); + }); + + it.each(["stopped", "fenced"] as const)("retains independent controller-instance %s evidence for an unrecorded Native space", async state => { + const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null); + const controllerInstance = vi.fn(async () => state); + expect(await proveOwnerGone(stamp({ processSpace: null, processSpaceUnrecorded: true, + requireRecordedProcessSpace: true }), probe({ pidExists, readStartedAt, controllerInstance }))) + .toEqual({ verdict: "gone", reason: state === "stopped" ? "controller_stopped" : "controller_fenced" }); + expect(controllerInstance).toHaveBeenCalledOnce(); + expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled(); + }); + + it("accepts audited operator stop evidence without probing a recorded-space-required owner", async () => { + const pidExists = vi.fn(() => false), readStartedAt = vi.fn(async () => null); + expect(await proveOwnerGone(stamp({ processSpace: "operator_attested", requireRecordedProcessSpace: true }), + probe({ pidExists, readStartedAt }))).toEqual({ verdict: "gone", reason: "operator_attested" }); + expect(pidExists).not.toHaveBeenCalled(); expect(readStartedAt).not.toHaveBeenCalled(); + }); + it("judges a row saved before its PID space locally for one release, on single-host servers only", async () => { const legacy = stamp({ host: null, processSpace: null, processSpaceUnrecorded: true }); expect(await proveOwnerGone(legacy, probe())).toEqual({ verdict: "gone", reason: "pid_dead", legacyHostFallback: true }); diff --git a/server/src/services/owner-proof.ts b/server/src/services/owner-proof.ts index 61c6129346e..cd45d1a414b 100644 --- a/server/src/services/owner-proof.ts +++ b/server/src/services/owner-proof.ts @@ -33,6 +33,7 @@ export type LocalOwnerStamp = { /** Record kind or subsystem that wrote the stamp, for reports only. */ scope: string; generation: number | null; + fenceToken?: string | null; /** Hostname at stamp time. Informational; `processSpace` is the proof key. */ host: string | null; /** Hash of host, kernel boot and PID namespace (`currentProcessSpaceHash`). */ @@ -58,6 +59,7 @@ export type LocalOwnerStamp = { * An ephemeral (multi-host) server never judges them. */ processSpaceUnrecorded?: boolean; + requireRecordedProcessSpace?: boolean; /** When the owned record was last written, if the caller knows (for the controller lease proof). */ touchedAt?: string | null; }; @@ -247,7 +249,7 @@ export async function proveOwnerGone( async function proveLocalOwnerGone(stamp: LocalOwnerStamp, probe: ProcessProbe): Promise { const proof = await proveLocalProcessGone(stamp, probe); - if (proof.verdict !== "unprovable" || !CROSS_SPACE_REASONS.has(proof.reason) || !stamp.bootId || !probe.controllerInstance + if (proof.verdict !== "unprovable" || (!CROSS_SPACE_REASONS.has(proof.reason) && !(stamp.requireRecordedProcessSpace && proof.reason === "process_space_unrecorded")) || !stamp.bootId || !probe.controllerInstance || validPid(stamp.processGroupId ?? null)) return proof; // Another host or PID space: the owner's server boot answers through its lease. Only a record the // server process itself held qualifies; agent processes in a recorded group may outlive it. @@ -272,7 +274,7 @@ async function proveLocalProcessGone(stamp: LocalOwnerStamp, probe: ProcessProbe // fallback on single-host servers; an ephemeral server cannot assume the // PID was recorded here, so a board operator decides. const legacy = !stamp.processSpace && stamp.processSpaceUnrecorded === true; - if (legacy && ephemeralServerEnabled()) return { verdict: "unprovable", reason: "process_space_unrecorded" }; + if (legacy && (stamp.requireRecordedProcessSpace || ephemeralServerEnabled())) return { verdict: "unprovable", reason: "process_space_unrecorded" }; if (!validPid(stamp.pid) || (!stamp.processSpace && !legacy)) return { verdict: "unprovable", reason: "missing_identity" }; if (legacy) { const proof = await proveLocalOwnerGone({ ...stamp, processSpace: probe.processSpace, host: null, diff --git a/server/src/services/recovery/ownership-registry.ts b/server/src/services/recovery/ownership-registry.ts index 00bb3a345bf..bb4954921c8 100644 --- a/server/src/services/recovery/ownership-registry.ts +++ b/server/src/services/recovery/ownership-registry.ts @@ -33,7 +33,7 @@ import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js import { OWNER_GONE_ATTESTATION_KEY, attestOwnerGoneByRecordVersion, requestOwnerGoneConfirmation } from "./owner-gone-attestation.js"; import { requestNativeAcceptedResultLeaseOperatorAction, - settleNativeAcceptedResultLease, + settleNativeAcceptedResultLease, classifyNativeAcceptedResultLeaseOwners, } from "../native-runtime/native-accepted-result-lease-settlement.js"; import { settleClaimedWakeRequest } from "./wake-request-claimed-settlement.js"; import { settleAbandonedRemoteSnapshotSave, settleAbandonedWorkspaceCommandOperation, settleAbandonedWorkspaceFinalizeAttempt } from "./workspace-finalize-attempt-settlement.js"; @@ -656,6 +656,11 @@ const agentLifecycleLeaseKind: OwnableKind = { }, }; +const acceptedNativeResultLease = nrfKind("native_accepted_result_lease", "Accepted-result executor lease", "accepted_result", + and(isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, TERMINAL_PHASES), + isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, + sql`${nativeRunFinalizations.leaseExpiresAt} > now()`)); + export const OWNABLE_KINDS: readonly OwnableKind[] = [ // Native runtime and Runner nrfKind("native_executor_claim", "Executor execution claim (fresh run)", "ownership", @@ -671,10 +676,13 @@ export const OWNABLE_KINDS: readonly OwnableKind[] = [ // exact lease so finalization runs now (RC-41). // Outstanding leases only: an expired lease no longer holds the result, // so it is not in progress for this kind (finalization reclaims it). - ...nrfKind("native_accepted_result_lease", "Accepted-result executor lease", "accepted_result", - and(isNotNull(nativeRunFinalizations.resultId), notInArray(nativeRunFinalizations.phase, TERMINAL_PHASES), - isNotNull(nativeRunFinalizations.leaseOwner), notFinalizerOrCleanup, - sql`${nativeRunFinalizations.leaseExpiresAt} > now()`)), + ...acceptedNativeResultLease, + listInProgress: async (db, limit, offset) => { + const listed = await acceptedNativeResultLease.listInProgress!(db, limit, offset); + return { ...listed, records: await classifyNativeAcceptedResultLeaseOwners(db, listed.records) }; + }, + listForRuns: async (db, runIds) => classifyNativeAcceptedResultLeaseOwners(db, + await acceptedNativeResultLease.listForRuns!(db, runIds)), settle: settleNativeAcceptedResultLease, onUnprovable: requestNativeAcceptedResultLeaseOperatorAction, }, diff --git a/ui/src/components/IssueRecoveryActionCard.test.tsx b/ui/src/components/IssueRecoveryActionCard.test.tsx index cd513226984..886a1491a93 100644 --- a/ui/src/components/IssueRecoveryActionCard.test.tsx +++ b/ui/src/components/IssueRecoveryActionCard.test.tsx @@ -1,11 +1,16 @@ // @vitest-environment jsdom +import { canBoardConfirmOwnerStop } from "../lib/recovery-reconcile"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { createRoot } from "react-dom/client"; import { flushSync } from "react-dom"; import type { AnchorHTMLAttributes, ReactElement } from "react"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { Agent, IssueRecoveryAction } from "@paperclipai/shared"; -import { IssueRecoveryActionCard, deriveRecoveryCardState } from "./IssueRecoveryActionCard"; +import { IssueRecoveryActionCard, NativeControllerContainmentRecovery, deriveRecoveryCardState } from "./IssueRecoveryActionCard"; + +const containmentApi = vi.hoisted(() => ({ resolveRecoveryAction: vi.fn() })); +vi.mock("../api/issues", () => ({ issuesApi: containmentApi })); vi.mock("@/lib/router", () => ({ Link: ({ children, to, ...props }: AnchorHTMLAttributes & { to: string }) => ( @@ -1168,3 +1173,93 @@ it("requires owner-stop recovery even for board false-positive dispositions", () expect(node.querySelector("[data-testid='recovery-action-resolve-trigger']")).toBeNull(); expect(node.textContent).not.toContain("False positive"); }); + + +describe("original-host controller containment in task recovery", () => { + let client: QueryClient; + const digest = "a".repeat(64); + const runId = "00000000-0000-4000-8000-000000000041"; + const onRecorded = vi.fn(); + beforeEach(() => { + client = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } }); + containmentApi.resolveRecoveryAction.mockReset().mockResolvedValue({}); + onRecorded.mockReset(); + }); + afterEach(() => client.clear()); + function mount(canManage = true, reason = "controller_process_space_unverified", authority = digest) { + const action = buildAction({ kind: "active_run_watchdog", ownerType: "board", cause: "native_restart_recovery_blocked", + evidence: { runId, reason, controllerAuthority: { schema: "paperclip.native-controller-authority.v1", digest: authority } } }); + const issue = { id: "ordinary-task", status: "in_progress" as const, activeRecoveryAction: action }; + const element = ; + if (!root) return render(element); + act(() => root?.render(element)); + return container!; + } + function fillFields(node: HTMLDivElement) { + act(() => { + const area = node.querySelector("textarea")!; + Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(area, + "Verified the original deployment controller and provider stopped; completed actions were reviewed."); + area.dispatchEvent(new Event("input", { bubbles: true })); + const select = node.querySelector("select")!; + select.value = "mixed"; + select.dispatchEvent(new Event("change", { bubbles: true })); + }); + } + function recordButton(node: HTMLDivElement) { + return Array.from(node.querySelectorAll("button")).find(button => button.textContent === "Record containment")!; + } + it("requires separate controller and provider confirmations on an ordinary task", async () => { + const node = mount(); + expect(node.textContent).toContain("original host"); + expect(node.textContent).toContain("does not retry"); + expect(node.textContent).not.toContain(digest); + fillFields(node); + const boxes = node.querySelectorAll('[role="checkbox"]'); + expect(recordButton(node).disabled).toBe(true); + click(boxes[0]); + expect(recordButton(node).disabled).toBe(true); + click(boxes[1]); + expect(recordButton(node).disabled).toBe(false); + click(recordButton(node)); + await new Promise(resolve => setTimeout(resolve, 0)); + expect(containmentApi.resolveRecoveryAction).toHaveBeenCalledOnce(); + expect(containmentApi.resolveRecoveryAction).toHaveBeenCalledWith("ordinary-task", expect.objectContaining({ + outcome: "restored", sourceIssueStatus: "in_progress", executionReconciliation: { + runId, controllerStopped: true, providerStopped: true, controllerAuthorityDigest: digest, + actionOutcome: "mixed", outcomeEvidence: "Verified the original deployment controller and provider stopped; completed actions were reviewed.", + }, + })); + }); + it("does not expose a confirmation form to non-board runtime managers", () => { + const node = mount(false); + expect(node.querySelector("button")).toBeNull(); + expect(node.textContent).toContain("company owner"); + expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled(); + }); + it.each(["owner", "admin", "operator", "viewer"] as const)("shows the containment form only for a board %s with owner-stop authority", role => { + const allowed = canBoardConfirmOwnerStop("company", { source: "session", user: null, userId: "board-user", keyId: null, companyIds: ["company"], isInstanceAdmin: false, + memberships: [{ companyId: "company", status: "active", membershipRole: role }] }); + const node = mount(allowed); + expect(node.querySelectorAll('[role="checkbox"]').length).toBe(role === "owner" || role === "admin" ? 2 : 0); + expect(Boolean(node.querySelector("button"))).toBe(role === "owner" || role === "admin"); + expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled(); + }); + it("does not offer local containment for remote or unrelated recovery", () => { + const node = mount(true, "remote_controller_process_space_unverified"); + expect(node.querySelector("section")).toBeNull(); + mount(true, "some_other_hold"); + expect(node.querySelector("section")).toBeNull(); + }); + it("clears old confirmations when the server supplies a different authority snapshot", () => { + const node = mount(); + fillFields(node); + for (const box of node.querySelectorAll('[role="checkbox"]')) click(box); + expect(recordButton(node).disabled).toBe(false); + mount(true, "controller_process_space_unverified", "b".repeat(64)); + expect(recordButton(node).disabled).toBe(true); + expect(node.querySelector("textarea")?.value).toBe(""); + expect(containmentApi.resolveRecoveryAction).not.toHaveBeenCalled(); + }); +}); diff --git a/ui/src/components/IssueRecoveryActionCard.tsx b/ui/src/components/IssueRecoveryActionCard.tsx index e61adbbb966..57dba70849f 100644 --- a/ui/src/components/IssueRecoveryActionCard.tsx +++ b/ui/src/components/IssueRecoveryActionCard.tsx @@ -1,3 +1,9 @@ +import { useMutation } from "@tanstack/react-query"; +import { issuesApi } from "../api/issues"; +import { describeError } from "../api/errors"; +import { Checkbox } from "./ui/checkbox"; +import { NativeSelect } from "./ui/select"; +import type { Issue } from "@paperclipai/shared"; import { isNativeWorkspaceExportRepairCause } from "@paperclipai/shared"; import { useWorkspaceIsolationControls } from "@/hooks/useWorkspaceIsolationControls"; import { requiresExecutionReconciliation } from "@paperclipai/shared"; @@ -1510,3 +1516,55 @@ export function IssueRecoveryActionCard({ export type { IssueRecoveryActionStatus }; export default IssueRecoveryActionCard; + +/** Separate original-host confirmations: the opaque authority is supplied by + * the server snapshot, never entered or edited by the operator. */ +export function NativeControllerContainmentRecovery({ issue, canManage, onRecorded }: { + issue: Pick; canManage: boolean; onRecorded: () => void; +}) { + const action = issue.activeRecoveryAction; + const [controllerStopped, setControllerStopped] = useState(false); + const [providerStopped, setProviderStopped] = useState(false); + const [outcome, setOutcome] = useState<"" | "completed" | "not_performed" | "mixed">(""); + const [evidence, setEvidence] = useState(""); + const authority = action?.evidence?.controllerAuthority; + const digest = authority && typeof authority === "object" && !Array.isArray(authority) + ? (authority as Record).digest : undefined; + const runId = action?.evidence?.runId; + const submit = useMutation({ mutationFn: async () => { + if (!action || typeof runId !== "string" || typeof digest !== "string" || !outcome + || !controllerStopped || !providerStopped) throw new Error("Confirm both original-host stops and review the action outcomes."); + return issuesApi.resolveRecoveryAction(issue.id, { actionId: action.id, outcome: "restored", sourceIssueStatus: issue.status, + executionReconciliation: { runId, controllerStopped: true, providerStopped: true, + controllerAuthorityDigest: digest, actionOutcome: outcome, outcomeEvidence: evidence.trim() } }); + }, onSuccess: onRecorded }); + if (!action || action.kind !== "active_run_watchdog" || action.ownerType !== "board" + || action.cause !== "native_restart_recovery_blocked" || action.evidence?.reason !== "controller_process_space_unverified" + || !["active", "escalated"].includes(action.status) || typeof digest !== "string") return null; + return
+

Verify the original execution stopped

+

Paperclip cannot compare process IDs across hosts or kernel sessions. Stop both the controller and provider on the original host before recording evidence. The saved lease must also expire and environment/workspace cleanup must finish. Recording containment does not retry the task.

+ {canManage ? <> + + + +