From e0c6992426f107e98a0361efbb2e637c9d6b4c00 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:17:00 -0600 Subject: [PATCH 01/16] fix(heartbeat): prevent saved input starvation during queue reloads (#18819) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accepted messages and task Retry requests use durable wake receipts. > - The recovery worker loads receipts in pages of 100 and checks batches of 25. > - Company activity can reload those pages before the pending batch drains. > - Reloads reset ready receipts' dispatch times and can repeatedly favor receipts already checked. > - This pull request preserves the priority of unchecked ready receipts so saved input can reach normal admission. ## Linked Issues or Issue Description **What happened?** Saved input can remain deferred after the preceding run finishes. Repeated company eligibility changes reset the dispatch times of pending receipts during paginated reconstruction. A PostgreSQL regression checks only 75 of 178 receipts on the original code despite twelve eligibility changes. **Expected behavior** Each ready receipt must reach its admission check. Existing wait deadlines, retry backoffs, authorization, and execution gates must still apply. **Steps to reproduce** 1. Seed more than one page of deferred receipts. 2. Keep those receipts pending after their checks. 3. Send company eligibility notifications between scheduler passes. 4. Check that every receipt is reconsidered, including receipts later in the backlog. **Paperclip version or commit** Reproduced on `367f29478d`. **Deployment mode** Built from source with PostgreSQL. The defect occurs before provider execution. Related: #10195 reports a similar starvation symptom in a different promotion loop. #18621 added saved-input Retry after a successful run. This change fixes fairness in the deferred-receipt worker. ## What Changed - Add an optional entry merge callback to scoped deadline reconstruction. - Use it in the deferred-receipt worker to retain unread ready receipts' priority while refreshing their state. - Place receipts whose wait has expired into the ready queue during reconstruction. - Keep real wait, claim, and dispatch deadlines authoritative. - Add a paginated starvation regression, real process dispatch coverage, and GitHub lock-collision recovery coverage. - Document the recovery fairness rule. ## Verification - The PostgreSQL starvation regression checks only 75 of 178 receipts before the fix. It checks all 178 afterward, both with ordinary receipts and with expired execution waits. - Seven focused scheduler cases pass, including moved interrupt grace, moved native error deadline, fresh claim/wait deadlines, and unchanged-gate grace. - GitHub endpoint lock-collision recovery, retained board authority, HTTP Retry, and five current admission-gate cases pass. - Real process acceptance passes on the final code: task Retry crosses the repeatedly invalidated backlog, reconstructs its persisted queued run after a worker restart, saves `receipt-proof` in run output, and has one successor for the Retry receipt. The test holds executor capacity while advancing queue time, then starts fresh workers at PostgreSQL time before releasing a real process lease. The defect is in admission scheduling before provider execution; this acceptance uses PostgreSQL and a local process adapter. - The full workspace build and typecheck (`node scripts/typecheck-low-memory.mjs`) pass locally. The affected worker suites pass 52 of 53 cases together; the existing native multi-company batch case times out in that combined run and passes in isolation without code changes. All new regression and acceptance cases pass. - Local `pnpm test:run` stops in its first chat shard on three 15-second timeouts (Teams long native HTTP picture delivery, direct final-response materialization, and contended inbound notices). All three pass in isolation without code changes. Later full-suite groups did not run locally after that shard failure. - [The full GitHub Actions matrix passes](https://github.com/paperclipai/paperclip/actions/runs/41697489374) on `ac91df2cc6db6c77529d923177b375b8c92efd45`, including all general-test, serialized-server, browser, build, and typecheck jobs. Server shard 8 initially failed an upstream process-adoption test and passed its one rerun without code changes. - [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/18819#issuecomment-6612091535) on the same current head, with no actionable findings or unresolved review threads. The separate Paperclip Review check is neutral; its previous request was bound to an older commit. Focused command: ```sh pnpm exec vitest run server/src/__tests__/deferred-execution-receipt-work.test.ts server/src/__tests__/scoped-deadline-work.test.ts ``` ## Risks - Ready receipt order changes during repeated reconstruction. Admission still reads current durable state. - A ready receipt must not inherit an obsolete priority after it acquires a future deadline. The merge applies only when both pending and loaded entries are immediate. Deadline safety tests cover that boundary. - The merge callback is optional. Other scoped workers retain their current reconstruction behavior. ## Model Used OpenAI `gpt-6.1-sol` in Codex, max reasoning effort, with tool use and local code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused regressions and acceptance; full-run timing limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 2 + .../chat-channels.integration.test.ts | 39 +++++++++ .../deferred-execution-receipt-work.test.ts | 84 ++++++++++++++++++- .../heartbeat/deferred-receipt-work.ts | 22 +++-- server/src/services/scoped-deadline-work.ts | 5 +- 5 files changed, 140 insertions(+), 12 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 325148106d2..d93ae631b6c 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -756,6 +756,8 @@ Normal issue-thread feedback can still wake the current assignee according to th Restart recovery revisits accepted deferred messages through the latest issue run, including historical mention queues for another agent. Ordinary messages for a former owner retain their assignment handoff. Native sources require their matching committed finalization, settled environment leases, and released finalizer and copyback ownership before recovery clears old issue claims. A successful retained warm release can satisfy the lease check. Released local bookkeeping without a provider allocation and a matching SSH bookkeeping release need no provider cleanup receipt. Plugin releases record their successful acknowledgement. Pending or failed cleanup and durable owners still hold work. A successful fenced cleanup completion clears its in-flight claim in the same database write, so a restart before retry bookkeeping cannot strand saved input. The normal release checks preserve cleanup holds, pauses, and operator Stops. Recovery checks for a competing turn under the issue lock before it promotes saved input. +Repeated eligibility notifications must not move an unchecked, ready deferred receipt behind receipts already reconsidered. Paginated reconstruction preserves pending ready receipts' dispatch priority while refreshing their current state. Receipts whose wait deadline has passed join the ready queue on reconstruction. Future wait grace periods, claimed-work deadlines, infrastructure retry backoffs, and current admission checks remain effective. + A paused recipient does not fail an accepted deferred request. Release and periodic recovery preserve its pending receipt and may process an eligible request behind it. Resuming the agent allows normal admission to reconsider the saved input. Missing, foreign, terminated, and unapproved recipients retain their rejection rules. Explicit task holds and operator Stops still prevent unauthorized execution. This does not replay historical failed requests. Accepted agent feedback must survive a child changing to `done` before its active run exits. Deferred wake promotion may reopen that completed child only for its current assignee, with explicit agent resume intent and live tracked comments from another author. Claim promotion before reopening. Cancelled tasks, deleted comments, self-authored comments, empty continuations, and agent continuations without explicit intent keep their existing suppression rules. Normal pause, ownership, authorization, and budget gates still apply. diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 25d571411f2..89f7c9741d8 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -64441,6 +64441,45 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { } finally { await f.service.shutdown(); } }); + it("retries saved GitHub input after an endpoint admission lock collision", async () => { + const f = await successfulGitHubQueuedInputFixture(); + const heartbeat = heartbeatService(db); + const failures: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, + onError: error => failures.push(error) }); + let unlock!: () => void, locked!: () => void; + const acquired = new Promise(resolve => { locked = resolve; }); + const released = new Promise(resolve => { unlock = resolve; }); + const competingAdmission = db.transaction(async tx => { + await tx.select({ id: chatEndpoints.id }).from(chatEndpoints) + .where(eq(chatEndpoints.id, f.endpoint.id)).for("no key update"); + locked(); + await released; + }); + try { + await acquired; + await heartbeat.registerDeferredReceiptWork(coordinator, () => true).ready; + const priorFailures = failures.length; + const retry = await requestTaskRetry(f.input); + await expect.poll(() => failures.length, { timeout: 5_000 }).toBeGreaterThan(priorFailures); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0].runId).toBeNull(); + unlock(); await competingAdmission; + await expect.poll(async () => (await readAgentWakeupRequestsById(f.pending!.id))[0].runId, + { timeout: 40_000 }).toBeTruthy(); + await expect.poll(async () => (await readAgentWakeupRequestsById(retry.requestId!))[0].status, + { timeout: 40_000 }).toBe("coalesced"); + const [saved] = await readAgentWakeupRequestsById(f.pending!.id); + expect((await readHeartbeatRunsByCompanyId(f.fixture.companyId)) + .filter(run => run.wakeupRequestId === f.pending!.id)).toHaveLength(1); + expect((await readAgentWakeupRequestsById(retry.requestId!))[0].runId).toBe(saved.runId); + expect((await readHeartbeatRunsById(f.runId))[0]).toMatchObject({ status: "succeeded" }); + } finally { + unlock(); await competingAdmission; + await coordinator.stop(); + await f.service.shutdown(); + } + }, 90_000); + it.each(["reconnected", "newer_input", "reconnected_newer_input", "snapshot_newer_input"] as const)( "admits current GitHub task Retry without replaying obsolete input: %s", async kind => { const f = await currentGitHubTaskRetryFixture(kind.includes("newer_input")); diff --git a/server/src/__tests__/deferred-execution-receipt-work.test.ts b/server/src/__tests__/deferred-execution-receipt-work.test.ts index e902c734429..7c028e3c2e8 100644 --- a/server/src/__tests__/deferred-execution-receipt-work.test.ts +++ b/server/src/__tests__/deferred-execution-receipt-work.test.ts @@ -72,14 +72,18 @@ describe("deferred receipt durable dispatch", () => { // during actual process execution; accelerated steps only settle idle work. vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); heartbeat = heartbeatService(db); - coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !drain, canReconcile: () => true, + coordinator = createCoordinator(); + }); + function createCoordinator() { + const worker = createDeliveryWorkCoordinator({ owner: db, canRun: () => !drain, canReconcile: () => true, onError: error => { errors.push(error); errorTimes.push(Date.now()); } }); - const register = coordinator.register.bind(coordinator); - coordinator.register = (queue, task) => register(queue, { ...task, run: signal => { + const register = worker.register.bind(worker); + worker.register = (queue, task) => register(queue, { ...task, run: signal => { const pass = Promise.resolve(task.run(signal)); passes.add(pass); void pass.then(() => passes.delete(pass), () => passes.delete(pass)); return pass; } }); - }); + return worker; + } afterEach(async () => { await coordinator.stop(); await heartbeat.drainActiveRunExecutions(); for (const stop of extraStops.splice(0)) await stop(); @@ -153,6 +157,78 @@ describe("deferred receipt durable dispatch", () => { source: "on_demand", status: "completed", reason: "task_retry", payload: { issueId: f.issue.id } }); await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; await quiet(); }); + it.each([false, true])("reconsiders every saved receipt during repeated company eligibility changes (expired wait: %s)", async expiredWait => { + const f = await fixture(); + const ids = Array.from({ length: 178 }, (_, index) => `${(index + 1).toString(16).padStart(8, "0")}-0000-4000-8000-000000000000`); + await db.insert(agentWakeupRequests).values(ids.map(id => ({ id, companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution", + updatedAt: new Date(Date.now() - 60_000), + payload: { issueId: f.issue.id, _paperclipWakeContext: { wakeCommentIds: [randomUUID()] }, + ...(expiredWait ? { executionWait: { reason: "recovery" } } : {}) }, + }))); + const checked = new Set(); + await registerDeferredReceiptWork(db, coordinator, { + resume: async scope => { checked.add(scope.requestId!); }, + settleTaskChat: async () => {}, createNativeBatch: () => async () => {}, schedulingSuppressed: async () => false, + }, () => !drain).ready; + for (let pass = 0; pass < 12; pass++) { + await db.transaction(tx => notifyDeferredExecutionEligibility(tx, { companyId: f.companyId })); + await vi.advanceTimersByTimeAsync(31_000); + await Promise.allSettled([...passes]); + await immediate(); + } + expect(checked.size).toBe(ids.length); + await quiet(); + }); + it("dispatches task Retry past a repeatedly invalidated backlog into real process output", async () => { + const f = await fixture(); + // Hold executor capacity while queue time advances; reconstruct the saved + // run with fresh workers at PostgreSQL time before real process execution. + const [occupant] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, + status: "running", startedAt: new Date() }).returning(); + await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, + wakeOnDemand: true, maxConcurrentRuns: 1 } } }).where(eq(agents.id, f.agent.id)); + const [closed] = await db.insert(issues).values({ companyId: f.companyId, title: "Closed task history", + status: "done", assigneeAgentId: f.agent.id }).returning(); + await db.insert(agentWakeupRequests).values(Array.from({ length: 178 }, (_, index) => ({ + id: `${(index + 1).toString(16).padStart(8, "0")}-0000-4000-8000-000000000000`, + companyId: f.companyId, agentId: f.agent.id, source: "on_demand", status: "deferred_issue_execution", + payload: { issueId: closed.id, _paperclipWakeContext: { wakeCommentIds: [randomUUID()] } }, + }))); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + const read = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0]; + for (let pass = 0; pass < 12 && !(await read()).runId; pass++) { + await db.transaction(tx => notifyDeferredExecutionEligibility(tx, { companyId: f.companyId })); + await vi.advanceTimersByTimeAsync(31_000); + await Promise.allSettled([...passes]); + await immediate(); + } + const receipt = await read(); + expect(receipt.runId).toBeTruthy(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)))[0].status).toBe("queued"); + await coordinator.stop(); + await heartbeat.drainActiveRunExecutions(); + vi.setSystemTime(vi.getRealSystemTime()); + heartbeat = heartbeatService(db); + coordinator = createCoordinator(); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, occupant.id)); + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${retry.requestId}`)).toHaveLength(1); + await finish(f.issue.id); + await quiet(); + }); it("dispatches an ordinary committed task Retry into saved process output", async () => { const f = await fixture(); await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; diff --git a/server/src/services/heartbeat/deferred-receipt-work.ts b/server/src/services/heartbeat/deferred-receipt-work.ts index 9cca7528228..b799f10b0a7 100644 --- a/server/src/services/heartbeat/deferred-receipt-work.ts +++ b/server/src/services/heartbeat/deferred-receipt-work.ts @@ -6,7 +6,7 @@ import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { DeferredExecutionScope, DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; -type Entry = { id: string; companyId: string; dueAt: number }; +type Entry = { id: string; companyId: string; dueAt: number; immediate: boolean }; type NativeIntentOptions = DeferredExecutionScope & { onDeferredDeadline(at: number): void; onError(error: unknown): void; @@ -39,6 +39,11 @@ export function registerDeferredReceiptWork( }, decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint), matches: (entry, scope) => !scope || entry.companyId === scope.companyId && (!scope.requestId || entry.id === scope.requestId), + // A busy company can invalidate a paginated scan before the pending + // batch drains. Keep unread ready receipts ahead of freshly reloaded + // receipts already checked; actual wait/claim deadlines still replace. + mergeEntry: (pending, loaded) => pending.immediate && loaded.immediate + ? { ...loaded, dueAt: Math.min(pending.dueAt, loaded.dueAt) } : loaded, load: async scope => { const key = scope ? JSON.stringify({ companyId: scope.companyId, requestId: scope.requestId }) : "*"; if (!enabled) { cursors.delete(key); return []; } @@ -76,12 +81,15 @@ export function registerDeferredReceiptWork( )).orderBy(asc(agentWakeupRequests.id)).limit(100); const complete = rows.length < 100; if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); - return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, - dueAt: kind === "native" && row.status === "claimed" && row.claimedAt ? row.claimedAt.getTime() + 60_000 + const now = Date.now(); + return { rows: rows.map(row => { + const deadline = kind === "native" && row.status === "claimed" && row.claimedAt ? row.claimedAt.getTime() + 60_000 : kind === "native" && row.error ? row.updatedAt.getTime() + 30_000 : kind === "deferred" && row.delayed && !row.interruptedLegacy ? row.updatedAt.getTime() + 30_000 - : Date.now(), - })), complete }; + : null; + const immediate = deadline === null || deadline <= now; + return { id: row.id, companyId: row.companyId, dueAt: immediate ? now : deadline, immediate }; + }), complete }; }, check: async entry => { if (!enabled) return null; @@ -101,7 +109,7 @@ export function registerDeferredReceiptWork( : kind === "deferred" && current.status === "deferred_issue_execution" && !current.interruptedLegacy && (current.payload?.queuedCommentInterrupt || current.payload?.executionWait) ? current.updatedAt.getTime() + 30_000 : null; - if (due !== null && due > Date.now()) return { ...entry, dueAt: due }; + if (due !== null && due > Date.now()) return { ...entry, dueAt: due, immediate: false }; } const errors: unknown[] = []; let next: number | null = null; @@ -117,7 +125,7 @@ export function registerDeferredReceiptWork( catch (error) { failed(error); } } if (errors.length) throw new AggregateError(errors, `Deferred receipt ${entry.id} could not settle`); - return next === null ? null : { ...entry, dueAt: next }; + return next === null ? null : { ...entry, dueAt: next, immediate: false }; }, }, canRun); } diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts index cdd69a6801e..6ce95086ad0 100644 --- a/server/src/services/scoped-deadline-work.ts +++ b/server/src/services/scoped-deadline-work.ts @@ -18,6 +18,8 @@ export function registerScopedDeadlineWork( decodeScope(hint: string): Scope | undefined; matches(entry: Entry, scope?: Scope): boolean; load(scope?: Scope, signal?: AbortSignal): Promise; + // Owners may retain pending queue priority while refreshing eligibility. + mergeEntry?(pending: Entry, loaded: Entry): Entry; check(entry: Entry, signal?: AbortSignal): Promise; }, canRun: () => boolean, @@ -90,7 +92,8 @@ export function registerScopedDeadlineWork( const loaded = await options.load(scope, signal); const result = Array.isArray(loaded) ? { rows: loaded, complete: true } : loaded; for (const entry of result.rows ?? []) { - entries.set(entry.id, entry); + const pending = entries.get(entry.id); + entries.set(entry.id, pending && options.mergeEntry ? options.mergeEntry(pending, entry) : entry); for (const [other, progress] of scans) if (options.matches(entry, options.decodeScope(other))) progress.seen.add(entry.id); } if (!result.complete) { dirty.add(hint); continue; } From fb6554ac6208b176a4109aba103264146e56a7fe Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:17:16 -0600 Subject: [PATCH 02/16] fix(native): settle source wake with terminal projection (#18804) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each agent wake request can bind to one heartbeat run. > - Native recovery can finish and project a run after its live executor has stopped. > - The legacy completion callback settles the wake, but native terminal projection did not. > - A succeeded run could therefore leave its exact wake claimed with no finish time. > - This pull request settles that claim in each native terminal projection transaction and reuses durable projection recovery for old committed rows. > - The benefit is consistent run and wake receipts without a provider replay or a new sweeper. ## Linked Issues or Issue Description Refs #13856 and #10109. These open PRs describe orphan claimed wakes and cleanup after terminal persistence. This change fixes the native projection owner and uses its existing durable worker. It does not modify those contributor branches. **What happened?** A native run can be accepted and committed, then reach a terminal state through recovery. Its source wake stays claimed because the live completion callback does not return to settle it. **Expected behavior** Terminal projection must settle only the same company's exact source wake bound to that agent and run. Completed, failed and cancelled outcomes must retain their meaning. Other live, queued, cancelled, unbound or newly rebound wake records must remain unchanged. **Steps to reproduce** Commit an accepted native result in the real database fixture. Leave its source wake claimed. Call native finalization with run-status projection enabled, without the live completion callback. Before this change, the run becomes succeeded and finished while the exact wake remains claimed. ## What Changed - Settle the exact source claim inside native finalization, committed recovery, and exhausted-finalization projection transactions. Fence it by wake ID, company, agent, run and current claimed state. - Lock the matching claim and register settlement notifications before the write. A failed transaction rolls back both run projection and wake settlement. - Let the existing durable projection worker discover a terminal run with an unfinished exact claim. A repaired committed row leaves the pending set. No new timer, sweep or table is added. - Add real database regressions for native recovery, all three terminal outcomes, old committed receipts, idempotence, scoped denial controls, notification ordering and rollback. Cover provider/workspace finalization exhaustion and denial when the caller does not own terminal projection. - Preserve the existing provider, accounting, workspace, permissions and result contracts. No API, schema, user command, access-map entry or documentation contract changes are needed. ## Verification - The new native coordinator/finalizer regression failed before the production change: the run was succeeded and committed, but the bound wake was still claimed. - The initial focused database run did not reach the assertion because the SDK was unbuilt. A second attempt hit sandbox loopback restrictions. A third exposed skipped package library hydration. These setup failures were corrected before the meaningful red was captured. - Real PostgreSQL projection and response suites passed 89 tests before the independent review correction. The reviewer requested notification intent registration before mutation. The correction adds a notification-order and rollback case. All 90 tests now pass across both suites. - Server typecheck passed after building the isolated checkout's Runner and SDK prerequisites. Fresh corrected-source server typecheck and production server build also pass. Full repository checks run in current-head CI. The earlier serialized-9 CI failure was the independently repaired inbox snapshot fixture in #18798 (expected nine reader calls, observed eight). A normal merge of the containing master supplies that fix; the native suites plus the repaired inbox suite now pass 96 tests together at merge head `456c92ef8d60c3fcbeaf66ff8010a7ebd3d1109d`. Server typecheck passes. That head was superseded by the reviewed retry-exhaustion correction below. The subsequent review found a third terminal write in retry exhaustion. Real database provider and workspace exhaustion regressions failed with claimed/null-finished wakes before the fix. The same fenced helper now runs inside that transaction. At final source head `9718d4c2ec6066accf549dc6fea795891f6b90ec`, all 100 focused tests, server typecheck and production server build pass, including projection-disabled denial and rollback of run/coordinator/wake on notification failure. - Final current-head CI passed: 54 success, 2 skipped, and the neutral Paperclip Review at 5/5. Greptile also reports 5/5. Required ci / verify and ci / e2e passed. All review threads are resolved. - No live task write, provider work, accounting repair, deployed replay or manual rollout was performed. ## Risks - A stale wake binding must never be settled. The update requires the exact wake ID, company, agent, run and claimed state. Tests cover unbound and newer-run bindings, foreign company and agent, queued and cancelled records, and retained live ownership. - Old claimed debt now participates in the existing projection predicate. It uses the source wake primary key and stops matching after settlement. Existing quiet-worker tests remain in the focused suite. - Wake settlement must share the projection transaction. Notification failure and outer rollback tests preserve both original rows. - Interruption, timeout, retained historical terminal-failure coordinators, and delayed response presentation keep their current owners. This change does not infer or replay provider work. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation where needed; existing contracts are restored - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip --- .../native-run-projection-work.test.ts | 155 +++++++++++++++++- .../agent-chat-response-recovery.test.ts | 20 +++ .../native-runtime/native-run-finalizer.ts | 4 +- .../native-runtime/native-run-projection.ts | 36 +++- 4 files changed, 210 insertions(+), 5 deletions(-) diff --git a/server/src/__tests__/native-run-projection-work.test.ts b/server/src/__tests__/native-run-projection-work.test.ts index 26f0a9d239a..f9718354a1c 100644 --- a/server/src/__tests__/native-run-projection-work.test.ts +++ b/server/src/__tests__/native-run-projection-work.test.ts @@ -1,3 +1,4 @@ +import * as wakeSettlementNotifications from "../services/external-conversation-settlement-notifications.js"; import { heartbeatService } from "../services/heartbeat.js"; import { instanceSettingsService } from "../services/instance-settings.js"; import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "../services/heartbeat/run-lifecycle.js"; @@ -8,14 +9,14 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { setTimeout as delay } from "node:timers/promises"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, agents, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { nativeRunProjectionWork } from "../services/native-runtime/native-run-projection-work.js"; import { projectCommittedRun } from "../services/native-runtime/native-run-projection.js"; import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; -import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js"; +import { finalizeNativeRun, recordNativeFinalizationFailure } from "../services/native-runtime/native-run-finalizer.js"; import * as statusCommitter from "../services/native-runtime/status-decision-committer.js"; import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js"; import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js"; @@ -120,6 +121,154 @@ describe("committed native run projection lifecycle", () => { async function accepted(status: "succeeded" | "failed" | "cancelled" = "succeeded") { const binding = await seed(); await complete(binding, "control_plane", db, status); await barrier(binding); return binding; } + async function claimedWake(binding: Awaited>) { + const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "issue", + status: "claimed", runId: binding.runId, claimedAt: new Date(), payload: { issueId: binding.issueId } }).returning(); + await db.update(heartbeatRuns).set({ wakeupRequestId: wake.id }).where(eq(heartbeatRuns.id, binding.runId)); + return wake; + } + const readWake = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0]; + + it.each(["succeeded", "failed", "cancelled"] as const)("settles the exact wake atomically with a fresh %s projection", async status => { + const binding = await accepted(status), wake = await claimedWake(binding); + await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const run = await read(binding.runId), settled = await readWake(wake.id); + expect(run.status).toBe(status); + expect(settled).toMatchObject({ status: status === "succeeded" ? "completed" : status, runId: binding.runId, + companyId, agentId, finishedAt: run.finishedAt, error: run.error }); + expect(settled.finishedAt).not.toBeNull(); + await projectCommittedRun({ db, run, coordinator: await state(binding.runId) }); + expect(await readWake(wake.id)).toEqual(settled); + }); + + it.each(["provider", "workspace"] as const)("settles the exact wake when %s finalization retries exhaust", async scope => { + const binding = await accepted(), wake = await claimedWake(binding); + if (scope === "workspace") await db.update(workspaceOperations).set({ status: "failed" }) + .where(eq(workspaceOperations.heartbeatRunId, binding.runId)); + if (scope === "provider") await db.update(nativeRunResults).set({ + resultJson: sql`jsonb_set(${nativeRunResults.resultJson}, '{terminal,runTerminalState}', '"unknown"'::jsonb)`, + }).where(eq(nativeRunResults.runId, binding.runId)); + for (let attempt = 0; attempt < 3; attempt++) { + await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) }) + .where(eq(nativeRunFinalizations.runId, binding.runId)); + if (scope === "provider") await reconcileNativeFinalizations(db, [binding.runId]); + else await recordNativeFinalizationFailure({ db, runId: binding.runId, projectRunStatus: true, + failureScope: "workspace", error: new Error("native_workspace_finalize_failed") }); + if (attempt < 2) { + expect((await read(binding.runId)).status).toBe("running"); + expect(await readWake(wake.id)).toMatchObject({ status: "claimed", finishedAt: null }); + } + } + const run = await read(binding.runId); + expect((await state(binding.runId)).phase).toBe("terminal_failure"); + expect(run).toMatchObject({ status: scope === "provider" ? "succeeded" : "failed", nativePhase: "terminal_failure" }); + expect(run.finishedAt).not.toBeNull(); + expect(await readWake(wake.id)).toMatchObject({ status: scope === "provider" ? "completed" : "failed", + runId: binding.runId, finishedAt: run.finishedAt }); + }); + + it("keeps the source wake claimed when exhausted failure does not own terminal projection", async () => { + const binding = await accepted(), wake = await claimedWake(binding); + await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.heartbeatRunId, binding.runId)); + for (let attempt = 0; attempt < 3; attempt++) await recordNativeFinalizationFailure({ db, runId: binding.runId, + projectRunStatus: false, failureScope: "workspace", error: new Error("native_workspace_finalize_failed") }); + expect((await state(binding.runId)).phase).toBe("terminal_failure"); + expect((await read(binding.runId)).status).toBe("running"); + expect(await readWake(wake.id)).toEqual(wake); + }); + + it("rolls back exhausted projection and its claim if wake notification registration fails", async () => { + const binding = await accepted(), wake = await claimedWake(binding); + await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.heartbeatRunId, binding.runId)); + const fail = () => recordNativeFinalizationFailure({ db, runId: binding.runId, + projectRunStatus: true, failureScope: "workspace" as const, error: new Error("native_workspace_finalize_failed") }); + await fail(); await fail(); + const beforeRun = await read(binding.runId), beforeCoordinator = await state(binding.runId); + const notify = wakeSettlementNotifications.notifyExternalConversationSettlement; + vi.spyOn(wakeSettlementNotifications, "notifyExternalConversationSettlement").mockImplementation(async (tx, scope) => { + const [current] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); + expect(current.status).toBe("claimed"); + await notify(tx, scope); + throw new Error("exhaustion wake notification failure"); + }); + await expect(fail()).rejects.toThrow("exhaustion wake notification failure"); + expect(await read(binding.runId)).toEqual(beforeRun); + expect(await state(binding.runId)).toEqual(beforeCoordinator); + expect(await readWake(wake.id)).toEqual(wake); + }); + + it.each(["succeeded", "failed", "cancelled"] as const)("discovers and repairs an old %s run's claimed wake once", async status => { + const binding = await accepted(status); + await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + const run = await read(binding.runId), wake = await claimedWake(binding); + const work = nativeRunProjectionWork(db); + const [entry] = await work.load({ companyId, runId: binding.runId }); + expect(entry).toMatchObject({ id: binding.runId }); + await work.check(entry); + expect(await readWake(wake.id)).toMatchObject({ status: status === "succeeded" ? "completed" : status, + finishedAt: run.finishedAt }); + expect((await read(binding.runId)).finishedAt).toEqual(run.finishedAt); + expect(await work.load({ companyId, runId: binding.runId })).toEqual([]); + }); + + it.each(["unbound", "newer-run", "queued", "cancelled", "foreign-company", "foreign-agent", "live-owner"] as const)( + "does not settle a %s wake claim", async fault => { + const binding = await accepted(), wake = await claimedWake(binding); + await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" }); + if (fault === "unbound" || fault === "newer-run") await db.update(agentWakeupRequests) + .set({ runId: fault === "unbound" ? null : randomUUID() }).where(eq(agentWakeupRequests.id, wake.id)); + if (fault === "queued" || fault === "cancelled") await db.update(agentWakeupRequests) + .set({ status: fault }).where(eq(agentWakeupRequests.id, wake.id)); + if (fault === "foreign-company") { + const otherCompany = randomUUID(); + await db.insert(companies).values({ id: otherCompany, name: "Other wake owner", issuePrefix: `OW${randomUUID().slice(0, 6)}` }); + await db.update(agentWakeupRequests).set({ companyId: otherCompany }).where(eq(agentWakeupRequests.id, wake.id)); + } + if (fault === "foreign-agent") { + const otherAgent = randomUUID(); + await db.insert(agents).values({ id: otherAgent, companyId, name: "Other wake owner", adapterType: "paperclip_runner" }); + await db.update(agentWakeupRequests).set({ agentId: otherAgent }).where(eq(agentWakeupRequests.id, wake.id)); + } + if (fault === "live-owner") await db.update(heartbeatRuns).set({ status: "running", nativePhase: "terminal_failure", + errorCode: "native_execution_ownership_unverified" }).where(eq(heartbeatRuns.id, binding.runId)); + const before = await readWake(wake.id); + await projectCommittedRun({ db, run: await read(binding.runId), coordinator: await state(binding.runId) }); + expect(await readWake(wake.id)).toEqual(before); + if (fault === "live-owner") expect((await read(binding.runId)).status).toBe("running"); + }, + ); + + it("registers wake settlement intent before mutation and rolls back notification failure", async () => { + const binding = await accepted(), wake = await claimedWake(binding); + await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" }); + const before = await read(binding.runId); + const notify = wakeSettlementNotifications.notifyExternalConversationSettlement; + const notification = vi.spyOn(wakeSettlementNotifications, "notifyExternalConversationSettlement") + .mockImplementation(async (tx, scope) => { + const [current] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); + expect(current.status).toBe("claimed"); + await notify(tx, scope); + throw new Error("wake intent registration failure"); + }); + await expect(projectCommittedRun({ db, run: before, coordinator: await state(binding.runId) })) + .rejects.toThrow("wake intent registration failure"); + expect(notification).toHaveBeenCalledOnce(); + expect(await read(binding.runId)).toEqual(before); + expect(await readWake(wake.id)).toEqual(wake); + }); + + it("rolls back terminal projection and exact wake settlement together", async () => { + const binding = await accepted(), wake = await claimedWake(binding); + await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" }); + const before = await read(binding.runId); + await expect(db.transaction(async tx => { + await projectCommittedRun({ db: tx as unknown as Db, run: before, coordinator: await state(binding.runId) }); + throw new Error("projection rollback fixture"); + })).rejects.toThrow("projection rollback fixture"); + expect(await read(binding.runId)).toEqual(before); + expect(await readWake(wake.id)).toEqual(wake); + }); + it("reconstructs a committed decision with an unfinished run projection and stays quiet after repair", async () => { const binding = await accepted(); await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" }); diff --git a/server/src/services/native-runtime/agent-chat-response-recovery.test.ts b/server/src/services/native-runtime/agent-chat-response-recovery.test.ts index 918653a5fd1..b0d4ce5481a 100644 --- a/server/src/services/native-runtime/agent-chat-response-recovery.test.ts +++ b/server/src/services/native-runtime/agent-chat-response-recovery.test.ts @@ -65,6 +65,26 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () => const input = { companyId, issueId, runId }; return { ...input, input, agentId, userId, wakeId, prpEvent, reply }; } + it("settles the exact claimed wake when recovery projects a committed native run", async () => { + // The provider and result are already committed. Recovery owns terminal + // projection because the live executor will not return to completeRun. + const f = await fixture(false, { succeed: false }); + const sourceWakeId = randomUUID(); + await db.insert(agentWakeupRequests).values({ id: sourceWakeId, companyId: f.companyId, + agentId: f.agentId, source: "issue", reason: "issue_assigned", status: "claimed", + runId: f.runId, claimedAt: new Date(), payload: { issueId: f.issueId } }); + await db.update(heartbeatRuns).set({ wakeupRequestId: sourceWakeId }).where(eq(heartbeatRuns.id, f.runId)); + + await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, sourceWakeId)); + expect(run).toMatchObject({ status: "succeeded", nativePhase: "committed" }); + expect(run.finishedAt).not.toBeNull(); + expect(wake).toMatchObject({ status: "completed", runId: f.runId }); + expect(wake.finishedAt).not.toBeNull(); + }); + const comments = (runId: string) => db.select().from(issueComments).where(eq(issueComments.createdByRunId, runId)); const issue = async (issueId: string) => (await db.select().from(issues).where(eq(issues.id, issueId)))[0]; diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 1cbba5703cd..3a68b7d8f3b 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -3,7 +3,7 @@ import { notifyExternalAgentSettlement } from "../../modules/external-agents/ind import { notifyNativeRecoveryNotices } from "../native-recovery-notice-notifications.js"; import { resolveCommittedNativeRecoveryNotices } from "./native-recovery-notices.js"; import { notifyNativeRunProjection } from "../native-run-projection-notifications.js"; -import { projectCommittedRun, recoveredExecutionFailureMetadata } from "./native-run-projection.js"; +import { projectCommittedRun, recoveredExecutionFailureMetadata, settleNativeTerminalWake } from "./native-run-projection.js"; import { notifyNativeResultCommit } from "../native-result-commit-notifications.js"; import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; @@ -465,6 +465,7 @@ async function recordRetryableFailure(input: { }) .where(eq(heartbeatRuns.id, input.run.id)) .returning(); + if (projectsTerminalStatus && updatedRun) await settleNativeTerminalWake(tx as unknown as Db, updatedRun); if (projectsTerminalStatus) terminalRunToEmit = updatedRun ?? null; if (supersededByNewerRun) { await issueRecoveryActionService( @@ -1679,6 +1680,7 @@ export async function finalizeNativeRun(input: { await notifyExecutionRunWork(tx, { companyId: run.companyId }); await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId: coordinator.issueId }); const [after] = await projectRun(tx); + if (after && input.projectRunStatus) await settleNativeTerminalWake(tx, after); return after ? { before, after } : null; }); const updatedRun = projection?.after; diff --git a/server/src/services/native-runtime/native-run-projection.ts b/server/src/services/native-runtime/native-run-projection.ts index 2542b8bb1e3..706f80e7366 100644 --- a/server/src/services/native-runtime/native-run-projection.ts +++ b/server/src/services/native-runtime/native-run-projection.ts @@ -1,3 +1,5 @@ +import { notifyExternalConversationSettlement } from "../external-conversation-settlement-notifications.js"; +import { notifyStrandedAgentStatus } from "../live-run-recovery-notifications.js"; import { notifyNativeEnvironmentSettlement } from "../native-environment-settlement-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { notifyNativeRecoveryNotices } from "../native-recovery-notice-notifications.js"; @@ -5,7 +7,7 @@ import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-wor import { notifyNativeRunProjection } from "../native-run-projection-notifications.js"; import { randomUUID } from "node:crypto"; import { and, eq, exists, inArray, isNotNull, isNull, not, or, sql, type SQL } from "drizzle-orm"; -import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { agentWakeupRequests, heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; @@ -26,11 +28,41 @@ export function projectNativeTerminalRunStatus( +function claimedNativeWakeNeedsSettlement() { + return sql`exists (select 1 from ${agentWakeupRequests} where + ${agentWakeupRequests.id} = ${heartbeatRuns.wakeupRequestId} + and ${agentWakeupRequests.companyId} = ${heartbeatRuns.companyId} + and ${agentWakeupRequests.agentId} = ${heartbeatRuns.agentId} + and ${agentWakeupRequests.runId} = ${heartbeatRuns.id} + and ${agentWakeupRequests.status} = 'claimed')`; +} + +/** Settle only the exact persisted source claim inside terminal projection. */ +export async function settleNativeTerminalWake(tx: Db, run: Pick) { + if (run.runtimeMode !== "native" || !run.wakeupRequestId || !run.finishedAt + || !["succeeded", "failed", "cancelled"].includes(run.status)) return; + const claim = and(eq(agentWakeupRequests.id, run.wakeupRequestId), + eq(agentWakeupRequests.companyId, run.companyId), eq(agentWakeupRequests.agentId, run.agentId), + eq(agentWakeupRequests.runId, run.id), eq(agentWakeupRequests.status, "claimed")); + const [current] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests) + .where(claim).for("update").limit(1); + if (!current) return; + await notifyExternalConversationSettlement(tx, { companyId: run.companyId }); + await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId }); + await tx.update(agentWakeupRequests).set({ + status: run.status === "succeeded" ? "completed" : run.status, + finishedAt: run.finishedAt, error: run.error, updatedAt: new Date(), + }).where(claim); +} + export function nativeRunProjectionNeedsRepair(projectedStatus: "succeeded" | "failed" | "cancelled" | SQL) { return and( eq(heartbeatRuns.runtimeMode, "native"), or( inArray(heartbeatRuns.status, ["queued", "running", "failed"]), + and(eq(heartbeatRuns.status, projectedStatus), + inArray(heartbeatRuns.status, ["succeeded", "failed", "cancelled"]), claimedNativeWakeNeedsSettlement()), and( eq(heartbeatRuns.status, "succeeded"), or( @@ -51,6 +83,7 @@ export function nativeRunProjectionNeedsRepair(projectedStatus: "succeeded" | "f or( sql`${heartbeatRuns.status} is distinct from ${projectedStatus}`, + claimedNativeWakeNeedsSettlement(), isNull(heartbeatRuns.finishedAt), sql`${heartbeatRuns.nativePhase} is distinct from 'committed'`, sql`${heartbeatRuns.resultJson}->>'finalizationPhase' is distinct from 'committed'`, @@ -164,6 +197,7 @@ export async function projectCommittedRun(input: { }) .where(needsRepair) .returning(); + if (after) await settleNativeTerminalWake(tx, after); return after ? { before, after } : null; }); From d8a9b3eb0c8229d1d0f1f6af2141006dee7b4425 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:28:15 -0600 Subject: [PATCH 03/16] fix(ui): show task identifiers in activity toasts (#18802) Project the company-scoped identifier for authorized task events and use it in toast titles, fallback bodies, and links. Retain authorized delivery when the optional identifier lookup fails. Co-Authored-By: Paperclip --- doc/TOASTS.md | 4 + .../src/__tests__/live-events-toast.test.ts | 87 +++++++++++++++++-- .../src/realtime/activity-toast-identity.ts | 25 +++++- ui/src/context/LiveUpdatesProvider.test.ts | 32 ++++++- ui/src/context/LiveUpdatesProvider.tsx | 13 +-- 5 files changed, 142 insertions(+), 19 deletions(-) diff --git a/doc/TOASTS.md b/doc/TOASTS.md index f2e0489fb00..fc984f1a640 100644 --- a/doc/TOASTS.md +++ b/doc/TOASTS.md @@ -63,6 +63,10 @@ never persisted or published on the shared company event bus. Authorized events still reach cache handling and component subscribers when eligibility is false. Lookup failure is logged but does not discard the authorized event. Existing task/run privacy checks and payload redaction remain in force. +Task activity titles, fallback text, and links use the authorized task identifier. +If the optional identifier lookup fails, the event still arrives and uses `Task` +unless the browser already has the identifier. Permission-check failures do not +use this fallback. When an authenticated comment event omits the comment ID, the visible task thread refreshes through its authorized HTTP read instead of trying to hydrate the comment. Closed sockets cannot deliver stale notifications after account/company changes. diff --git a/server/src/__tests__/live-events-toast.test.ts b/server/src/__tests__/live-events-toast.test.ts index e82af433d73..c877b70f090 100644 --- a/server/src/__tests__/live-events-toast.test.ts +++ b/server/src/__tests__/live-events-toast.test.ts @@ -29,7 +29,7 @@ function database(onPredicate: (kind: "where" | "join", condition: SQL) => void innerJoin: (_table: unknown, condition: SQL) => { onPredicate("join", condition); return query; }, where: (condition: SQL) => { onPredicate("where", condition); return query; }, limit: async () => { - if (table === "issues") return [{ id: "task" }]; + if (table === "issues") return [{ id: "task", identifier: "PAP-1234" }]; if (table === "company_memberships") return fields.name ? [{ name: "Alex Example", image: "/alex.png" }] : [{ id: "membership" }]; if (table === "agents") return [{ name: "Coder", appearance: { schemaVersion: 1, characterVersion: "cap-v1", paletteId: "bubblegum-sky" } }]; return []; @@ -39,6 +39,60 @@ function database(onPredicate: (kind: "where" | "join", condition: SQL) => void } }; } describe("authenticated activity notifications", () => { + it.each(["local_trusted", "authenticated"] as const)("keeps authorized notifications when optional identifier reads fail (%s)", async deploymentMode => { + const db = database(); + const select = db.select; + const identifierRead = vi.fn(); + db.select = fields => { + if (fields.identifier) { + identifierRead(); + if (identifierRead.mock.calls.length <= 3) throw new Error("Identifier lookup unavailable"); + } + return select(fields); + }; + const server = createServer(); + setupLiveEventsWebSocketServer(server, db as never, { deploymentMode, + resolveCloudActor: async () => ({ userId: "viewer", companyIds: ["company"] }) }); + server.listen(0, "127.0.0.1"); + await once(server, "listening"); + const socket = new WebSocket(`ws://127.0.0.1:${(server.address() as { port: number }).port}/api/companies/company/events/ws`); + try { + await once(socket, "open"); + const messages: Array<{ payload: Record }> = []; + const delivered = new Promise(resolve => socket.on("message", data => { + const event = JSON.parse(data.toString()); + messages.push(event); + if (event.payload.entityId === "after-failure") resolve(); + })); + const activity = (entityId: string, actorType: string) => publishLiveEvent({ + companyId: "company", type: "activity.logged", payload: { + action: "issue.updated", entityType: "issue", entityId, actorType, + actorId: actorType === "user" ? "viewer" : actorType === "agent" ? "agent" : null, + issueIdentifier: "SPOOF-1", details: { secret: "private task details" }, + }, + }); + if (deploymentMode === "authenticated") activity("hidden", "system"); + for (const actorType of ["system", "agent", "user"]) { + activity("visible", actorType); + } + activity("after-failure", "system"); + await delivered; + expect(messages).toHaveLength(4); + expect(identifierRead).toHaveBeenCalledTimes(4); + for (const [index, actorType] of ["system", "agent", "user"].entries()) { + const event = messages[index]; + expect(event.payload).toMatchObject({ entityId: "visible", actorType, issueIdentifier: null }); + if (actorType === "agent") expect(event.payload.actorName).toBe("Coder"); + if (actorType === "user") expect(event.payload.actorName).toBe("Alex Example"); + if (deploymentMode === "authenticated") expect(event.payload).not.toHaveProperty("details"); + } + expect(messages[3].payload.issueIdentifier).toBe("PAP-1234"); + } finally { + socket.terminate(); + await new Promise(resolve => server.close(() => resolve())); + } + }); + it("preserves only an authorized handoff task identity for empty case-panel refresh", async () => { const server = createServer(); setupLiveEventsWebSocketServer(server, database() as never, { deploymentMode: "authenticated", @@ -88,6 +142,7 @@ describe("authenticated activity notifications", () => { } }); const [data] = await delivery; expect(JSON.parse(data.toString()).toastEligible).toBe(true); + expect(JSON.parse(data.toString()).payload.issueIdentifier).toBe("PAP-1234"); expect(taskToastEligible).toHaveBeenLastCalledWith(expect.anything(), expect.anything(), "local-board"); } finally { socket.terminate(); @@ -123,6 +178,8 @@ describe("authenticated activity notifications", () => { expect(b.toastEligible).toBe(false); expect(a.payload.entityId).toBe(entityId); expect(b.payload.entityId).toBe(entityId); + expect(a.payload.issueIdentifier).toBe("PAP-1234"); + expect(b.payload.issueIdentifier).toBe("PAP-1234"); expect(original).not.toHaveProperty("toastEligible"); } } finally { @@ -132,9 +189,9 @@ describe("authenticated activity notifications", () => { } }); - it("keeps actor identity after authorization and excludes hidden tasks and private details", async () => { + it.each(["authenticated", "local_trusted"] as const)("keeps task and actor identity for %s without exposing hidden tasks or copying private details", async (deploymentMode) => { const server = createServer(); - setupLiveEventsWebSocketServer(server, database() as never, { deploymentMode: "authenticated", + setupLiveEventsWebSocketServer(server, database() as never, { deploymentMode, resolveCloudActor: async () => ({ userId: "viewer", companyIds: ["company"] }) }); server.listen(0, "127.0.0.1"); await once(server, "listening"); @@ -145,19 +202,21 @@ describe("authenticated activity notifications", () => { socket.on("message", (data) => messages.push(JSON.parse(data.toString()))); const activity = (entityId: string, actorType = "user") => publishLiveEvent({ companyId: "company", type: "activity.logged", payload: { action: "issue.created", entityType: "issue", entityId, actorType, actorId: actorType === "user" ? "viewer" : "agent", + issueIdentifier: "SPOOF-1", details: { bodySnippet: "private comment", secret: "not for the socket" }, } }); - activity("hidden"); + if (deploymentMode === "authenticated") activity("hidden"); const first = once(socket, "message"); activity("visible"); await first; expect(messages).toHaveLength(1); - expect(messages[0].payload).toEqual({ action: "issue.created", entityType: "issue", entityId: "visible", actorType: "user", actorId: "viewer", actorName: "Alex Example", actorImage: "/alex.png" }); + expect(messages[0].payload).toMatchObject({ action: "issue.created", entityType: "issue", entityId: "visible", issueIdentifier: "PAP-1234", actorType: "user", actorId: "viewer", actorName: "Alex Example", actorImage: "/alex.png" }); + if (deploymentMode === "authenticated") expect(messages[0].payload).not.toHaveProperty("details"); const second = once(socket, "message"); activity("visible", "agent"); await second; - expect(messages[1].payload).toMatchObject({ actorName: "Coder", actorType: "agent", actorId: "agent", actorAppearance: { paletteId: "bubblegum-sky" } }); - expect(messages[1].payload).not.toHaveProperty("details"); + expect(messages[1].payload).toMatchObject({ issueIdentifier: "PAP-1234", actorName: "Coder", actorType: "agent", actorId: "agent", actorAppearance: { paletteId: "bubblegum-sky" } }); + if (deploymentMode === "authenticated") expect(messages[1].payload).not.toHaveProperty("details"); } finally { socket.terminate(); await new Promise((resolve) => server.close(() => resolve())); @@ -167,6 +226,20 @@ describe("authenticated activity notifications", () => { describe("activity profile query boundaries", () => { + it("reads the task identifier within its company even when there is no actor", async () => { + const predicates: { sql: string; params: unknown[] }[] = []; + const db = database((_kind, condition) => { + const { sql, params } = new PgDialect().sqlToQuery(condition); + predicates.push({ sql, params }); + }); + expect(await activityToastIdentity(db as never, "company-A", { + action: "issue.updated", entityType: "issue", entityId: "task-A", actorType: "system", + })).toMatchObject({ issueIdentifier: "PAP-1234" }); + expect(predicates).toEqual([ + { sql: '("issues"."company_id" = $1 and "issues"."id" = $2)', params: ["company-A", "task-A"] }, + ]); + }); + it("requires the exact company, human actor, and active membership before reading a person profile", async () => { const predicates: { kind: string; sql: string; params: unknown[] }[] = []; const db = database((kind, condition) => { diff --git a/server/src/realtime/activity-toast-identity.ts b/server/src/realtime/activity-toast-identity.ts index 414f46e199f..b845c0630c5 100644 --- a/server/src/realtime/activity-toast-identity.ts +++ b/server/src/realtime/activity-toast-identity.ts @@ -1,5 +1,6 @@ import { and, eq } from "drizzle-orm"; -import { agents, authUsers, companyMemberships, type Db } from "@paperclipai/db"; +import { agents, authUsers, companyMemberships, issues, type Db } from "@paperclipai/db"; +import { logger } from "../middleware/logger.js"; const TOAST_ACTIONS = new Set(["issue.created", "issue.updated", "issue.comment_added"]); @@ -11,8 +12,26 @@ export async function activityToastIdentity( ) { const actorType = typeof payload.actorType === "string" ? payload.actorType : null; const actorId = typeof payload.actorId === "string" ? payload.actorId : null; - const identity = { actorType, actorId }; - if (!actorId || !TOAST_ACTIONS.has(String(payload.action))) return identity; + const actorIdentity = { actorType, actorId }; + if (!TOAST_ACTIONS.has(String(payload.action))) return actorIdentity; + + const issueId = payload.entityType === "issue" && typeof payload.entityId === "string" + ? payload.entityId : null; + let issueIdentifier: string | null = null; + if (issueId) { + try { + const [issue] = await db + .select({ identifier: issues.identifier }) + .from(issues) + .where(and(eq(issues.companyId, companyId), eq(issues.id, issueId))) + .limit(1); + issueIdentifier = issue?.identifier ?? null; + } catch (err) { + logger.warn({ err, companyId, issueId }, "task toast identifier lookup failed"); + } + } + const identity = { ...actorIdentity, ...(issueId ? { issueIdentifier } : {}) }; + if (!actorId) return identity; if (actorType === "agent") { const [agent] = await db diff --git a/ui/src/context/LiveUpdatesProvider.test.ts b/ui/src/context/LiveUpdatesProvider.test.ts index d45d2244b75..e7d3603e003 100644 --- a/ui/src/context/LiveUpdatesProvider.test.ts +++ b/ui/src/context/LiveUpdatesProvider.test.ts @@ -2085,7 +2085,31 @@ describe("task subtree notification context", () => { describe("activity toast attribution", () => { const build = __liveUpdatesTestUtils.buildActivityToast; - const payload = { action: "issue.created", entityType: "issue", entityId: "task-1", actorType: "user", actorId: "user-1", actorName: "Alex Example", actorImage: "/alex.png" }; + const payload = { action: "issue.created", entityType: "issue", entityId: "task-1", issueIdentifier: "PAP-1234", actorType: "user", actorId: "user-1", actorName: "Alex Example", actorImage: "/alex.png" }; + it.each([ + ["issue.created", "Alex Example created PAP-1234"], + ["issue.updated", "Alex Example updated PAP-1234"], + ["issue.comment_added", "Alex Example commented on PAP-1234"], + ])("uses the server task identifier for %s with an empty cache and no activity details", (action, title) => { + const toast = build(new QueryClient(), "company", { ...payload, action }, { userId: "other-user", agentId: null }); + expect(toast?.title).toBe(title); + expect(toast?.action).toEqual({ label: "View PAP-1234", href: "/issues/PAP-1234" }); + if (action === "issue.updated") expect(toast?.body).toBe("PAP-1234"); + expect(JSON.stringify(toast)).not.toContain("Task task-1"); + }); + it("uses a generic task label when the identifier is unavailable", () => { + const toast = build(new QueryClient(), "company", { ...payload, action: "issue.updated", issueIdentifier: undefined }, { userId: "other-user", agentId: null }); + expect(toast?.title).toBe("Alex Example updated Task"); + expect(toast?.body).toBe("Task"); + expect(toast?.action).toEqual({ label: "View Task", href: "/issues/task-1" }); + }); + it("resolves cached task titles by the server identifier", () => { + const queryClient = new QueryClient(); + queryClient.setQueryData(queryKeys.issues.detail("PAP-1234"), { id: "task-1", identifier: "PAP-1234", title: "Fix task toasts" }); + const toast = build(queryClient, "company", { ...payload, action: "issue.updated" }, { userId: "other-user", agentId: null }); + expect(toast?.title).toBe("Alex Example updated PAP-1234"); + expect(toast?.body).toBe("Fix task toasts"); + }); it.each(["issue.created", "issue.updated", "issue.comment_added"])("hides the viewer's own %s", (action) => { expect(build(new QueryClient(), "company", { ...payload, action }, { userId: "user-1", agentId: null })).toBeNull(); }); @@ -2094,18 +2118,18 @@ describe("activity toast attribution", () => { }); it("uses the real name and photo with an empty directory cache", () => { const toast = build(new QueryClient(), "company", payload, { userId: "other-user", agentId: null }); - expect(toast?.title).toBe("Alex Example created Task task-1"); + expect(toast?.title).toBe("Alex Example created PAP-1234"); expect(toast?.actor).toMatchObject({ type: "user", name: "Alex Example", image: "/alex.png" }); }); it("uses the agent's custom avatar with an empty agent cache", () => { const appearance = { schemaVersion: 1, characterVersion: "cap-v1", paletteId: "bubblegum-sky", customAvatarAssetId: "11111111-1111-4111-8111-111111111111" }; const toast = build(new QueryClient(), "company", { ...payload, actorType: "agent", actorId: "agent-1", actorName: "Coder", actorAppearance: appearance }, { userId: "user-1", agentId: null }); - expect(toast?.title).toBe("Coder created Task task-1"); + expect(toast?.title).toBe("Coder created PAP-1234"); expect(toast?.actor?.appearance).toEqual(appearance); }); it("keeps system activity distinct from people", () => { const toast = build(new QueryClient(), "company", { ...payload, actorType: "system", actorName: undefined }, { userId: "user-1", agentId: null }); - expect(toast?.title).toBe("System created Task task-1"); + expect(toast?.title).toBe("System created PAP-1234"); expect(toast?.actor).toBeUndefined(); }); }); diff --git a/ui/src/context/LiveUpdatesProvider.tsx b/ui/src/context/LiveUpdatesProvider.tsx index e66fc151b08..d099b7e6d6f 100644 --- a/ui/src/context/LiveUpdatesProvider.tsx +++ b/ui/src/context/LiveUpdatesProvider.tsx @@ -375,11 +375,11 @@ function resolveIssueToastContext( issueRefs.some((ref) => issue.id === ref || issue.identifier === ref), ); const cachedIssue = detailIssue ?? listIssue ?? null; - const ref = + const identifier = readString(details?.identifier) ?? readString(details?.issueIdentifier) ?? - cachedIssue?.identifier ?? - `Task ${shortId(issueId)}`; + cachedIssue?.identifier; + const ref = identifier ?? "Task"; const title = readString(details?.title) ?? readString(details?.issueTitle) ?? @@ -389,7 +389,7 @@ function resolveIssueToastContext( ref, title, label: title ? `${ref} - ${truncate(title, 72)}` : ref, - href: `/issues/${cachedIssue?.identifier ?? issueId}`, + href: `/issues/${identifier ?? issueId}`, }; } @@ -1064,7 +1064,10 @@ function buildActivityToast( queryClient, companyId, entityId, - details, + { + ...details, + identifier: readString(payload.issueIdentifier) ?? readString(details?.identifier), + }, ); const actor = readString(readString(payload.actorName)?.trim()) ?? resolveActorLabel(queryClient, companyId, actorType, actorId); const cachedAgent = actorType === "agent" && actorId From fef86e32e98d95b412b472542160e34ff2f5a0df Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Tue, 3 Nov 2026 11:35:28 -0800 Subject: [PATCH 04/16] feat(execution-admission): add project concurrency policy and admission storage (#18746) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects group tasks. Agents run tasks as heartbeat runs. > - Today, workspace mode and shared-workspace flags decide whether tasks run at the same time. A project cannot set its own limit on concurrent task executions. > - A correct project limit must count every project that a task uses, including @-mentioned projects. It must also keep the count when a project is archived or deleted, and when a run is admitted more than one time. > - This change cannot go into one pull request with claims and settlement. Thus this pull request adds only the policy semantics and the durable accounting storage that later claims will use. > - This pull request adds a new `execution-admission` module, three tables and the deletion guards. No route writes a cap and no claim path enforces it. > - The benefit is a tested storage contract (constraints, idempotency, rollback, indexed counts, lifecycle preservation) before any behavior changes. ## Linked Issues or Issue Description Refs #2831, Refs #14333 (related agent-level and adapter-level limits; not duplicates). **Subsystem affected** Server execution admission (new module `server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** A project cannot limit how many of its task executions are active. The current shared-workspace flags mix resource safety with capacity. A future project limit needs storage that records each admission attempt and every project that the attempt charges. Run IDs cannot be the identity, because a run can be released before dispatch and admitted again. **Proposed solution** Add a separate policy table that keeps "never configured" apart from "explicit no limit". Add admission epochs with one charge row per distinct project. Settlement deletes the charge set in one transaction. Later pull requests integrate claims, settlement, waits and the public control. **Alternatives considered** Columns on `projects` would leak an unenforced cap through every project reader. A project column or a `uuid[]` on `heartbeat_runs` cannot enforce per-project uniqueness or company scope. `budget_reservations` has no project set. A counter would be a second source of truth. ## What Changed - Added `project_task_execution_policies`. No row means never configured. A null maximum means an explicit "no limit". A check accepts only positive maximums. A composite foreign key keeps the project and company together. The row is deleted with its project. - Added `execution_admissions`. Each row is one admission attempt of a run: `(run_id, attempt)` is unique, and a partial unique index allows one unsettled attempt per run. It keeps the context revision, the nullable primary project (provenance, no foreign key) and the settlement identity. A check sets `settlement_id` and `settled_at` together. - Added `execution_admission_project_charges`. It has one row per distinct admitted project, unlimited projects included. Composite foreign keys keep the admission and the project in one company. The project foreign key has no cascade, so a charge cannot disappear with its project. `(company_id, project_id)` is indexed for active counts. - Added migration `0426_absent_guardian.sql` (renumbered from `0423`, then `0425`, as master added `0423`–`0425`). The generated SQL was reordered so the unique indexes exist before the composite foreign keys that need them. Snapshots `0419` and `0420` were pruned to keep five snapshots. - Added the `execution-admission` module (domain → application → adapters): - Domain: policy patch parsing and application (omitted keeps, null clears, partial updates keep other fields, malformed values are rejected), canonical project-set preparation (primary ∪ references, deduplicated, ascending, bounded with an explicit `too_many_projects` result), and the settlement replay decision. - Application: internal `readTaskExecutionPolicy` and `updateTaskExecutionPolicy` commands. No route, tool, CLI or import calls them. - Adapter: the project-set lock (`FOR NO KEY UPDATE`, ascending, one statement), the policy store, and the storage transitions `recordAdmission`, `settleAdmission` and `countActiveProjectCharges`. These transitions are not exported from the module entry point. Claims (later) must add the capacity check and the lifecycle evidence. - Project deletion returns `409 project_execution_admission_pending` while the project has a charge. Agent deletion returns `409 agent_execution_admission_pending` while its runs have an unsettled admission, and removes settled history. Company deletion calls a new module participant before it deletes runs and projects. - Documented the tables, the lifecycle rules and the proposed global lock order in `doc/DATABASE.md`. Added module ownership to `server/src/modules/README.md`. ## Verification - `cd server && npx vitest run src/modules/execution-admission` → 2 files, 26 tests pass (the Greptile finding on an empty primary ID added an assertion to an existing test). The adapter tests use embedded PostgreSQL and cover: - policy semantics, the maximum check and the company foreign key; - charges for every distinct project, a primaryless set and an empty set (no rows); - replay of one admission ID, conflict on a changed record, refusal of a second live attempt and the partial unique index; - settlement of the whole set once, replay, a different settlement, then attempt 2 for the same run; - company isolation of projects, runs and admission IDs, the duplicate-charge key, and rollback of an aborted transaction; - active counts per project, and an `EXPLAIN` that uses `execution_admission_project_charges_project_idx`; - archive keeps charges, `projectService.remove` returns 409, a direct project delete fails on the foreign key, and delete succeeds after settlement; - `deleteAgentDependencies` returns 409 while unsettled, and `deleteCompany` removes only that company's admissions. - `cd packages/db && npx vitest run src/execution-admission-migration.test.ts` → pass. It removes the new tables and receipt, seeds a legacy project and run, applies pending migrations, and compares the existing rows. - `pnpm --filter @paperclipai/db check:migrations` → pass. Five snapshots remain (`0421`, `0422`, `0424`, `0425`, `0426`; `0423` is SQL-only on master). - `node scripts/check-module-boundaries.mjs` → pass. - `packages/db` `tsc --noEmit` → pass. A scoped `tsc` of the module entry points and domain tests → pass. - Existing tests for the changed deletion services → 6 files, 161 tests pass (`resource-lifecycle-events`, `prompt-injection-prevention`, `agent-lifecycle/lifecycle`, `case-deletion`, `project-workspace-root-provenance`, `companies-service`). - Not run locally: the full `server` typecheck. `tsc` and `tsgo` were stopped by the out-of-memory killer on the 7 GB machine (exit 137, also single-threaded). The full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not run locally; CI ran them on head `520b69d553`: all gates pass, including Typecheck, Build and the 12 server test shards. After merging master (`1c9ca5829f`) into head `c088729361`, the migration check, migration upgrade test, module tests, module-boundary check and `packages/db` typecheck were rerun locally and pass; CI is rerunning the full suite. ## Risks - Migration risk is low. It creates three empty tables and indexes. It does not change existing tables or rows. - Behavior changes only when admission rows exist. Nothing writes them yet, so project, agent and company deletion behave as before. - Later claim work must take the project-set lock in the documented order. A claim that takes project locks after issue locks could deadlock with project deletion. - Settlement deletes charge rows, so a settled admission keeps its primary project and context revision but not its full project set. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 Co-authored-by: Paperclip --- doc/DATABASE.md | 37 + .../src/execution-admission-migration.test.ts | 53 ++ .../src/migrations/0426_absent_guardian.sql | 39 ++ ...{0420_snapshot.json => 0426_snapshot.json} | 640 +++++++++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + .../db/src/schema/execution_admissions.ts | 59 ++ packages/db/src/schema/index.ts | 2 + server/src/modules/README.md | 6 + .../adapters/company-deletion.ts | 11 + .../adapters/postgres.test.ts | 239 +++++++ .../execution-admission/adapters/postgres.ts | 165 +++++ .../execution-admission/application/policy.ts | 50 ++ .../execution-admission/company-deletion.ts | 1 + .../execution-admission/domain/admission.ts | 53 ++ .../execution-admission/domain/policy.test.ts | 88 +++ .../execution-admission/domain/policy.ts | 47 ++ .../src/modules/execution-admission/index.ts | 17 + server/src/services/agent-deletion.ts | 6 + server/src/services/company-deletion.ts | 2 + server/src/services/projects.ts | 6 + 20 files changed, 1526 insertions(+), 2 deletions(-) create mode 100644 packages/db/src/execution-admission-migration.test.ts create mode 100644 packages/db/src/migrations/0426_absent_guardian.sql rename packages/db/src/migrations/meta/{0420_snapshot.json => 0426_snapshot.json} (99%) create mode 100644 packages/db/src/schema/execution_admissions.ts create mode 100644 server/src/modules/execution-admission/adapters/company-deletion.ts create mode 100644 server/src/modules/execution-admission/adapters/postgres.test.ts create mode 100644 server/src/modules/execution-admission/adapters/postgres.ts create mode 100644 server/src/modules/execution-admission/application/policy.ts create mode 100644 server/src/modules/execution-admission/company-deletion.ts create mode 100644 server/src/modules/execution-admission/domain/admission.ts create mode 100644 server/src/modules/execution-admission/domain/policy.test.ts create mode 100644 server/src/modules/execution-admission/domain/policy.ts create mode 100644 server/src/modules/execution-admission/index.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index f69fb4eca8a..3f1c105a130 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -746,6 +746,43 @@ blocks concurrent task or run updates. It allows audit inserts to retain their foreign-key `KEY SHARE` locks without waiting on identity acquisition. The audit foreign keys and their deletion behavior remain enforced. +## Execution admission accounting + +The execution-admission module (`server/src/modules/execution-admission/`) owns +three tables. Nothing writes admissions or caps in production yet, and no claim +path enforces a project maximum. + +- `project_task_execution_policies` stores a project's maximum concurrent tasks + apart from `projects`, so project readers never return an unenforced cap. No + row means never configured. A row with a null maximum means an explicit "no + limit", which a later legacy migration must not overwrite. Maximums are + positive. The row is deleted with its project. +- `execution_admissions` stores one admission attempt of a run. A run can be + released before dispatch and admitted again, so `(run_id, attempt)` is unique + and a partial unique index allows one unsettled attempt per run. The row keeps + the context revision, the nullable primary project as provenance (no foreign + key) and the settlement identity, which is set together with `settled_at`. +- `execution_admission_project_charges` stores one row per distinct admitted + project, including unlimited projects. An admission without projects has no + rows. Composite `(company_id, …)` foreign keys keep admissions and projects in + one company. Rows exist only while the admission is unsettled: settlement + deletes the whole set in one transaction, and `(company_id, project_id)` is + indexed for active counts, so lookups scale with active work, not history. + +Archiving a project changes no charge. Deleting a project with a charge returns +`409 project_execution_admission_pending`; the foreign key is the backstop. +Deleting an agent refuses unsettled admissions on its runs and removes settled +ones. Company deletion removes the company's admissions before its runs and +projects. + +The proposed global lock order for the admission transaction is: company row +(`FOR NO KEY UPDATE`), native finalization recovery owner, the whole project set +in ascending ID order (`FOR NO KEY UPDATE`, one statement), issues, wake +requests, the run, the external-admission advisory lock, the admission row and +its charges, then budget reservations. Policy updates take the same project +lock. Settlement takes no project lock. Only the project-set step is implemented +so far; claim paths keep their current locks until they are integrated. + ## Switching between modes The database mode is controlled by `DATABASE_URL`: diff --git a/packages/db/src/execution-admission-migration.test.ts b/packages/db/src/execution-admission-migration.test.ts new file mode 100644 index 00000000000..6fefa53f9bd --- /dev/null +++ b/packages/db/src/execution-admission-migration.test.ts @@ -0,0 +1,53 @@ +import { createHash, randomUUID } from "node:crypto"; +import { readFile } from "node:fs/promises"; +import postgres from "postgres"; +import { describe, expect, it } from "vitest"; +import { applyPendingMigrations, inspectMigrations } from "./client.js"; +import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; + +const migrationName = "0426_absent_guardian.sql"; +const migrationHash = createHash("sha256") + .update(await readFile(new URL(`./migrations/${migrationName}`, import.meta.url), "utf8")).digest("hex"); +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; + +describePostgres("execution admission migration", () => { + it("adds empty admission storage without changing existing projects or runs", async () => { + const database = await startEmbeddedPostgresTestDatabase("paperclip-execution-admission-migration-"); + const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} }); + try { + await sql`DROP TABLE execution_admission_project_charges, execution_admissions, project_task_execution_policies`; + await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash = ${migrationHash}`; + const companyId = randomUUID(), agentId = randomUUID(), projectId = randomUUID(); + await sql`INSERT INTO companies (id, name, issue_prefix) VALUES (${companyId}, 'Upgrade', 'UPG')`; + await sql`INSERT INTO agents (id, company_id, name, adapter_type) VALUES (${agentId}, ${companyId}, 'Agent', 'process')`; + await sql`INSERT INTO projects (id, company_id, name, execution_workspace_policy, archived_at) VALUES + (${projectId}, ${companyId}, 'Legacy', '{"enabled":true,"sharedWorkspaceConcurrency":"serialize"}', now())`; + await sql`INSERT INTO heartbeat_runs (company_id, agent_id, status, context_snapshot) VALUES + (${companyId}, ${agentId}, 'running', ${sql.json({ projectId })})`; + const contents = async () => ({ + projects: await sql`SELECT * FROM projects ORDER BY id`, + runs: await sql`SELECT * FROM heartbeat_runs ORDER BY id`, + }); + const before = await contents(); + expect(await inspectMigrations(database.connectionString)) + .toMatchObject({ status: "needsMigrations", pendingMigrations: [migrationName] }); + + await applyPendingMigrations(database.connectionString); + + expect(await contents()).toEqual(before); + for (const table of ["project_task_execution_policies", "execution_admissions", "execution_admission_project_charges"]) { + expect(await sql.unsafe(`SELECT count(*)::int AS count FROM "${table}"`)).toEqual([{ count: 0 }]); + } + expect((await sql`SELECT conname FROM pg_constraint WHERE conrelid = 'execution_admission_project_charges'::regclass + AND contype = 'f' ORDER BY conname`).map(row => row.conname)).toEqual([ + "execution_admission_project_charges_company_admission_fk", + "execution_admission_project_charges_company_project_fk", + ]); + expect(await inspectMigrations(database.connectionString)).toMatchObject({ status: "upToDate" }); + } finally { + await sql.end(); + await database.cleanup(); + } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); +}); diff --git a/packages/db/src/migrations/0426_absent_guardian.sql b/packages/db/src/migrations/0426_absent_guardian.sql new file mode 100644 index 00000000000..2bee4c7874f --- /dev/null +++ b/packages/db/src/migrations/0426_absent_guardian.sql @@ -0,0 +1,39 @@ +CREATE TABLE "execution_admission_project_charges" ( + "admission_id" uuid NOT NULL, + "company_id" uuid NOT NULL, + "project_id" uuid NOT NULL, + CONSTRAINT "execution_admission_project_charges_admission_id_project_id_pk" PRIMARY KEY("admission_id","project_id") +); +--> statement-breakpoint +CREATE TABLE "execution_admissions" ( + "id" uuid PRIMARY KEY NOT NULL, + "company_id" uuid NOT NULL, + "run_id" uuid NOT NULL, + "attempt" integer NOT NULL, + "primary_project_id" uuid, + "context_revision" text NOT NULL, + "settlement_id" uuid, + "settled_at" timestamp with time zone, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "execution_admissions_attempt_check" CHECK ("execution_admissions"."attempt" > 0), + CONSTRAINT "execution_admissions_settlement_check" CHECK (("execution_admissions"."settlement_id" is null) = ("execution_admissions"."settled_at" is null)) +); +--> statement-breakpoint +CREATE TABLE "project_task_execution_policies" ( + "project_id" uuid PRIMARY KEY NOT NULL, + "company_id" uuid NOT NULL, + "max_concurrent_tasks" integer, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "project_task_execution_policies_maximum_check" CHECK ("project_task_execution_policies"."max_concurrent_tasks" is null or "project_task_execution_policies"."max_concurrent_tasks" > 0) +); +--> statement-breakpoint +CREATE INDEX "execution_admission_project_charges_project_idx" ON "execution_admission_project_charges" USING btree ("company_id","project_id");--> statement-breakpoint +CREATE UNIQUE INDEX "execution_admissions_company_id_uq" ON "execution_admissions" USING btree ("company_id","id");--> statement-breakpoint +CREATE UNIQUE INDEX "execution_admissions_run_attempt_uq" ON "execution_admissions" USING btree ("run_id","attempt");--> statement-breakpoint +CREATE UNIQUE INDEX "execution_admissions_live_run_uq" ON "execution_admissions" USING btree ("run_id") WHERE "execution_admissions"."settled_at" is null;--> statement-breakpoint +ALTER TABLE "execution_admission_project_charges" ADD CONSTRAINT "execution_admission_project_charges_company_admission_fk" FOREIGN KEY ("company_id","admission_id") REFERENCES "public"."execution_admissions"("company_id","id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "execution_admission_project_charges" ADD CONSTRAINT "execution_admission_project_charges_company_project_fk" FOREIGN KEY ("company_id","project_id") REFERENCES "public"."projects"("company_id","id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "execution_admissions" ADD CONSTRAINT "execution_admissions_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "execution_admissions" ADD CONSTRAINT "execution_admissions_run_id_heartbeat_runs_id_fk" FOREIGN KEY ("run_id") REFERENCES "public"."heartbeat_runs"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "project_task_execution_policies" ADD CONSTRAINT "project_task_execution_policies_company_project_fk" FOREIGN KEY ("company_id","project_id") REFERENCES "public"."projects"("company_id","id") ON DELETE cascade ON UPDATE no action; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0420_snapshot.json b/packages/db/src/migrations/meta/0426_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0420_snapshot.json rename to packages/db/src/migrations/meta/0426_snapshot.json index 1f538a50635..0f939fcb071 100644 --- a/packages/db/src/migrations/meta/0420_snapshot.json +++ b/packages/db/src/migrations/meta/0426_snapshot.json @@ -1,6 +1,6 @@ { - "id": "8f75a1bb-32d9-4aed-9f93-134a6ae71f79", - "prevId": "52cf22f3-3bd9-4b0f-8d23-b422bb0d2540", + "id": "a1fcb4c6-40e7-47a6-b3f2-c43cd9bf332b", + "prevId": "8d9f4abf-57c7-443a-aa8b-3e3fd4f09dae", "version": "7", "dialect": "postgresql", "tables": { @@ -2233,6 +2233,74 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.agent_file_roots": { + "name": "agent_file_roots", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "generation": { + "name": "generation", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "entries": { + "name": "entries", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "agent_file_roots_company_id_agent_id_agents_company_id_id_fk": { + "name": "agent_file_roots_company_id_agent_id_agents_company_id_id_fk", + "tableFrom": "agent_file_roots", + "tableTo": "agents", + "columnsFrom": [ + "company_id", + "agent_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_file_roots_company_id_agent_id_pk": { + "name": "agent_file_roots_company_id_agent_id_pk", + "columns": [ + "company_id", + "agent_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.agent_identity_keys": { "name": "agent_identity_keys", "schema": "", @@ -28757,6 +28825,327 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.execution_admission_project_charges": { + "name": "execution_admission_project_charges", + "schema": "", + "columns": { + "admission_id": { + "name": "admission_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "execution_admission_project_charges_project_idx": { + "name": "execution_admission_project_charges_project_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "execution_admission_project_charges_company_admission_fk": { + "name": "execution_admission_project_charges_company_admission_fk", + "tableFrom": "execution_admission_project_charges", + "tableTo": "execution_admissions", + "columnsFrom": [ + "company_id", + "admission_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "execution_admission_project_charges_company_project_fk": { + "name": "execution_admission_project_charges_company_project_fk", + "tableFrom": "execution_admission_project_charges", + "tableTo": "projects", + "columnsFrom": [ + "company_id", + "project_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "execution_admission_project_charges_admission_id_project_id_pk": { + "name": "execution_admission_project_charges_admission_id_project_id_pk", + "columns": [ + "admission_id", + "project_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.execution_admissions": { + "name": "execution_admissions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "attempt": { + "name": "attempt", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "primary_project_id": { + "name": "primary_project_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "context_revision": { + "name": "context_revision", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "settlement_id": { + "name": "settlement_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "settled_at": { + "name": "settled_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "execution_admissions_company_id_uq": { + "name": "execution_admissions_company_id_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_admissions_run_attempt_uq": { + "name": "execution_admissions_run_attempt_uq", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "attempt", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_admissions_live_run_uq": { + "name": "execution_admissions_live_run_uq", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"execution_admissions\".\"settled_at\" is null", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "execution_admissions_company_id_companies_id_fk": { + "name": "execution_admissions_company_id_companies_id_fk", + "tableFrom": "execution_admissions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "execution_admissions_run_id_heartbeat_runs_id_fk": { + "name": "execution_admissions_run_id_heartbeat_runs_id_fk", + "tableFrom": "execution_admissions", + "tableTo": "heartbeat_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "execution_admissions_attempt_check": { + "name": "execution_admissions_attempt_check", + "value": "\"execution_admissions\".\"attempt\" > 0" + }, + "execution_admissions_settlement_check": { + "name": "execution_admissions_settlement_check", + "value": "(\"execution_admissions\".\"settlement_id\" is null) = (\"execution_admissions\".\"settled_at\" is null)" + } + }, + "isRLSEnabled": false + }, + "public.project_task_execution_policies": { + "name": "project_task_execution_policies", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "max_concurrent_tasks": { + "name": "max_concurrent_tasks", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "project_task_execution_policies_company_project_fk": { + "name": "project_task_execution_policies_company_project_fk", + "tableFrom": "project_task_execution_policies", + "tableTo": "projects", + "columnsFrom": [ + "company_id", + "project_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": { + "project_task_execution_policies_maximum_check": { + "name": "project_task_execution_policies_maximum_check", + "value": "\"project_task_execution_policies\".\"max_concurrent_tasks\" is null or \"project_task_execution_policies\".\"max_concurrent_tasks\" > 0" + } + }, + "isRLSEnabled": false + }, "public.execution_workspace_repositories": { "name": "execution_workspace_repositories", "schema": "", @@ -29705,6 +30094,241 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.execution_workspace_snapshots": { + "name": "execution_workspace_snapshots", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "execution_workspace_id": { + "name": "execution_workspace_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "source_operation_id": { + "name": "source_operation_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "source_run_id": { + "name": "source_run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "parent_snapshot_id": { + "name": "parent_snapshot_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "epoch": { + "name": "epoch", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "format": { + "name": "format", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'paperclip.remote-workspace-archive.v1'" + }, + "object_key": { + "name": "object_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "sha256": { + "name": "sha256", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "byte_size": { + "name": "byte_size", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "key": { + "name": "key", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "manifest_digest": { + "name": "manifest_digest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "delete_after": { + "name": "delete_after", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "execution_workspace_snapshots_operation_uniq": { + "name": "execution_workspace_snapshots_operation_uniq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "source_operation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_generation_uniq": { + "name": "execution_workspace_snapshots_generation_uniq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "generation", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_workspace_idx": { + "name": "execution_workspace_snapshots_workspace_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_cleanup_idx": { + "name": "execution_workspace_snapshots_cleanup_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delete_after", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.execution_workspaces": { "name": "execution_workspaces", "schema": "", @@ -50177,6 +50801,12 @@ "primaryKey": false, "notNull": false }, + "created_by_run_id": { + "name": "created_by_run_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "dispatch_fingerprint": { "name": "dispatch_fingerprint", "type": "text", @@ -62139,6 +62769,12 @@ "primaryKey": false, "notNull": false }, + "cleanup_verified_at": { + "name": "cleanup_verified_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, "created_at": { "name": "created_at", "type": "timestamp with time zone", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index b42ac91d174..b8a5f20dd9a 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2969,6 +2969,13 @@ "when": 1793724587619, "tag": "0425_awesome_living_lightning", "breakpoints": true + }, + { + "idx": 426, + "version": "7", + "when": 1793733446322, + "tag": "0426_absent_guardian", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/execution_admissions.ts b/packages/db/src/schema/execution_admissions.ts new file mode 100644 index 00000000000..2a9e9fb641f --- /dev/null +++ b/packages/db/src/schema/execution_admissions.ts @@ -0,0 +1,59 @@ +import { sql } from "drizzle-orm"; +import { check, foreignKey, index, integer, pgTable, primaryKey, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core"; +import { companies } from "./companies.js"; +import { heartbeatRuns } from "./heartbeat_runs.js"; +import { projects } from "./projects.js"; + +/** + * Project task-execution policy, kept apart from `projects` so unenforced caps + * never reach project readers. No row: never configured. Null maximum: explicit + * unlimited. Owned by the execution-admission module. + */ +export const projectTaskExecutionPolicies = pgTable("project_task_execution_policies", { + projectId: uuid("project_id").primaryKey(), + companyId: uuid("company_id").notNull(), + maxConcurrentTasks: integer("max_concurrent_tasks"), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), +}, t => ({ + project: foreignKey({ columns: [t.companyId, t.projectId], foreignColumns: [projects.companyId, projects.id], + name: "project_task_execution_policies_company_project_fk" }).onDelete("cascade"), + maximum: check("project_task_execution_policies_maximum_check", sql`${t.maxConcurrentTasks} is null or ${t.maxConcurrentTasks} > 0`), +})); + +/** + * One admission attempt of a run. A run can be admitted again after a settled + * attempt, so the run ID is not the admission identity. Primary project is + * provenance only; the charges below are the admitted project set. + */ +export const executionAdmissions = pgTable("execution_admissions", { + id: uuid("id").primaryKey(), + companyId: uuid("company_id").notNull().references(() => companies.id), + runId: uuid("run_id").notNull().references(() => heartbeatRuns.id), + attempt: integer("attempt").notNull(), + primaryProjectId: uuid("primary_project_id"), + contextRevision: text("context_revision").notNull(), + settlementId: uuid("settlement_id"), + settledAt: timestamp("settled_at", { withTimezone: true }), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), +}, t => ({ + companyIdentity: uniqueIndex("execution_admissions_company_id_uq").on(t.companyId, t.id), + runAttempt: uniqueIndex("execution_admissions_run_attempt_uq").on(t.runId, t.attempt), + liveRun: uniqueIndex("execution_admissions_live_run_uq").on(t.runId).where(sql`${t.settledAt} is null`), + attemptCheck: check("execution_admissions_attempt_check", sql`${t.attempt} > 0`), + settlementCheck: check("execution_admissions_settlement_check", sql`(${t.settlementId} is null) = (${t.settledAt} is null)`), +})); + +/** One active charge per distinct admitted project. Settlement deletes the whole set; a charged project cannot be deleted. */ +export const executionAdmissionProjectCharges = pgTable("execution_admission_project_charges", { + admissionId: uuid("admission_id").notNull(), + companyId: uuid("company_id").notNull(), + projectId: uuid("project_id").notNull(), +}, t => ({ + identity: primaryKey({ columns: [t.admissionId, t.projectId] }), + admission: foreignKey({ columns: [t.companyId, t.admissionId], foreignColumns: [executionAdmissions.companyId, executionAdmissions.id], + name: "execution_admission_project_charges_company_admission_fk" }).onDelete("cascade"), + project: foreignKey({ columns: [t.companyId, t.projectId], foreignColumns: [projects.companyId, projects.id], + name: "execution_admission_project_charges_company_project_fk" }), + activeProject: index("execution_admission_project_charges_project_idx").on(t.companyId, t.projectId), +})); diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 1af7fbcfdf2..f12096d60cb 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -274,3 +274,5 @@ export { workflows } from "./workflows.js"; export { userWorkspacePreferences } from "./user_workspace_preferences.js"; export { issueThreadInteractionIdempotencyKeys } from "./issue_thread_interaction_idempotency_keys.js"; + +export { projectTaskExecutionPolicies, executionAdmissions, executionAdmissionProjectCharges } from "./execution_admissions.js"; diff --git a/server/src/modules/README.md b/server/src/modules/README.md index 6c3bb456c49..1da9c9b7c67 100644 --- a/server/src/modules/README.md +++ b/server/src/modules/README.md @@ -56,3 +56,9 @@ Ordinary configuration updates use the root lifecycle command. The [agent lifecycle module](../../../doc/AGENT-LIFECYCLE.md) owns agent creation, pause, resume, and termination. Use its commands for these changes. + +The execution admission module owns project task-execution policy and the +durable admission records that charge project capacity. Today it exposes only +internal policy commands and the deletion guards that project, agent, and +company deletion call. Execution claims still belong to `services/heartbeat/queue.ts` +and native review dispatch. See [execution admission accounting](../../../doc/DATABASE.md#execution-admission-accounting). diff --git a/server/src/modules/execution-admission/adapters/company-deletion.ts b/server/src/modules/execution-admission/adapters/company-deletion.ts new file mode 100644 index 00000000000..5e421170738 --- /dev/null +++ b/server/src/modules/execution-admission/adapters/company-deletion.ts @@ -0,0 +1,11 @@ +import { executionAdmissions } from "@paperclipai/db"; +import { eq } from "drizzle-orm"; +import type { CompanyDeletionParticipant } from "../../../lib/company-deletion.js"; + +/** Removes every admission, settled or not, before the company's runs and projects are deleted. */ +export const executionAdmissionCompanyDeletion: CompanyDeletionParticipant = { + async deleteCompanyData(tx, companyId) { + if (!("nestedIndex" in tx)) throw new Error("Company deletion requires a database transaction"); + await tx.delete(executionAdmissions).where(eq(executionAdmissions.companyId, companyId)); + }, +}; diff --git a/server/src/modules/execution-admission/adapters/postgres.test.ts b/server/src/modules/execution-admission/adapters/postgres.test.ts new file mode 100644 index 00000000000..ecdfef3cf07 --- /dev/null +++ b/server/src/modules/execution-admission/adapters/postgres.test.ts @@ -0,0 +1,239 @@ +import { randomUUID } from "node:crypto"; +import { + agents, + companies, + createDb, + executionAdmissionProjectCharges as charges, + executionAdmissions as admissions, + heartbeatRuns, + projectTaskExecutionPolicies as policies, + projects, +} from "@paperclipai/db"; +import { asc, eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js"; +import { withAccountingTransaction } from "../../../services/accounting-transaction.js"; +import { deleteAgentDependencies } from "../../../services/agent-deletion.js"; +import { deleteCompany } from "../../../services/company-deletion.js"; +import { projectService } from "../../../services/projects.js"; +import { prepareAdmissionRecord, type AdmissionRecord } from "../domain/admission.js"; +import { createExecutionAdmission } from "../index.js"; +import { countActiveProjectCharges, recordAdmission, settleAdmission } from "./postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +type TestDb = ReturnType; + +function record(primaryProjectId: string | null, referencedProjectIds: string[], contextRevision = "issue:1") { + const result = prepareAdmissionRecord({ primaryProjectId, referencedProjectIds, contextRevision }); + if (result.kind !== "ok") throw new Error(result.kind); + return result.record; +} + +describe.skipIf(!support.supported)("execution admission storage", () => { + let database: Awaited>; + let db: TestDb; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("execution-admission-"); + db = createDb(database.connectionString); + }, 60_000); + + afterAll(async () => { + await db?.$client.end(); + await database?.cleanup(); + }); + + async function fixture() { + const [company] = await db.insert(companies).values({ name: "Admission fixture", issuePrefix: `A${randomUUID().slice(0, 6)}` }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Admission agent", adapterType: "process" }).returning(); + const projectRows = await db.insert(projects).values(["Design", "Docs", "Unlimited"].map(name => ({ companyId: company.id, name }))) + .returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id }).returning(); + return { companyId: company.id, agentId: agent.id, projectIds: projectRows.map(row => row.id).sort(), runId: run.id }; + } + + const inTransaction = (work: (tx: TestDb) => Promise) => db.transaction(tx => work(tx as unknown as TestDb)); + const admit = (input: { companyId: string; runId: string; admissionId?: string; record: AdmissionRecord }) => + inTransaction(tx => recordAdmission(tx, { ...input, admissionId: input.admissionId ?? randomUUID() })); + const settle = (companyId: string, admissionId: string, settlementId = randomUUID()) => + inTransaction(tx => settleAdmission(tx, { companyId, admissionId, settlementId })); + const chargedProjects = async (admissionId: string) => (await db.select({ projectId: charges.projectId }).from(charges) + .where(eq(charges.admissionId, admissionId)).orderBy(asc(charges.projectId))).map(row => row.projectId); + + it("stores absent, explicit unlimited and positive maximums with partial updates", async () => { + const { companyId, projectIds: [projectId] } = await fixture(); + const policy = createExecutionAdmission(db); + const scope = { companyId, projectId }; + expect(await policy.readTaskExecutionPolicy(scope)).toEqual({ kind: "found", policy: { state: "absent", maxConcurrentTasks: null } }); + expect(await policy.updateTaskExecutionPolicy(scope, {})).toMatchObject({ kind: "unchanged", policy: { state: "absent" } }); + expect(await db.select().from(policies)).not.toContainEqual(expect.objectContaining({ projectId })); + + expect(await policy.updateTaskExecutionPolicy(scope, { maxConcurrentTasks: 2 })) + .toEqual({ kind: "updated", policy: { state: "configured", maxConcurrentTasks: 2 } }); + expect(await policy.updateTaskExecutionPolicy(scope, undefined)).toMatchObject({ kind: "unchanged", policy: { maxConcurrentTasks: 2 } }); + expect(await policy.updateTaskExecutionPolicy(scope, { maxConcurrentTasks: 0 })).toMatchObject({ kind: "invalid" }); + expect(await policy.updateTaskExecutionPolicy(scope, null)) + .toEqual({ kind: "updated", policy: { state: "configured", maxConcurrentTasks: null } }); + expect(await policy.readTaskExecutionPolicy(scope)).toEqual({ kind: "found", policy: { state: "configured", maxConcurrentTasks: null } }); + + const foreign = { companyId: (await fixture()).companyId, projectId }; + expect(await policy.readTaskExecutionPolicy(foreign)).toEqual({ kind: "project_not_found" }); + expect(await policy.updateTaskExecutionPolicy(foreign, { maxConcurrentTasks: 1 })).toEqual({ kind: "project_not_found" }); + expect(await policy.readTaskExecutionPolicy(scope)).toMatchObject({ policy: { maxConcurrentTasks: null } }); + }); + + it("enforces policy constraints and company scope in the database", async () => { + const { companyId, projectIds: [projectId] } = await fixture(); + const other = await fixture(); + await expect(db.insert(policies).values({ companyId, projectId, maxConcurrentTasks: 0 })) + .rejects.toMatchObject({ cause: { constraint_name: "project_task_execution_policies_maximum_check" } }); + await expect(db.insert(policies).values({ companyId: other.companyId, projectId, maxConcurrentTasks: 1 })) + .rejects.toMatchObject({ cause: { constraint_name: "project_task_execution_policies_company_project_fk" } }); + }); + + it("charges every distinct project, including unlimited and primaryless sets", async () => { + const { companyId, runId, projectIds: [a, b, c] } = await fixture(); + await createExecutionAdmission(db).updateTaskExecutionPolicy({ companyId, projectId: c }, null); + const admissionId = randomUUID(); + const admitted = record(a, [c, b, a]); + expect(await admit({ companyId, runId, admissionId, record: admitted })).toEqual({ kind: "recorded", attempt: 1 }); + expect(await chargedProjects(admissionId)).toEqual([a, b, c]); + const [epoch] = await db.select().from(admissions).where(eq(admissions.id, admissionId)); + expect(epoch).toMatchObject({ companyId, runId, attempt: 1, primaryProjectId: a, contextRevision: "issue:1", settlementId: null }); + + const other = await fixture(); + const primaryless = randomUUID(); + expect(await admit({ ...other, admissionId: primaryless, record: record(null, [other.projectIds[1]]) })) + .toMatchObject({ kind: "recorded" }); + expect(await chargedProjects(primaryless)).toEqual([other.projectIds[1]]); + + const empty = await fixture(); + const emptyId = randomUUID(); + expect(await admit({ ...empty, admissionId: emptyId, record: record(null, []) })).toMatchObject({ kind: "recorded" }); + expect(await chargedProjects(emptyId)).toEqual([]); + expect([...(await countActiveProjectCharges(db, empty.companyId, empty.projectIds)).values()]).toEqual([0, 0, 0]); + }); + + it("replays one admission identity and refuses a second live epoch for the run", async () => { + const { companyId, runId, projectIds: [a, b] } = await fixture(); + const admissionId = randomUUID(); + const admitted = record(a, [b]); + await admit({ companyId, runId, admissionId, record: admitted }); + expect(await admit({ companyId, runId, admissionId, record: admitted })).toEqual({ kind: "replayed", attempt: 1 }); + expect(await admit({ companyId, runId, admissionId, record: record(a, []) })).toEqual({ kind: "admission_conflict" }); + expect(await admit({ companyId, runId, admissionId, record: record(a, [b], "issue:2") })).toEqual({ kind: "admission_conflict" }); + expect(await admit({ companyId, runId, record: admitted })).toEqual({ kind: "run_already_admitted", admissionId }); + expect(await db.select({ id: admissions.id }).from(admissions).where(eq(admissions.runId, runId))).toHaveLength(1); + await expect(db.insert(admissions).values({ id: randomUUID(), companyId, runId, attempt: 2, contextRevision: "x" })) + .rejects.toMatchObject({ cause: { constraint_name: "execution_admissions_live_run_uq" } }); + }); + + it("settles the whole set once and starts a new attempt for the same run", async () => { + const { companyId, runId, projectIds: [a, b] } = await fixture(); + const first = randomUUID(); + await admit({ companyId, runId, admissionId: first, record: record(a, [b]) }); + const settlementId = randomUUID(); + expect(await settle(companyId, first, settlementId)).toEqual({ kind: "settled", releasedProjectIds: [a, b] }); + expect(await settle(companyId, first, settlementId)).toEqual({ kind: "already_settled" }); + expect(await settle(companyId, first)).toEqual({ kind: "settled_by_other" }); + expect(await settle(randomUUID(), first)).toEqual({ kind: "not_found" }); + expect(await chargedProjects(first)).toEqual([]); + expect(await admit({ companyId, runId, admissionId: first, record: record(a, [b]) })).toEqual({ kind: "admission_settled" }); + + const second = randomUUID(); + expect(await admit({ companyId, runId, admissionId: second, record: record(a, [], "issue:2") })).toEqual({ kind: "recorded", attempt: 2 }); + expect(await db.select({ attempt: admissions.attempt, settled: sql`${admissions.settledAt} is not null` }) + .from(admissions).where(eq(admissions.runId, runId)).orderBy(asc(admissions.attempt))) + .toEqual([{ attempt: 1, settled: true }, { attempt: 2, settled: false }]); + await expect(db.update(admissions).set({ settlementId: randomUUID() }).where(eq(admissions.id, second))) + .rejects.toMatchObject({ cause: { constraint_name: "execution_admissions_settlement_check" } }); + }); + + it("keeps company isolation and rolls back the whole record", async () => { + const { companyId, runId, projectIds: [a] } = await fixture(); + const other = await fixture(); + expect(await admit({ companyId, runId, record: record(a, [other.projectIds[0]]) })) + .toEqual({ kind: "projects_unavailable", projectIds: [other.projectIds[0]] }); + expect(await admit({ companyId, runId: other.runId, record: record(a, []) })).toEqual({ kind: "run_not_found" }); + const admissionId = randomUUID(); + await admit({ companyId, runId, admissionId, record: record(a, []) }); + expect(await admit({ companyId: other.companyId, runId: other.runId, admissionId, record: record(null, []) })) + .toEqual({ kind: "admission_conflict" }); + await expect(db.insert(charges).values({ admissionId, companyId, projectId: other.projectIds[0] })) + .rejects.toMatchObject({ cause: { constraint_name: "execution_admission_project_charges_company_project_fk" } }); + await expect(db.insert(charges).values({ admissionId, companyId, projectId: a })) + .rejects.toMatchObject({ cause: { constraint_name: "execution_admission_project_charges_admission_id_project_id_pk" } }); + + const rolledBack = randomUUID(); + await expect(inTransaction(async tx => { + await recordAdmission(tx, { companyId: other.companyId, runId: other.runId, admissionId: rolledBack, record: record(null, other.projectIds) }); + throw new Error("claim lost"); + })).rejects.toThrow("claim lost"); + expect(await db.select().from(admissions).where(eq(admissions.id, rolledBack))).toEqual([]); + expect(await chargedProjects(rolledBack)).toEqual([]); + await expect(recordAdmission(db, { companyId, runId, admissionId: randomUUID(), record: record(a, []) })) + .rejects.toThrow("requires a database transaction"); + }); + + it("counts active charges per project through the project index", async () => { + const first = await fixture(); + const [a, b, c] = first.projectIds; + const [secondRun] = await db.insert(heartbeatRuns).values({ companyId: first.companyId, agentId: first.agentId }).returning(); + const held = randomUUID(); + await admit({ ...first, admissionId: held, record: record(a, [b]) }); + await admit({ companyId: first.companyId, runId: secondRun.id, record: record(null, [a]) }); + expect(Object.fromEntries(await countActiveProjectCharges(db, first.companyId, [a, b, c]))).toEqual({ [a]: 2, [b]: 1, [c]: 0 }); + await settle(first.companyId, held); + expect(Object.fromEntries(await countActiveProjectCharges(db, first.companyId, [a, b, c]))).toEqual({ [a]: 1, [b]: 0, [c]: 0 }); + expect(Object.fromEntries(await countActiveProjectCharges(db, randomUUID(), [a]))).toEqual({ [a]: 0 }); + + const plan = await inTransaction(async tx => { + await tx.execute(sql`set local enable_seqscan = off`); + return tx.execute(sql`explain select project_id, count(*) from execution_admission_project_charges + where company_id = ${first.companyId} and project_id in (${a}, ${b}) group by project_id`); + }); + expect(JSON.stringify(plan)).toContain("execution_admission_project_charges_project_idx"); + }); + + it("preserves unsettled charges through project archive and delete", async () => { + const { companyId, runId, projectIds: [a, b] } = await fixture(); + const admissionId = randomUUID(); + await admit({ companyId, runId, admissionId, record: record(a, [b]) }); + const projectsService = projectService(db); + await db.update(projects).set({ archivedAt: new Date() }).where(eq(projects.id, b)); + expect(await chargedProjects(admissionId)).toEqual([a, b]); + await expect(projectsService.remove(a)).rejects.toMatchObject({ status: 409, details: { code: "project_execution_admission_pending" } }); + await expect(db.delete(projects).where(eq(projects.id, b))) + .rejects.toMatchObject({ cause: { constraint_name: "execution_admission_project_charges_company_project_fk" } }); + expect(await chargedProjects(admissionId)).toEqual([a, b]); + + await createExecutionAdmission(db).updateTaskExecutionPolicy({ companyId, projectId: a }, { maxConcurrentTasks: 1 }); + await settle(companyId, admissionId); + expect(await projectsService.remove(a)).toMatchObject({ id: a }); + expect(await db.select().from(policies).where(eq(policies.projectId, a))).toEqual([]); + expect(await db.select({ primaryProjectId: admissions.primaryProjectId }).from(admissions).where(eq(admissions.id, admissionId))) + .toEqual([{ primaryProjectId: a }]); + }); + + it("guards agent deletion while unsettled and removes admissions with the company", async () => { + const { companyId, agentId, runId, projectIds: [a] } = await fixture(); + const admissionId = randomUUID(); + await admit({ companyId, runId, admissionId, record: record(a, []) }); + await expect(withAccountingTransaction(db, companyId, tx => deleteAgentDependencies(tx, companyId, agentId))) + .rejects.toMatchObject({ status: 409, details: { code: "agent_execution_admission_pending" } }); + expect(await chargedProjects(admissionId)).toEqual([a]); + await settle(companyId, admissionId); + await withAccountingTransaction(db, companyId, tx => deleteAgentDependencies(tx, companyId, agentId)); + expect(await db.select().from(admissions).where(eq(admissions.runId, runId))).toEqual([]); + + const deleted = await fixture(); + const retained = await fixture(); + const retainedAdmission = randomUUID(); + await admit({ ...retained, admissionId: retainedAdmission, record: record(retained.projectIds[0], []) }); + await admit({ ...deleted, record: record(deleted.projectIds[0], deleted.projectIds) }); + await db.update(agents).set({ lifecycleState: "terminated", status: "terminated" }).where(eq(agents.id, deleted.agentId)); + expect(await deleteCompany(db, deleted.companyId)).toMatchObject({ id: deleted.companyId }); + expect(await db.select().from(admissions).where(eq(admissions.companyId, deleted.companyId))).toEqual([]); + expect(await chargedProjects(retainedAdmission)).toEqual([retained.projectIds[0]]); + }, 60_000); +}); diff --git a/server/src/modules/execution-admission/adapters/postgres.ts b/server/src/modules/execution-admission/adapters/postgres.ts new file mode 100644 index 00000000000..937eef72e14 --- /dev/null +++ b/server/src/modules/execution-admission/adapters/postgres.ts @@ -0,0 +1,165 @@ +import { + executionAdmissionProjectCharges as charges, + executionAdmissions as admissions, + heartbeatRuns, + projectTaskExecutionPolicies as policies, + projects, + type Db, +} from "@paperclipai/db"; +import { and, asc, eq, inArray, isNull, sql } from "drizzle-orm"; +import type { TaskExecutionPolicyStore } from "../application/policy.js"; +import { decideSettlement, sameAdmissionRecord, type AdmissionRecord } from "../domain/admission.js"; +import { ABSENT_TASK_EXECUTION_POLICY, type TaskExecutionPolicy } from "../domain/policy.js"; + +function assertTransaction(tx: Db) { + if (!("nestedIndex" in tx)) throw new Error("Execution admission storage requires a database transaction"); +} + +/** + * Global lock order step 3: locks the whole project set in ascending ID order + * in one statement. Returns the IDs that are not projects of the company. + */ +export async function lockAdmissionProjects(tx: Db, companyId: string, projectIds: readonly string[]) { + assertTransaction(tx); + if (projectIds.length === 0) return []; + const rows = await tx.select({ id: projects.id }).from(projects) + .where(and(eq(projects.companyId, companyId), inArray(projects.id, [...projectIds]))) + .orderBy(asc(projects.id)).for("no key update"); + const found = new Set(rows.map(row => row.id)); + return projectIds.filter(id => !found.has(id)); +} + +function toPolicy(row: { maxConcurrentTasks: number | null } | undefined): TaskExecutionPolicy { + return row ? { state: "configured", maxConcurrentTasks: row.maxConcurrentTasks } : ABSENT_TASK_EXECUTION_POLICY; +} + +export function createPostgresTaskExecutionPolicyStore(db: Db): TaskExecutionPolicyStore { + return { + async read({ companyId, projectId }) { + const [row] = await db.select({ configured: policies.projectId, maxConcurrentTasks: policies.maxConcurrentTasks }) + .from(projects).leftJoin(policies, eq(policies.projectId, projects.id)) + .where(and(eq(projects.companyId, companyId), eq(projects.id, projectId))); + if (!row) return null; + return row.configured ? toPolicy(row) : ABSENT_TASK_EXECUTION_POLICY; + }, + + update: ({ companyId, projectId }, change) => db.transaction(async transaction => { + const tx = transaction as unknown as Db; + if ((await lockAdmissionProjects(tx, companyId, [projectId])).length > 0) return null; + const [row] = await tx.select({ maxConcurrentTasks: policies.maxConcurrentTasks }).from(policies) + .where(eq(policies.projectId, projectId)); + const current = toPolicy(row); + const next = change(current); + if (next === current) return current; + if (next.state === "absent") throw new Error("A stored task execution policy cannot become absent"); + await tx.insert(policies).values({ projectId, companyId, maxConcurrentTasks: next.maxConcurrentTasks }) + .onConflictDoUpdate({ target: policies.projectId, set: { maxConcurrentTasks: next.maxConcurrentTasks, updatedAt: sql`now()` } }); + return next; + }), + }; +} + +export type RecordAdmissionInput = { companyId: string; runId: string; admissionId: string; record: AdmissionRecord }; + +export type RecordAdmissionOutcome = + | { kind: "recorded" | "replayed"; attempt: number } + | { kind: "projects_unavailable"; projectIds: string[] } + | { kind: "run_not_found" } + | { kind: "run_already_admitted"; admissionId: string } + | { kind: "admission_settled" } + | { kind: "admission_conflict" }; + +/** + * Storage transition only: persists one admission attempt and its charges in + * the caller's transaction. It neither checks capacity nor claims work; the + * composed admission command must do both under the same locks. Takes lock + * steps 3 (projects) and 6 (run); composed callers take them earlier. + */ +export async function recordAdmission(tx: Db, input: RecordAdmissionInput): Promise { + assertTransaction(tx); + const { companyId, runId, admissionId, record } = input; + const missing = await lockAdmissionProjects(tx, companyId, record.projectIds); + if (missing.length > 0) return { kind: "projects_unavailable", projectIds: missing }; + const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).for("no key update"); + if (!run) return { kind: "run_not_found" }; + + const [existing] = await tx.select().from(admissions).where(eq(admissions.id, admissionId)); + if (existing) { + if (existing.companyId !== companyId || existing.runId !== runId) return { kind: "admission_conflict" }; + if (existing.settledAt) return { kind: "admission_settled" }; + const charged = await tx.select({ projectId: charges.projectId }).from(charges) + .where(and(eq(charges.companyId, companyId), eq(charges.admissionId, admissionId))).orderBy(asc(charges.projectId)); + const stored = { primaryProjectId: existing.primaryProjectId, contextRevision: existing.contextRevision, + projectIds: charged.map(row => row.projectId) }; + return sameAdmissionRecord(stored, record) ? { kind: "replayed", attempt: existing.attempt } : { kind: "admission_conflict" }; + } + const [live] = await tx.select({ id: admissions.id }).from(admissions) + .where(and(eq(admissions.runId, runId), isNull(admissions.settledAt))); + if (live) return { kind: "run_already_admitted", admissionId: live.id }; + + const [{ attempt }] = await tx.select({ attempt: sql`coalesce(max(${admissions.attempt}), 0)::int + 1` }) + .from(admissions).where(eq(admissions.runId, runId)); + await tx.insert(admissions).values({ id: admissionId, companyId, runId, attempt, + primaryProjectId: record.primaryProjectId, contextRevision: record.contextRevision }); + if (record.projectIds.length > 0) { + await tx.insert(charges).values(record.projectIds.map(projectId => ({ admissionId, companyId, projectId }))); + } + return { kind: "recorded", attempt }; +} + +export type SettleAdmissionOutcome = + | { kind: "settled"; releasedProjectIds: string[] } + | { kind: "already_settled" | "settled_by_other" | "not_found" }; + +/** + * Storage transition only: releases the admission's whole charge set once. + * Callers must first verify authoritative lifecycle facts. Takes no project lock. + */ +export async function settleAdmission(tx: Db, input: { companyId: string; admissionId: string; settlementId: string }): +Promise { + assertTransaction(tx); + const { companyId, admissionId, settlementId } = input; + const [row] = await tx.select({ settlementId: admissions.settlementId }).from(admissions) + .where(and(eq(admissions.companyId, companyId), eq(admissions.id, admissionId))).for("no key update"); + if (!row) return { kind: "not_found" }; + const decision = decideSettlement(row.settlementId, settlementId); + if (decision !== "settle") return { kind: decision }; + const released = await tx.delete(charges) + .where(and(eq(charges.companyId, companyId), eq(charges.admissionId, admissionId))) + .returning({ projectId: charges.projectId }); + await tx.update(admissions).set({ settlementId, settledAt: sql`now()` }).where(eq(admissions.id, admissionId)); + return { kind: "settled", releasedProjectIds: released.map(charge => charge.projectId).sort() }; +} + +/** Active charge counts per project. Every stored charge is unsettled. */ +export async function countActiveProjectCharges(db: Db, companyId: string, projectIds: readonly string[]) { + const counts = new Map(projectIds.map(id => [id, 0])); + if (projectIds.length === 0) return counts; + const rows = await db.select({ projectId: charges.projectId, active: sql`count(*)::int` }).from(charges) + .where(and(eq(charges.companyId, companyId), inArray(charges.projectId, [...projectIds]))) + .groupBy(charges.projectId); + for (const row of rows) counts.set(row.projectId, row.active); + return counts; +} + +/** Project deletion holds the project row lock, so no charge can be added concurrently. */ +export async function hasUnsettledProjectCharges(db: Db, scope: { companyId: string; projectId: string }) { + const [row] = await db.select({ admissionId: charges.admissionId }).from(charges) + .where(and(eq(charges.companyId, scope.companyId), eq(charges.projectId, scope.projectId))).limit(1); + return Boolean(row); +} + +/** Removes settled history of an agent's runs before they are deleted; refuses while any admission is unsettled. */ +export async function deleteAgentAdmissionHistory(tx: Db, scope: { companyId: string; agentId: string }) { + assertTransaction(tx); + const agentRuns = tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.companyId, scope.companyId), eq(heartbeatRuns.agentId, scope.agentId))); + const [live] = await tx.select({ id: admissions.id }).from(admissions) + .where(and(eq(admissions.companyId, scope.companyId), inArray(admissions.runId, agentRuns), isNull(admissions.settledAt))) + .limit(1); + if (live) return { kind: "unsettled" as const }; + await tx.delete(admissions).where(and(eq(admissions.companyId, scope.companyId), inArray(admissions.runId, agentRuns))); + return { kind: "deleted" as const }; +} + diff --git a/server/src/modules/execution-admission/application/policy.ts b/server/src/modules/execution-admission/application/policy.ts new file mode 100644 index 00000000000..3498b31102a --- /dev/null +++ b/server/src/modules/execution-admission/application/policy.ts @@ -0,0 +1,50 @@ +import { + applyTaskExecutionPolicyPatch, + parseTaskExecutionPolicyPatch, + type TaskExecutionPolicy, +} from "../domain/policy.js"; + +export type ProjectScope = { companyId: string; projectId: string }; + +export interface TaskExecutionPolicyStore { + /** Null when the project is not in the company. */ + read(scope: ProjectScope): Promise; + /** + * Locks the project in admission lock order and stores `change(current)`. + * Returning `current` itself writes nothing. Null when the project is not in the company. + */ + update(scope: ProjectScope, change: (current: TaskExecutionPolicy) => TaskExecutionPolicy): Promise; +} + +export type PolicyReadResult = { kind: "found"; policy: TaskExecutionPolicy } | { kind: "project_not_found" }; + +export type PolicyUpdateResult = + | { kind: "updated" | "unchanged"; policy: TaskExecutionPolicy } + | { kind: "invalid"; field: "policy" | "maxConcurrentTasks"; reason: string } + | { kind: "project_not_found" }; + +/** + * Internal policy commands. No route, tool or import calls `update` until + * admission enforces the maximum; a cap written earlier would mislead users. + */ +export function createTaskExecutionPolicyCommands(store: TaskExecutionPolicyStore) { + return { + async readTaskExecutionPolicy(scope: ProjectScope): Promise { + const policy = await store.read(scope); + return policy ? { kind: "found", policy } : { kind: "project_not_found" }; + }, + + async updateTaskExecutionPolicy(scope: ProjectScope, input: unknown): Promise { + const parsed = parseTaskExecutionPolicyPatch(input); + if (parsed.kind === "invalid") return parsed; + let changed = false; + const policy = await store.update(scope, current => { + const next = applyTaskExecutionPolicyPatch(current, parsed.patch); + changed = next !== current; + return next; + }); + if (!policy) return { kind: "project_not_found" }; + return { kind: changed ? "updated" : "unchanged", policy }; + }, + }; +} diff --git a/server/src/modules/execution-admission/company-deletion.ts b/server/src/modules/execution-admission/company-deletion.ts new file mode 100644 index 00000000000..cfaf4097f2a --- /dev/null +++ b/server/src/modules/execution-admission/company-deletion.ts @@ -0,0 +1 @@ +export { executionAdmissionCompanyDeletion } from "./adapters/company-deletion.js"; diff --git a/server/src/modules/execution-admission/domain/admission.ts b/server/src/modules/execution-admission/domain/admission.ts new file mode 100644 index 00000000000..4177e9c4c06 --- /dev/null +++ b/server/src/modules/execution-admission/domain/admission.ts @@ -0,0 +1,53 @@ +/** Referenced projects are rejected beyond this bound instead of admitting a subset. */ +export const MAX_ADMISSION_PROJECTS = 100; +const MAX_CONTEXT_REVISION_LENGTH = 256; +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +export type AdmissionRecordInput = { + primaryProjectId: string | null; + referencedProjectIds: readonly string[]; + contextRevision: string; +}; + +/** + * `projectIds` is the complete distinct set in canonical lock order. Lowercase + * UUID text order matches PostgreSQL uuid order. An empty set is valid and + * charges no project. + */ +export type AdmissionRecord = { + primaryProjectId: string | null; + projectIds: string[]; + contextRevision: string; +}; + +export type AdmissionRecordResult = + | { kind: "ok"; record: AdmissionRecord } + | { kind: "invalid_project_id" } + | { kind: "too_many_projects"; count: number; limit: number } + | { kind: "invalid_context_revision" }; + +export function prepareAdmissionRecord(input: AdmissionRecordInput): AdmissionRecordResult { + const contextRevision = input.contextRevision.trim(); + if (!contextRevision || contextRevision.length > MAX_CONTEXT_REVISION_LENGTH) return { kind: "invalid_context_revision" }; + const primaryProjectId = input.primaryProjectId?.toLowerCase() ?? null; + const candidates = [...input.referencedProjectIds, ...(primaryProjectId === null ? [] : [primaryProjectId])].map(id => id.toLowerCase()); + if (candidates.some(id => !UUID.test(id))) return { kind: "invalid_project_id" }; + const projectIds = [...new Set(candidates)].sort(); + if (projectIds.length > MAX_ADMISSION_PROJECTS) { + return { kind: "too_many_projects", count: projectIds.length, limit: MAX_ADMISSION_PROJECTS }; + } + return { kind: "ok", record: { primaryProjectId, projectIds, contextRevision } }; +} + +export function sameAdmissionRecord(left: AdmissionRecord, right: AdmissionRecord) { + return left.primaryProjectId === right.primaryProjectId && left.contextRevision === right.contextRevision + && left.projectIds.length === right.projectIds.length && left.projectIds.every((id, index) => id === right.projectIds[index]); +} + +export type SettlementDecision = "settle" | "already_settled" | "settled_by_other"; + +/** A settlement identity can be replayed; a second, different settlement cannot take over. */ +export function decideSettlement(existingSettlementId: string | null, settlementId: string): SettlementDecision { + if (existingSettlementId === null) return "settle"; + return existingSettlementId === settlementId ? "already_settled" : "settled_by_other"; +} diff --git a/server/src/modules/execution-admission/domain/policy.test.ts b/server/src/modules/execution-admission/domain/policy.test.ts new file mode 100644 index 00000000000..0c5ce8ebd6d --- /dev/null +++ b/server/src/modules/execution-admission/domain/policy.test.ts @@ -0,0 +1,88 @@ +import { describe, expect, it } from "vitest"; +import { decideSettlement, MAX_ADMISSION_PROJECTS, prepareAdmissionRecord } from "./admission.js"; +import { ABSENT_TASK_EXECUTION_POLICY, applyTaskExecutionPolicyPatch, parseTaskExecutionPolicyPatch, type TaskExecutionPolicy } from "./policy.js"; + +const configured = (maxConcurrentTasks: number | null): TaskExecutionPolicy => ({ state: "configured", maxConcurrentTasks }); +const apply = (current: TaskExecutionPolicy, input: unknown) => { + const parsed = parseTaskExecutionPolicyPatch(input); + if (parsed.kind !== "ok") throw new Error(parsed.reason); + return applyTaskExecutionPolicyPatch(current, parsed.patch); +}; + +describe("task execution policy", () => { + it("keeps never-configured distinct from explicit unlimited", () => { + expect(apply(ABSENT_TASK_EXECUTION_POLICY, null)).toEqual(configured(null)); + expect(apply(ABSENT_TASK_EXECUTION_POLICY, { maxConcurrentTasks: null })).toEqual(configured(null)); + expect(apply(ABSENT_TASK_EXECUTION_POLICY, { maxConcurrentTasks: 3 })).toEqual(configured(3)); + }); + + it("preserves the current value for omitted policies and fields", () => { + for (const current of [ABSENT_TASK_EXECUTION_POLICY, configured(null), configured(2)]) { + expect(apply(current, undefined)).toBe(current); + expect(apply(current, {})).toBe(current); + expect(apply(current, { maxConcurrentTasks: undefined })).toBe(current); + } + const two = configured(2); + expect(apply(two, { maxConcurrentTasks: 2 })).toBe(two); + const unlimited = configured(null); + expect(apply(unlimited, null)).toBe(unlimited); + }); + + it("clears or replaces a configured maximum", () => { + expect(apply(configured(2), null)).toEqual(configured(null)); + expect(apply(configured(2), { maxConcurrentTasks: null })).toEqual(configured(null)); + expect(apply(configured(null), { maxConcurrentTasks: 1 })).toEqual(configured(1)); + }); + + it.each([0, -1, 1.5, Number.NaN, Number.POSITIVE_INFINITY, 2_147_483_648, "2", true, {}])( + "rejects maximum %s", value => { + expect(parseTaskExecutionPolicyPatch({ maxConcurrentTasks: value })).toMatchObject({ kind: "invalid", field: "maxConcurrentTasks" }); + }, + ); + + it("rejects malformed policies and unknown fields", () => { + for (const input of [1, "unlimited", [], { maxConcurrentTasks: 1, perTaskOverride: 3 }]) { + expect(parseTaskExecutionPolicyPatch(input)).toMatchObject({ kind: "invalid", field: "policy" }); + } + expect(parseTaskExecutionPolicyPatch({ maxConcurrentTasks: 2_147_483_647 })).toEqual({ kind: "ok", patch: { maxConcurrentTasks: 2_147_483_647 } }); + }); +}); + +describe("admission record", () => { + const a = "0b9d8e52-6f2e-4bb0-9a4a-1f0a7f2e0a01"; + const b = "7c1f3d9a-2b4e-4d8f-8a6b-3e5c7d9f1b02"; + const c = "f3a2b1c0-9d8e-4f7a-8b6c-5d4e3f2a1b03"; + + it("unions the primary with referenced projects in canonical order", () => { + expect(prepareAdmissionRecord({ primaryProjectId: c, referencedProjectIds: [b, a, b.toUpperCase(), c], contextRevision: " r1 " })) + .toEqual({ kind: "ok", record: { primaryProjectId: c, projectIds: [a, b, c], contextRevision: "r1" } }); + }); + + it("keeps a primaryless set and an empty set without a project bucket", () => { + expect(prepareAdmissionRecord({ primaryProjectId: null, referencedProjectIds: [b], contextRevision: "r" })) + .toEqual({ kind: "ok", record: { primaryProjectId: null, projectIds: [b], contextRevision: "r" } }); + expect(prepareAdmissionRecord({ primaryProjectId: null, referencedProjectIds: [], contextRevision: "r" })) + .toEqual({ kind: "ok", record: { primaryProjectId: null, projectIds: [], contextRevision: "r" } }); + }); + + it("rejects invalid references, oversized sets and missing context identity instead of narrowing", () => { + expect(prepareAdmissionRecord({ primaryProjectId: null, referencedProjectIds: [a, "Design"], contextRevision: "r" })) + .toEqual({ kind: "invalid_project_id" }); + expect(prepareAdmissionRecord({ primaryProjectId: "", referencedProjectIds: [a], contextRevision: "r" })) + .toEqual({ kind: "invalid_project_id" }); + const many = Array.from({ length: MAX_ADMISSION_PROJECTS + 1 }, (_, index) => + `00000000-0000-4000-8000-${index.toString(16).padStart(12, "0")}`); + expect(prepareAdmissionRecord({ primaryProjectId: null, referencedProjectIds: many, contextRevision: "r" })) + .toEqual({ kind: "too_many_projects", count: MAX_ADMISSION_PROJECTS + 1, limit: MAX_ADMISSION_PROJECTS }); + expect(prepareAdmissionRecord({ primaryProjectId: a, referencedProjectIds: [], contextRevision: " " })) + .toEqual({ kind: "invalid_context_revision" }); + expect(prepareAdmissionRecord({ primaryProjectId: a, referencedProjectIds: [], contextRevision: "x".repeat(257) })) + .toEqual({ kind: "invalid_context_revision" }); + }); + + it("settles once and replays only the same settlement", () => { + expect(decideSettlement(null, a)).toBe("settle"); + expect(decideSettlement(a, a)).toBe("already_settled"); + expect(decideSettlement(a, b)).toBe("settled_by_other"); + }); +}); diff --git a/server/src/modules/execution-admission/domain/policy.ts b/server/src/modules/execution-admission/domain/policy.ts new file mode 100644 index 00000000000..e171011d3b2 --- /dev/null +++ b/server/src/modules/execution-admission/domain/policy.ts @@ -0,0 +1,47 @@ +/** + * A project's task-execution policy. `absent` was never configured, so a later + * legacy migration may still choose a value. `configured` with a null maximum + * is an explicit "no limit" that such a migration must preserve. + */ +export type TaskExecutionPolicy = + | { state: "absent"; maxConcurrentTasks: null } + | { state: "configured"; maxConcurrentTasks: number | null }; + +/** Null clears the cap explicitly. An omitted field preserves its value. */ +export type TaskExecutionPolicyPatch = { maxConcurrentTasks?: number | null } | null; + +export type PolicyPatchResult = + | { kind: "ok"; patch: TaskExecutionPolicyPatch } + | { kind: "invalid"; field: "policy" | "maxConcurrentTasks"; reason: string }; + +export const ABSENT_TASK_EXECUTION_POLICY: TaskExecutionPolicy = { state: "absent", maxConcurrentTasks: null }; + +// The column is a PostgreSQL integer. +const MAX_STORED_MAXIMUM = 2_147_483_647; + +/** Undefined (an omitted policy) preserves everything; null clears the cap. */ +export function parseTaskExecutionPolicyPatch(input: unknown): PolicyPatchResult { + if (input === undefined) return { kind: "ok", patch: {} }; + if (input === null) return { kind: "ok", patch: null }; + if (typeof input !== "object" || Array.isArray(input)) { + return { kind: "invalid", field: "policy", reason: "Policy must be an object or null" }; + } + const unknownKey = Object.keys(input).find(key => key !== "maxConcurrentTasks"); + if (unknownKey) return { kind: "invalid", field: "policy", reason: `Unknown policy field ${unknownKey}` }; + if (!("maxConcurrentTasks" in input)) return { kind: "ok", patch: {} }; + const value = (input as { maxConcurrentTasks: unknown }).maxConcurrentTasks; + if (value === undefined) return { kind: "ok", patch: {} }; + if (value === null) return { kind: "ok", patch: { maxConcurrentTasks: null } }; + if (typeof value !== "number" || !Number.isInteger(value) || value < 1 || value > MAX_STORED_MAXIMUM) { + return { kind: "invalid", field: "maxConcurrentTasks", reason: "Maximum concurrent tasks must be a positive whole number" }; + } + return { kind: "ok", patch: { maxConcurrentTasks: value } }; +} + +/** Applies a validated patch. Returns the current policy when nothing changes. */ +export function applyTaskExecutionPolicyPatch(current: TaskExecutionPolicy, patch: TaskExecutionPolicyPatch): TaskExecutionPolicy { + if (patch !== null && patch.maxConcurrentTasks === undefined) return current; + const maxConcurrentTasks = patch === null ? null : patch.maxConcurrentTasks ?? null; + if (current.state === "configured" && current.maxConcurrentTasks === maxConcurrentTasks) return current; + return { state: "configured", maxConcurrentTasks }; +} diff --git a/server/src/modules/execution-admission/index.ts b/server/src/modules/execution-admission/index.ts new file mode 100644 index 00000000000..85878c38a05 --- /dev/null +++ b/server/src/modules/execution-admission/index.ts @@ -0,0 +1,17 @@ +import type { Db } from "@paperclipai/db"; +import { createPostgresTaskExecutionPolicyStore } from "./adapters/postgres.js"; +import { createTaskExecutionPolicyCommands } from "./application/policy.js"; + +export type { TaskExecutionPolicy, TaskExecutionPolicyPatch } from "./domain/policy.js"; +export type { PolicyReadResult, PolicyUpdateResult, ProjectScope } from "./application/policy.js"; +// Lifecycle guards run inside the caller's deletion transaction. +export { deleteAgentAdmissionHistory, hasUnsettledProjectCharges } from "./adapters/postgres.js"; + +/** + * Execution admission owns project task-execution policy and the durable + * admission epochs that charge project capacity. Admission and settlement + * commands are not composed yet; nothing enforces the maximum. + */ +export function createExecutionAdmission(db: Db) { + return createTaskExecutionPolicyCommands(createPostgresTaskExecutionPolicyStore(db)); +} diff --git a/server/src/services/agent-deletion.ts b/server/src/services/agent-deletion.ts index ca18e8f02ec..245919fff3a 100644 --- a/server/src/services/agent-deletion.ts +++ b/server/src/services/agent-deletion.ts @@ -14,6 +14,7 @@ import { agents, agentApiKeys, agentRuntimeState, agentTaskSessions, agentWakeup caseHandoffs, principalPermissionGrants, type Db } from "@paperclipai/db"; import { conflict } from "../errors.js"; import { issueThreadInteractionService } from "./issue-thread-interactions.js"; +import { deleteAgentAdmissionHistory } from "../modules/execution-admission/index.js"; export async function deleteAgentDependencies(tx: Db, companyId: string, id: string) { const [retainedHandoff] = await tx.select({ id: caseHandoffs.id }).from(caseHandoffs).where(and( @@ -63,6 +64,11 @@ export async function deleteAgentDependencies(tx: Db, companyId: string, id: str await notifyInstructionCollection(tx, { companyId: companyId }); await notifyCancelledWorkspacePreservation(tx, { companyId: companyId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId }); + if ((await deleteAgentAdmissionHistory(tx, { companyId, agentId: id })).kind === "unsettled") { + throw conflict("Wait for this agent's admitted task executions to settle before deleting it", { + code: "agent_execution_admission_pending", + }); + } await tx.delete(heartbeatRuns).where(eq(heartbeatRuns.agentId, id)); await tx.delete(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, id)); await tx.delete(agentApiKeys).where(eq(agentApiKeys.agentId, id)); diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 66e81df7673..2d341072d2f 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -20,6 +20,7 @@ import { eq, inArray } from "drizzle-orm"; import { createAgentLifecycle, processAgentLifecycle } from "./agent-lifecycle.js"; import { requestRestorePointDeletion } from "./git-restore-point-ledger.js"; import { agentLifecycleCompanyDeletion } from "../modules/agent-lifecycle/company-deletion.js"; +import { executionAdmissionCompanyDeletion } from "../modules/execution-admission/company-deletion.js"; import { companies, agents, @@ -117,6 +118,7 @@ export async function deleteCompany(db: Db, id: string) { await notifyInstructionCollection(tx, { companyId: id }); await notifyCancelledWorkspacePreservation(tx, { companyId: id }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: id }); + await executionAdmissionCompanyDeletion.deleteCompanyData(tx, id); await tx.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, id)); await tx.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, id)); await tx.delete(agentApiKeys).where(eq(agentApiKeys.companyId, id)); diff --git a/server/src/services/projects.ts b/server/src/services/projects.ts index 45891a218a6..5af27d351a7 100644 --- a/server/src/services/projects.ts +++ b/server/src/services/projects.ts @@ -43,6 +43,7 @@ import { parseProjectExecutionWorkspacePolicy } from "./execution-workspace-poli import { mergeProjectWorkspaceRuntimeConfig, readProjectWorkspaceRuntimeConfig } from "./project-workspace-runtime-config.js"; import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; import { recordProjectLifecycleEvent, recordResourceCreationEvent } from "./resource-lifecycle-events.js"; +import { hasUnsettledProjectCharges } from "../modules/execution-admission/index.js"; type ProjectTransaction = Parameters[0]>[0]; @@ -1085,6 +1086,11 @@ export function projectService(db: Db, options: { captureWorkspaceUpdates?: bool const [retained] = await tx.select({ id: executionWorkspaces.id }).from(executionWorkspaces) .where(eq(executionWorkspaces.projectId, id)).limit(1); if (retained) throw conflict("This project still supplies access policy for retained task workspaces. Archive the project or explicitly clean up its workspaces before deleting it."); + if (await hasUnsettledProjectCharges(tx as unknown as Db, { companyId: project.companyId, projectId: id })) { + throw conflict("Task executions admitted to this project have not finished saving. Wait for them to settle before deleting the project.", { + code: "project_execution_admission_pending", + }); + } await tx.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(eq(projectWorkspaces.projectId, id)).for("update"); await recordProjectLifecycleEvent(tx as unknown as Db, project.companyId, id, "delete"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); From 203511852618cb729928d06c1e0353ef022da394 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:36:22 -0600 Subject: [PATCH 05/16] fix(workspaces): preserve archived data during remote migration (#18826) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace storage lets the web server run without a persistent volume. > - Existing deployments also contain archived and historical task workspaces. > - Retention can remove their folders while preserving encrypted restore points or Git reconstruction records. > - The importer rejected those records and only accepted the task's current workspace. > - This pull request imports verified saved data without reopening or rebinding tasks. > - The benefit is preservation of existing work during an explicit storage migration. ## Linked Issues or Issue Description Refs #18403 and #17472. **What happened?** An explicit remote-workspace import rejected archived records, historical task bindings, and removed folders with valid retained data. **Expected behavior** Import the recorded files or exact Git commit. Preserve archived state and the task's current binding. Refuse missing, expired, corrupt, or changed sources. **Steps to reproduce** Archive a task directory with a restore point, remove its folder, and run the remote-workspace import. Repeat with a Git restore point containing an unpushed commit and staged and unstaged edits. The previous importer reports an inactive workspace or missing source. ## What Changed - Preserve lifecycle state and current task binding while importing archived and historical records. Recheck the binding before publication and on resume. - Expose the task-directory owner's validated restore-point reader. Reuse it for ordinary restoration and migration. - Restore archived directories and Git data into private scratch space. Support saved Git recipes and exact legacy branch-deletion receipts. - Recover pruned unpushed Git commits from the encrypted bundle and its explicit prerequisites. - Keep independent snapshot download, restoration, source checks, writer guards, encryption, and atomic publication. - Document the supported import sources and the remaining tenant-enrollment gaps. ## Verification - Before production edits, two new regressions failed on archived status and historical task binding. - Final focused verification: 58 tests pass across remote-workspace migration and isolated task-directory restoration. - Real PostgreSQL, Git repositories, encrypted storage, and restored file bytes cover cold restoration after removing the old folder and source object, pruned unpushed history, index and worktree edits, corrupt/missing/expired sources, binding changes, and source-folder races. - A Git fixture initially lacked runtime ownership metadata. Correcting the fixture made the valid retained-source case pass; no production validation was weakened. - Full repository typecheck and build passed. Full `pnpm test:run` stopped in chat integration shard 2 after two failures: a retry-notice timeout and a subsequent fake-timer error. All four cases in that retry-notice group passed in a separate focused rerun without source changes; the complete suite has not passed locally. All exact-head CI jobs passed. The separate Paperclip Review app check is still queued; all existing review threads are resolved. Server typecheck passed after the review fixes. - Both review findings were reproduced: the real cleanup owner's verified-tip receipt was rejected, and a lost upload reply for a legacy Git source failed with `operation_content_changed`. The fixes accept the owner's scoped receipt metadata and rebuild reconstructed Git indexes from their staged trees. Both regressions now pass, including retained staged and unstaged edits. - Additional coverage restores from a fresh HTTP clone after deleting the original repository, imports a nested local checkout with encrypted settings, and rejects an unapproved nested source without publishing remote authority. - This PR does not claim a completed tenant enrollment or a new zero-downtime deployment proof. ## Risks Imports can fail when old receipts or repository objects are unavailable. They retain their migration journal and leave the original state intact. Operators must resolve pending writers and recovery before importing. Shared physical roots, nested worktrees, and unresolved sandbox recovery require separate migration work. This change does not remove those blockers or alter automatic retention. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task ID or instance-derived details - [x] I have run the focused tests locally and they pass; full typecheck/build passed and the full-suite failure is documented - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 16 ++ ...-existing-workspace-ephemeral-migration.md | 45 +++ .../isolated-task-directory-retention.ts | 30 +- .../remote-workspace-migration-git.ts | 142 ++++++++++ .../remote-workspace-migration.test.ts | 263 +++++++++++++++++- .../services/remote-workspace-migration.ts | 74 ++++- 6 files changed, 547 insertions(+), 23 deletions(-) create mode 100644 doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md create mode 100644 server/src/services/remote-workspace-migration-git.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 182afe1d850..78621549793 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1878,6 +1878,22 @@ or recovery, unsafe paths, and explicitly concurrent shared writers block import. An interrupted apply retains its journal and admission hold; retry the same command to reconcile it before starting another writer. +An import preserves the workspace's lifecycle state and the task's current +workspace binding. It can import an archived or superseded workspace without +reopening the task or selecting that older workspace for future turns. A changed +task binding during the import blocks publication. + +If an archived folder has already been removed, the importer can reconstruct it +in private scratch space from its encrypted task-directory restore point or +validated Git restoration recipe. For older Git workspaces without a recipe, +it requires a successful branch-deletion receipt and the exact saved commit in +the verified source repository. It never substitutes today's default branch. +Git restore points retain unpushed history and staged and unstaged edits, even +when the source repository has pruned the saved branch. Missing, expired, +corrupt, or unbound sources block the import. Publication still requires an +independent download and restore of the new encrypted snapshot. The old folder +remains absent, and the importer does not change the original restore point. + Supported sources include empty task folders, repository URLs, and managed mutable repositories. Authorized mentioned projects attach all their configured Git repositories as writable copies under `.paperclip-repositories/` in the task diff --git a/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md new file mode 100644 index 00000000000..65c7dd3d351 --- /dev/null +++ b/doc/plans/2026-11-03-existing-workspace-ephemeral-migration.md @@ -0,0 +1,45 @@ +# Preserve existing workspaces during ephemeral enrollment + +The deployment endpoint remains a volume-free Cloud tenant, followed by a real +rolling deployment while an agent works and new work is admitted. Enrollment must +preserve existing task files and historical recovery sources. It must not reset +task workspaces, manufacture successful copyback, or weaken migration guards. + +## Evidence and remaining work + +| Required outcome | Current evidence | Remaining gap | Owner / next action | +| --- | --- | --- | --- | +| Existing active task retains files | The current import encrypts, uploads, independently restores, compares the source again and publishes under the lifecycle lock. A live completed task imported successfully with its original directory retained. | The importer only accepts an active workspace with the current task binding. | Workspace migration owner: support historical records without changing the task binding or lifecycle state. | +| Archived task retains its saved files | Retention records include encrypted task-directory restore points, Git restore points and verified Git reconstruction recipes. | The importer rejects missing local folders even when an existing restore point can reconstruct them. | Restore-point owners: reconstruct into migration scratch space, reject expired/missing/corrupt sources, and feed the verified result into the normal import publication. | +| Shared project roots preserve all task files | Legacy roots contain nested task worktrees and multiple records can reference one physical folder. | A naive root capture recursively imports those other workspaces; the current per-record writer check does not fence aliases. | Workspace migration owner: inventory physical overlaps, fence all writers, and preserve nested workspace coverage before excluding any directory. | +| Failed runs retain unsaved files | Provider inspection distinguishes surviving archived sandboxes from missing allocations. Existing recovery validation accepts some sources and rejects others for missing provenance or ambiguous allocation ownership. | Recovery must settle or durably preserve each exact source. A successful old preservation action does not necessarily update the legacy sync descriptor. | Workspace recovery owner: verify existing preservation receipts, recover eligible sources, and retain explicit unresolved obligations for unavailable/ambiguous sources. Do not replay agent work. | +| Tenant is eligible without its volume | External DB and S3 are configured; the original volume remains attached. | Workspaces, local run logs and remaining owner-maintained data must pass the packaged maintenance receipt. | Cloud enrollment owner, after workspace migration is complete. | +| Ordinary upgrades remain available | The no-drain Cloud deployment path is merged and qualified on another enrolled tenant. | Exact final app image qualification and a live deployment on this tenant. | Cloud deployment owner: qualify, enroll, then prove reads, submissions and active-run adoption through an actual replacement. | + +## Acceptance boundaries + +The first implementation slice supports archived and historical records, +encrypted task-directory restoration, Git restoration recipes, and legacy +branch-deletion receipts. PostgreSQL and real-file tests prove cold restoration +of saved bytes, index state, and unpushed commits, including after pruning the +old Git object store. Invalid or unavailable retained data still blocks import. +Shared-root migration and recovery-obligation reconciliation remain separate +gaps; this slice does not make a whole tenant eligible for enrollment. + +- Import existing files or a verified retained source. A missing folder alone is + not permission to publish an empty workspace or choose a newer Git branch. +- Preserve archived status, task bindings, task history, unpushed Git history, + staged and unstaged edits, and saved recovery copies. +- Preserve original files and the detached original volume until verified + migration receipts and the retention interval permit removal. +- Parent roots may omit a nested workspace only with durable, inspectable + coverage of that exact nested directory. Arbitrary exclusion globs are not an + acceptable data migration. +- Resume interrupted imports with their original identity; cancellation may + release an empty reservation only after the producer is stopped and no upload + was journaled. Preserve an audit receipt. +- Verify restored file bytes independently, including after making the old + source unavailable in an isolated test. Exercise current and historical + bindings, archived sources, overlap/writer races, and unavailable sources. +- Unrelated tenants, production services, and agent task execution remain outside + the migration's mutation scope. diff --git a/server/src/services/isolated-task-directory-retention.ts b/server/src/services/isolated-task-directory-retention.ts index cb5a8dc8a37..b4c83c70e66 100644 --- a/server/src/services/isolated-task-directory-retention.ts +++ b/server/src/services/isolated-task-directory-retention.ts @@ -124,7 +124,7 @@ function restorePointObjectKey(companyId: string, workspaceId: string, generatio return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/task-directory/${id}.pcrp`; } -function validateRestorePointRef(ref: TaskDirectoryRestorePointRef | null | undefined, workspace: { companyId: string; id: string }) { +export function validateTaskDirectoryRestorePointRef(ref: TaskDirectoryRestorePointRef | null | undefined, workspace: { companyId: string; id: string }) { if (!ref || ref.schema !== TASK_DIRECTORY_RESTORE_POINT_SCHEMA || typeof ref.id !== "string" || !UUID.test(ref.id) || !Number.isSafeInteger(ref.generation) || ref.generation < 1 || ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id) @@ -136,6 +136,25 @@ function validateRestorePointRef(ref: TaskDirectoryRestorePointRef | null | unde return ref; } +/** Read a retained directory without reopening its workspace or replacing its + * original path. Administrative migration uses the same identity and byte checks + * as ordinary restore, and never treats an expired object as an empty folder. */ +export async function loadTaskDirectoryRestorePoint(input: { + storage: StorageService; + workspace: { companyId: string; id: string }; + ref: TaskDirectoryRestorePointRef; + signal?: AbortSignal; +}) { + const ref = validateTaskDirectoryRestorePointRef(input.ref, input.workspace); + if (ref.expiredAt) throw new TaskDirectoryRestoreFailure("restore_point_expired"); + const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(() => { + throw new TaskDirectoryRestoreFailure("restore_point_unavailable"); + }); + if (ciphertext.length !== ref.byteSize) throw new TaskDirectoryRestoreFailure("ciphertext_size_mismatch"); + return openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key, + identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation } }); +} + function skipReason(error: unknown) { if (error instanceof TaskDirectorySnapshotSkip) return error.reason; if (error instanceof IsolatedRepositorySourceBusy) return "source_busy"; @@ -341,7 +360,7 @@ export async function restoreIsolatedTaskDirectory(input: { const parent = path.dirname(cwd); await fs.mkdir(parent, { recursive: true, mode: 0o700 }); if (await fs.realpath(parent) !== parent) throw new TaskDirectoryRestoreFailure("task_directory_identity_changed"); - const ref = validateRestorePointRef(state.restorePoint, input.workspace); + const ref = validateTaskDirectoryRestorePointRef(state.restorePoint, input.workspace); if (await directoryState(cwd) === "present") { if (!state.removed) return { restored: false, expired: false }; if ((await fs.readdir(cwd)).length === 0) await fs.rmdir(cwd); @@ -355,12 +374,7 @@ export async function restoreIsolatedTaskDirectory(input: { await fs.mkdir(cwd, { mode: 0o700 }); return { restored: false, expired: true }; } - const ciphertext = await readRestorePointObject(input.storage, input.workspace.companyId, ref.objectKey, input.signal).catch(() => { - throw new TaskDirectoryRestoreFailure("restore_point_unavailable"); - }); - if (ciphertext.length !== ref.byteSize) throw new TaskDirectoryRestoreFailure("ciphertext_size_mismatch"); - const payload = await openRestorePoint({ ciphertext, sha256: ref.sha256, key: ref.key, - identity: { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: ref.id, generation: ref.generation } }); + const payload = await loadTaskDirectoryRestorePoint({ ...input, ref }); const staging = await materializeTaskDirectorySnapshot({ payload, target: cwd, expectedDigest: ref.digest, signal: input.signal }); try { await fs.rename(staging, cwd); diff --git a/server/src/services/remote-workspace-migration-git.ts b/server/src/services/remote-workspace-migration-git.ts new file mode 100644 index 00000000000..66d82639c38 --- /dev/null +++ b/server/src/services/remote-workspace-migration-git.ts @@ -0,0 +1,142 @@ +import { execFile } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { and, desc, eq, gte, or, sql } from "drizzle-orm"; +import { executionWorkspaces, projectWorkspaces, workspaceOperations, type Db } from "@paperclipai/db"; +import { PROJECT_REPOSITORIES_DIR } from "@paperclipai/adapter-utils/git-workspace-sync"; +import type { StorageService } from "../storage/types.js"; +import { createIssueGitRemoteAuthProvider } from "./git-credentials.js"; +import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit, loadGitRestorePoint } from "./git-restore-point-storage.js"; +import { restoreGitWorkspaceContents } from "./git-workspace-contents.js"; +import { ensureSavedGitCommit, gitRestorePointExpired, managedSourcePaths, readGitCacheRestoration, + verifiedGitCacheSource, type GitCacheRestoration } from "./git-workspace-restoration.js"; +import { localRepositoryPath } from "./local-repository-path.js"; +import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js"; + +type Workspace = typeof executionWorkspaces.$inferSelect; +export type ArchivedGitMigrationSource = { + commit: string; + sourcePath: string; + cleanupOperationId: string | null; + recipe: GitCacheRestoration | null; +}; +const execute = promisify(execFile); +const unavailable = () => new Error("remote_workspace_migration_archived_git_source_unavailable"); +async function git(cwd: string, args: string[]) { + return (await execute("git", ["-c", "core.fsmonitor=false", "-c", `core.hooksPath=${os.devNull}`, "-C", cwd, ...args], { + timeout: 120_000, maxBuffer: 1024 * 1024, + env: { ...process.env, GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: os.devNull, + GIT_TERMINAL_PROMPT: "0", GIT_OPTIONAL_LOCKS: "0" }, + })).stdout.trim(); +} + +/** A removed branch must be reconstructed from its saved commit or its exact + * successful deletion receipt. Today's default branch is never a substitute. */ +export async function inspectArchivedGitMigrationSource(db: Pick, workspace: Workspace): Promise { + if (workspace.status !== "archived" || !workspace.repoUrl || !workspace.branchName || !workspace.sourceIssueId) return null; + const recipe = readGitCacheRestoration(workspace); + if (recipe) { + if (gitRestorePointExpired(recipe)) throw new Error("remote_workspace_migration_restore_point_expired"); + return { commit: recipe.commit, sourcePath: recipe.sourcePath, recipe, cleanupOperationId: null }; + } + if (workspace.providerType !== "git_worktree" || !workspace.closedAt || workspace.metadata?.gitBranchOwnershipVersion !== 1) return null; + const [operation] = await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), + eq(workspaceOperations.phase, "worktree_cleanup"), eq(workspaceOperations.status, "succeeded"), + or(eq(workspaceOperations.command, `git branch -d ${workspace.branchName}`), and( + sql`${workspaceOperations.metadata}->>'cleanupAction' = 'branch_delete'`, + sql`${workspaceOperations.metadata}->>'workspaceId' = ${workspace.id}`, + sql`${workspaceOperations.metadata}->>'branchName' = ${workspace.branchName}`, + sql`${workspaceOperations.metadata}->>'workspacePath' = ${workspace.providerRef ?? workspace.cwd}`, + )), gte(workspaceOperations.startedAt, workspace.closedAt), + )).orderBy(desc(workspaceOperations.startedAt), desc(workspaceOperations.id)).limit(1); + const receipt = operation?.stdoutExcerpt?.trim(), prefix = `Deleted branch ${workspace.branchName} (was `; + if (!operation || !receipt?.startsWith(prefix) || !receipt.endsWith(").")) return null; + const abbreviation = receipt.slice(prefix.length, -2); + if (!/^[a-f0-9]{7,40}$/.test(abbreviation)) throw unavailable(); + const [projectWorkspace] = workspace.projectWorkspaceId ? await db.select({ cwd: projectWorkspaces.cwd, repoUrl: projectWorkspaces.repoUrl }) + .from(projectWorkspaces).where(and(eq(projectWorkspaces.id, workspace.projectWorkspaceId), + eq(projectWorkspaces.companyId, workspace.companyId), workspace.projectId ? eq(projectWorkspaces.projectId, workspace.projectId) : undefined)) : []; + const sourcePath = projectWorkspace?.cwd ?? managedSourcePaths(workspace)[0]; + if (!sourcePath || projectWorkspace?.repoUrl && projectWorkspace.repoUrl !== workspace.repoUrl) throw unavailable(); + await verifiedGitCacheSource(workspace, sourcePath); + const commit = await git(sourcePath, ["rev-parse", "--verify", `${abbreviation}^{commit}`]); + if (!/^[a-f0-9]{40}$/.test(commit)) throw unavailable(); + return { commit, sourcePath, cleanupOperationId: operation.id, recipe: null }; +} + +/** Restore only into private migration scratch. Original repositories, task + * folders, lifecycle state and restore-point metadata remain unchanged. */ +export async function materializeArchivedGitMigrationSource(input: { + db: Db; storage: StorageService; workspace: Workspace; source: ArchivedGitMigrationSource; scratch: string; +}) { + const { workspace, source } = input, cwd = path.join(input.scratch, "retained-git"); + if (!workspace.sourceIssueId || !workspace.repoUrl || !workspace.branchName) throw unavailable(); + const resolveGitAuth = createIssueGitRemoteAuthProvider(input.db, workspace.companyId, workspace.sourceIssueId); + const present = await fs.lstat(source.sourcePath).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (present) { + const verified = await verifiedGitCacheSource(workspace, source.sourcePath); + await fs.mkdir(cwd, { mode: 0o700 }); await git(cwd, ["init", "--quiet"]); + // Fetch objects, never the current working tree of a shared project root. + const presentCommit = await git(verified, ["cat-file", "-e", `${source.commit}^{commit}`]).then(() => true, () => false); + if (presentCommit) { + await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, source.commit]); + } else if (source.recipe?.restorePoint && !source.recipe.restorePoint.headOnRemote) { + // The archived branch can have been pruned from the base while its + // encrypted bundle still owns the unpushed commits. Import only that + // bundle's explicit prerequisites, then let the restore-point owner + // verify and import the saved history below. + const { manifest } = await loadGitRestorePoint({ storage: input.storage, workspace, ref: source.recipe.restorePoint }); + for (const commit of manifest.prerequisites) { + const local = await git(verified, ["cat-file", "-e", `${commit}^{commit}`]).then(() => true, () => false); + if (local) await git(cwd, ["fetch", "--no-tags", "--no-recurse-submodules", "--", verified, commit]); + else await ensureSavedGitCommit(cwd, workspace.repoUrl, commit, resolveGitAuth); + } + } + } else { + if (!source.recipe) throw unavailable(); + await materializeManagedProjectWorkspace(cwd, { repoUrl: workspace.repoUrl, resolveGitAuth }); + } + if (source.recipe?.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage, workspace, + ref: source.recipe.restorePoint, repositoryCwd: cwd }); + await ensureSavedGitCommit(cwd, workspace.repoUrl, source.commit, resolveGitAuth); + await git(cwd, ["check-ref-format", `refs/heads/${workspace.branchName}`]); + await git(cwd, ["checkout", "--detach", source.commit]); + await git(cwd, ["update-ref", `refs/heads/${workspace.branchName}`, source.commit]); + await git(cwd, ["symbolic-ref", "HEAD", `refs/heads/${workspace.branchName}`]); + await git(cwd, ["config", "remote.origin.url", workspace.repoUrl]); + if (source.recipe?.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage, workspace, + ref: source.recipe.restorePoint, repositoryCwd: cwd, worktreePath: cwd }); + const approvedLocalSources = new Set(); + await restoreGitWorkspaceContents({ cwd, companyId: workspace.companyId, workspaceId: workspace.id, + projectId: workspace.projectId, contents: source.recipe?.contents, storage: input.storage, resolveGitAuth, + authorizeSource: async saved => { + if (!workspace.projectId) throw unavailable(); + const [row] = await input.db.select().from(projectWorkspaces).where(and(eq(projectWorkspaces.id, saved.id), + eq(projectWorkspaces.companyId, workspace.companyId), eq(projectWorkspaces.projectId, workspace.projectId))); + if (!row || row.repoUrl !== saved.repoUrl) throw unavailable(); + const local = localRepositoryPath(saved.repoUrl); if (local) approvedLocalSources.add(local); + }, + prepareLocalSource: async local => { + if (!approvedLocalSources.has(local)) throw unavailable(); + const identity = await fs.realpath(local); + return async () => { if (await fs.realpath(local) !== identity) throw unavailable(); }; + }, + }); + // A checkout records scratch-file inode and time fields in its index. Those + // fields must not change the saved archive identity after an upload reply is + // lost. Rebuild only these reconstructed indexes from their staged trees; + // the original repository and unstaged files remain untouched. + for (const repository of [cwd, ...(source.recipe?.contents?.checkouts ?? []) + .map(checkout => path.join(cwd, PROJECT_REPOSITORIES_DIR, checkout.name))]) { + const tree = await git(repository, ["write-tree"]); + await git(repository, ["read-tree", "--empty"]); + await git(repository, ["read-tree", tree]); + } + return cwd; +} diff --git a/server/src/services/remote-workspace-migration.test.ts b/server/src/services/remote-workspace-migration.test.ts index 0e55b224eab..e2345e3cb88 100644 --- a/server/src/services/remote-workspace-migration.test.ts +++ b/server/src/services/remote-workspace-migration.test.ts @@ -4,6 +4,8 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { promisify } from "node:util"; +import { createServer, type Server } from "node:http"; +import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { activityLog, applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots, @@ -16,7 +18,14 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js"; import { getWorkspaceSnapshot, loadWorkspaceSnapshotArchive } from "./remote-workspace-snapshot-storage.js"; import { restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; -import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { captureTaskDirectorySnapshot, removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { captureRestorePoint, sealRestorePoint } from "./git-restore-point.js"; +import type { TaskDirectoryRestorePointRef } from "./isolated-task-directory-retention.js"; +import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; +import { cleanupExecutionWorkspaceArtifacts } from "./workspace-runtime.js"; +import { workspaceOperationService } from "./workspace-operations.js"; +import { managedProjectCheckoutName } from "./managed-project-checkout-name.js"; +import { captureGitWorkspaceContents, readManagedCheckoutContents } from "./git-workspace-contents.js"; const execute = promisify(execFile); const git = async (cwd: string, ...args: string[]) => (await execute("git", ["-C", cwd, ...args], { env: { ...process.env, @@ -28,6 +37,7 @@ if (url && !/^remote_workspace_[a-z_]*test$/.test(new URL(url).pathname.slice(1) const support = url ? { supported: true } : await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("remote workspace migration", () => { let db: ReturnType, root: string, storage: StorageService; + const servers: Server[] = []; let temporary: Awaited> | undefined; beforeAll(async () => { if (url) { await applyPendingMigrations(url); db = createDb(url); } @@ -37,6 +47,7 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { beforeEach(async () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "61".repeat(32)); root = await fs.mkdtemp(path.join(os.tmpdir(), "pc-migration-test-")); storage = createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))); }); afterEach(async () => { + await Promise.all(servers.splice(0).map(server => new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())))); vi.unstubAllEnvs(); await db.delete(activityLog); await db.delete(workspaceOperations); await db.delete(executionWorkspaceSnapshots); await db.delete(workspaceRuntimeServices); await db.delete(executionWorkspaceRuntimeLeases); await db.delete(issues); await db.delete(executionWorkspaces); await db.delete(projectWorkspaces); await db.delete(projects); @@ -53,6 +64,256 @@ describe.skipIf(!support.supported)("remote workspace migration", () => { return { companyId, workspaceId, issueId: issue!.id, cwd }; } async function workspace(id: string) { return (await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, id)))[0]!; } + async function gitArchiveFixture(repoUrl = "https://example.invalid/repository.git") { + const f = await seed(), projectId = randomUUID(), projectWorkspaceId = randomUUID(), branchName = "saved-task"; + vi.stubEnv("PAPERCLIP_HOME", root); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "default"); + const base = resolveManagedProjectWorkspaceDir({ companyId: f.companyId, projectId, repoName: "repository" }); + await fs.mkdir(base, { recursive: true }); await git(base, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(base, "base.txt"), "base\n"); await git(base, "add", "."); await git(base, "commit", "-qm", "base"); + await git(base, "remote", "add", "origin", repoUrl); const upstream = await git(base, "rev-parse", "HEAD"); + const cwd = path.join(base, ".paperclip", "worktrees", branchName); + await git(base, "worktree", "add", "-b", branchName, cwd); + await fs.copyFile(path.join(f.cwd, "saved.txt"), path.join(cwd, "saved.txt")); await fs.rm(f.cwd, { recursive: true }); + await git(cwd, "add", "."); await git(cwd, "commit", "-qm", "saved task commit"); const commit = await git(cwd, "rev-parse", "HEAD"); + await db.insert(projects).values({ id: projectId, companyId: f.companyId, name: "Archived project" }); + await db.insert(projectWorkspaces).values({ id: projectWorkspaceId, companyId: f.companyId, projectId, name: "Source", cwd: base, repoUrl }); + await db.update(executionWorkspaces).set({ cwd, providerRef: cwd, projectId, projectWorkspaceId, repoUrl, baseRef: "main", + branchName, strategyType: "git_worktree", providerType: "git_worktree", metadata: { createdByRuntime: true, gitBranchOwnershipVersion: 1 } }).where(eq(executionWorkspaces.id, f.workspaceId)); + return { ...f, cwd, base, upstream, commit, projectId, projectWorkspaceId, repoUrl, branchName }; + } + it.each(["legacy", "verified", "retry"])("recovers an archived Git workspace from its exact branch-deletion receipt (%s)", async receiptKind => { + const f = await gitArchiveFixture(), closedAt = new Date(); + await db.update(executionWorkspaces).set({ status: "archived", closedAt }).where(eq(executionWorkspaces.id, f.workspaceId)); + if (receiptKind !== "verified") { + await git(f.base, "merge", "--ff-only", f.branchName); + await git(f.base, "worktree", "remove", f.cwd); + const receipt = await git(f.base, "branch", "-d", f.branchName); + await db.insert(workspaceOperations).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, + phase: "worktree_cleanup", status: "succeeded", command: `git branch -d ${f.branchName}`, + stdoutExcerpt: receipt, startedAt: new Date(), finishedAt: new Date() }); + } else { + const cleaned = await cleanupExecutionWorkspaceArtifacts({ workspace: await workspace(f.workspaceId), + projectWorkspace: { cwd: f.base, cleanupCommand: null }, expectedBranchHeadSha: f.commit, runCleanupCommands: false, + recorder: workspaceOperationService(db).createRecorder({ companyId: f.companyId, executionWorkspaceId: f.workspaceId, issueId: f.issueId }) }); + expect(cleaned.warnings).toEqual([]); + } + await fs.writeFile(path.join(f.base, "base.txt"), "newer default branch\n"); + await git(f.base, "add", "base.txt"); await git(f.base, "commit", "-qm", "new default branch work"); + if (receiptKind === "retry") { + const put = storage.putFile.bind(storage); let fail = true; + storage.putFile = async input => { const result = await put(input); if (fail) { fail = false; throw new Error("upload reply lost"); } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload reply lost"); + } + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const row = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(row.metadata)!; + expect(row.status).toBe("archived"); expect(authority.source).toMatchObject({ kind: "git", ref: f.commit }); + await fs.rm(f.base, { recursive: true }); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "legacy-git.pcws"), targetDir = path.join(root, "restored-legacy-git"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + expect(await fs.readFile(path.join(targetDir, "base.txt"), "utf8")).toBe("base\n"); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + async function serveRepository() { + const directory = path.join(root, "http"), bare = path.join(directory, "repository.git"); + await fs.mkdir(bare, { recursive: true }); await git(bare, "init", "--bare", "--initial-branch=main"); + const server = createServer(async (request, response) => { + const relative = new URL(request.url!, "http://localhost").pathname.slice(1); + const target = path.resolve(directory, relative); + if (!target.startsWith(`${directory}/`)) { response.writeHead(403).end(); return; } + try { response.end(await fs.readFile(target)); } catch { response.writeHead(404).end(); } + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); servers.push(server); + const address = server.address(); if (!address || typeof address === "string") throw new Error("test server not bound"); + return { bare, repoUrl: `http://127.0.0.1:${address.port}/repository.git` }; + } + it.each(["present", "pruned", "retry", "missing-base", "nested", "unapproved-nested", "missing", "expired", "corrupt"])("preserves archived Git edits or refuses an unavailable restore point (%s)", async state => { + const remote = state === "missing-base" ? await serveRepository() : null; + const f = await gitArchiveFixture(remote?.repoUrl); + if (remote) { await git(remote.bare, "fetch", f.base, "main:refs/heads/main"); await git(remote.bare, "update-server-info"); } + await fs.writeFile(path.join(f.cwd, "saved.txt"), "staged edit\n"); await git(f.cwd, "add", "saved.txt"); + await fs.writeFile(path.join(f.cwd, "saved.txt"), "unstaged edit\n"); + const captured = (await captureRestorePoint({ cwd: f.cwd, upstreamCommits: [f.upstream] }))!; + const id = randomUUID(), generation = 1, sealed = await sealRestorePoint(captured.payload, + { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: "git.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); + const restorePoint = { schema: "paperclip.git-restore-point.v1", id, generation, objectKey, byteSize: sealed.ciphertext.length, + sha256: sealed.sha256, key: sealed.key, head: captured.manifest.head, index: captured.manifest.index, + worktree: captured.manifest.worktree, ignored: captured.manifest.ignored, modes: captured.manifest.modes, headOnRemote: false }; + let contents; + if (state === "nested" || state === "unapproved-nested") { + const sourceCwd = path.join(root, "nested-source"); + await fs.mkdir(sourceCwd); await git(sourceCwd, "init", "-q", "-b", "main"); + await fs.writeFile(path.join(sourceCwd, "nested.txt"), "nested source work\n"); + await git(sourceCwd, "add", "."); await git(sourceCwd, "commit", "-qm", "nested source"); + const source = { id: randomUUID(), repoUrl: pathToFileURL(sourceCwd).href, repoRef: "main", cwd: sourceCwd }; + const name = managedProjectCheckoutName(source), nestedCwd = path.join(f.cwd, ".paperclip-repositories", name); + await git(f.cwd, "clone", source.repoUrl, nestedCwd); await git(nestedCwd, "checkout", "--detach"); + await fs.mkdir(path.join(nestedCwd, ".claude")); + await fs.writeFile(path.join(nestedCwd, ".claude", "settings.local.json"), '{"permissions":{"allow":["Read"]}}'); + contents = await captureGitWorkspaceContents(f.cwd, { companyId: f.companyId, workspaceId: f.workspaceId }, + [await readManagedCheckoutContents(nestedCwd, name, source)]); + if (state === "nested") await db.insert(projectWorkspaces).values({ ...source, companyId: f.companyId, projectId: f.projectId, name: "Nested source" }); + } + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { + createdByRuntime: true, gitBranchOwnershipVersion: 1, gitCacheRestoration: { schema: "paperclip.git-cache-restoration.v2", companyId: f.companyId, + workspaceId: f.workspaceId, sourceIssueId: f.issueId, repoUrl: f.repoUrl, commit: f.commit, branchName: f.branchName, + sourceKind: "managed_project", projectId: f.projectId, sourcePath: f.base, worktreePath: f.cwd, + gitBranchOwnershipVersion: 1, verifiedAt: new Date().toISOString(), restorePoint, contents }, + } }).where(eq(executionWorkspaces.id, f.workspaceId)); + await git(f.base, "worktree", "remove", "--force", f.cwd); await git(f.base, "branch", "-D", f.branchName); + if (state === "missing-base") await fs.rm(f.base, { recursive: true }); + if (state === "unapproved-nested") { + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect((await workspace(f.workspaceId)).status).toBe("archived"); + return; + } + if (state === "pruned") { + await git(f.base, "reflog", "expire", "--expire=now", "--all"); + await git(f.base, "gc", "--prune=now"); + await expect(git(f.base, "cat-file", "-e", `${f.commit}^{commit}`)).rejects.toThrow(); + } + if (["missing", "expired", "corrupt"].includes(state)) { + if (state === "missing") await storage.deleteObject(f.companyId, objectKey); + else { + const row = await workspace(f.workspaceId); + const recipe = row.metadata!.gitCacheRestoration as Record; + await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, gitCacheRestoration: { ...recipe, + restorePoint: { ...restorePoint, ...(state === "expired" ? { expiredAt: new Date().toISOString() } : { sha256: "00".repeat(32) }) }, + } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + } + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + const row = await workspace(f.workspaceId); + expect(row).toMatchObject({ status: "archived", providerType: "git_worktree", cwd: f.cwd }); + expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + return; + } + if (state === "retry") { + const put = storage.putFile.bind(storage); let fail = true; + storage.putFile = async input => { const result = await put(input); if (fail) { fail = false; throw new Error("upload reply lost"); } return result; }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("upload reply lost"); + } + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const authority = readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)!; + await fs.rm(f.base, { recursive: true, force: true }); await fs.rm(path.join(root, "nested-source"), { recursive: true, force: true }); await storage.deleteObject(f.companyId, objectKey); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "retained-git.pcws"), targetDir = path.join(root, "restored-retained-git"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await git(targetDir, "rev-parse", "HEAD")).toBe(f.commit); + if (state === "nested") { + expect(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, "nested.txt"), "utf8")).toBe("nested source work\n"); + expect(JSON.parse(await fs.readFile(path.join(targetDir, ".paperclip-repositories", contents!.checkouts[0]!.name, ".claude/settings.local.json"), "utf8"))) + .toEqual({ permissions: { allow: ["Read"] } }); + } + expect(await git(targetDir, "show", ":saved.txt")).toBe("staged edit"); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("unstaged edit\n"); + }); + async function retainDirectory(f: Awaited>) { + const captured = await captureTaskDirectorySnapshot(f.cwd), id = randomUUID(), generation = 1; + const sealed = await sealRestorePoint(captured.payload, { companyId: f.companyId, workspaceId: f.workspaceId, restorePointId: id, generation }); + const objectKey = `${f.companyId}/git-restore-points/${f.workspaceId}/${generation}/task-directory/${id}.pcrp`; + await storage.putFile({ companyId: f.companyId, namespace: "git-restore-points", objectKey, + originalFilename: "retained.pcrp", contentType: "application/octet-stream", body: sealed.ciphertext }); + const ref: TaskDirectoryRestorePointRef = { schema: "paperclip.task-directory-restore-point.v1", id, generation, + objectKey, sha256: sealed.sha256, byteSize: sealed.ciphertext.length, key: sealed.key, + digest: captured.digest, files: captured.files, excluded: captured.excluded }; + await db.update(executionWorkspaces).set({ status: "archived", closedAt: new Date(), metadata: { + isolatedTaskDirectoryRetention: { schema: "paperclip.task-directory-retention.v1", generation, removed: true, + trash: path.join(root, "removed-trash"), restorePoint: ref }, + } }).where(eq(executionWorkspaces.id, f.workspaceId)); + await fs.rm(f.cwd, { recursive: true }); + return ref; + } + it("imports an archived directory from its encrypted restore point after the old folder is gone", async () => { + const f = await seed(), retained = await retainDirectory(f); + expect(await inventoryRemoteWorkspaceMigration(db, f)).toMatchObject({ eligible: true }); + await migrateRemoteWorkspace({ db, storage, ...f, apply: true }); + const saved = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(saved.metadata)!; + expect(saved.status).toBe("archived"); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + await storage.deleteObject(f.companyId, retained.objectKey); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + const archivePath = path.join(root, "retained.pcws"), targetDir = path.join(root, "restored-retained"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + it.each(["missing", "expired", "corrupt"])("does not replace an unavailable retained directory with an empty workspace (%s)", async failure => { + const f = await seed(), retained = await retainDirectory(f); + if (failure === "missing") await storage.deleteObject(f.companyId, retained.objectKey); + else { + const row = await workspace(f.workspaceId); + const retention = row.metadata!.isolatedTaskDirectoryRetention as Record; + await db.update(executionWorkspaces).set({ metadata: { ...row.metadata, isolatedTaskDirectoryRetention: { + ...retention, restorePoint: { ...retained, ...(failure === "expired" ? { expiredAt: new Date().toISOString() } : { sha256: "00".repeat(32) }) }, + } } }).where(eq(executionWorkspaces.id, f.workspaceId)); + } + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow(); + const row = await workspace(f.workspaceId); + expect(row).toMatchObject({ status: "archived", providerType: "local_fs", cwd: f.cwd }); + expect(readRemoteWorkspaceAuthority(row.metadata)).toBeNull(); + expect(await fs.stat(f.cwd).catch(() => null)).toBeNull(); + }); + it("keeps newly restored local files when they appear during an archived import", async () => { + const f = await seed(); await retainDirectory(f); + const put = storage.putFile.bind(storage); + storage.putFile = async input => { + const result = await put(input); + await fs.mkdir(f.cwd); await fs.writeFile(path.join(f.cwd, "saved.txt"), "new local edits\n"); + return result; + }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("new local edits\n"); + }); + it("preserves an archived workspace without reopening its task or losing its files", async () => { + const f = await seed(), closedAt = new Date("2026-10-20T12:00:00Z"); + await db.update(executionWorkspaces).set({ status: "archived", closedAt }).where(eq(executionWorkspaces.id, f.workspaceId)); + await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const saved = await workspace(f.workspaceId), authority = readRemoteWorkspaceAuthority(saved.metadata)!; + expect(saved.status).toBe("archived"); expect(saved.closedAt).toEqual(closedAt); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(issue).toMatchObject({ status: "done", executionWorkspaceId: f.workspaceId }); + const ref = await getWorkspaceSnapshot(db, { ...f, snapshotId: authority.snapshotId! }); + await fs.rm(f.cwd, { recursive: true }); + const archivePath = path.join(root, "archived.pcws"), targetDir = path.join(root, "restored-archived"); + await loadWorkspaceSnapshotArchive({ storage, reference: ref!, targetPath: archivePath }); + await restoreRemoteWorkspaceArchive({ archivePath, targetDir, expectedManifestDigest: ref!.manifestDigest }); + expect(await fs.readFile(path.join(targetDir, "saved.txt"), "utf8")).toBe("original saved work\n"); + }); + it("imports a historical workspace without replacing the task's current workspace", async () => { + const f = await seed(), currentId = randomUUID(), currentCwd = path.join(root, "current"); + await fs.mkdir(currentCwd); await fs.writeFile(path.join(currentCwd, "saved.txt"), "newer current work\n"); + await db.insert(executionWorkspaces).values({ id: currentId, companyId: f.companyId, sourceIssueId: f.issueId, + mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Current workspace", cwd: currentCwd }); + await db.update(issues).set({ executionWorkspaceId: currentId }).where(eq(issues.id, f.issueId)); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).resolves.toMatchObject({ applied: true }); + const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(issue?.executionWorkspaceId).toBe(currentId); + expect(await fs.readFile(path.join(currentCwd, "saved.txt"), "utf8")).toBe("newer current work\n"); + expect(readRemoteWorkspaceAuthority((await workspace(currentId)).metadata)).toBeNull(); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)?.snapshotId).toBeTruthy(); + }); + it("refuses publication when the task changes its workspace during import", async () => { + const f = await seed(), put = storage.putFile.bind(storage); + storage.putFile = async input => { + const result = await put(input); + await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.id, f.issueId)); + return result; + }; + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + expect(readRemoteWorkspaceAuthority((await workspace(f.workspaceId)).metadata)).toBeNull(); + expect(await fs.readFile(path.join(f.cwd, "saved.txt"), "utf8")).toBe("original saved work\n"); + await expect(migrateRemoteWorkspace({ db, storage, ...f, apply: true })).rejects.toThrow("remote_workspace_migration_source_changed"); + }); it("defaults to dry run and preserves both database authority and local files", async () => { const f = await seed(); expect(await migrateRemoteWorkspace({ db, storage, ...f })).toMatchObject({ eligible: true, applied: false, alreadyMigrated: false }); expect((await workspace(f.workspaceId)).providerType).toBe("local_fs"); diff --git a/server/src/services/remote-workspace-migration.ts b/server/src/services/remote-workspace-migration.ts index b26dcfcf1b7..47dd4c7435f 100644 --- a/server/src/services/remote-workspace-migration.ts +++ b/server/src/services/remote-workspace-migration.ts @@ -14,7 +14,9 @@ import { withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifec import { captureRemoteWorkspaceDirectory, restoreRemoteWorkspaceArchive } from "./remote-workspace-snapshot-archive.js"; import { beginWorkspaceSnapshot, loadWorkspaceSnapshotArchive, uploadWorkspaceSnapshot } from "./remote-workspace-snapshot-storage.js"; import { notifyRemoteWorkspaceSnapshots } from "./remote-workspace-snapshot-retention.js"; -import { removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { materializeTaskDirectorySnapshot, removeTaskDirectoryTree } from "./isolated-task-directory-snapshot.js"; +import { loadTaskDirectoryRestorePoint, mayBeIsolatedTaskDirectoryWorkspace, readTaskDirectoryRetention, validateTaskDirectoryRestorePointRef } from "./isolated-task-directory-retention.js"; +import { inspectArchivedGitMigrationSource, materializeArchivedGitMigrationSource, type ArchivedGitMigrationSource } from "./remote-workspace-migration-git.js"; const SCHEMA = "paperclip.remote-workspace-migration.v1"; type Workspace = typeof executionWorkspaces.$inferSelect; @@ -32,6 +34,9 @@ function fingerprint(workspace: Workspace) { providerType: workspace.providerType, cwd: workspace.cwd, providerRef: workspace.providerRef, repoUrl: workspace.repoUrl, baseRef: workspace.baseRef, branchName: workspace.branchName, status: workspace.status, closedAt: workspace.closedAt?.toISOString(), metadata })).digest("hex"); } +const gitSourceProof = (source: ArchivedGitMigrationSource | null) => source ? { + commit: source.commit, sourcePath: source.sourcePath, cleanupOperationId: source.cleanupOperationId, +} : null; async function inspect(db: ReadDb, input: { companyId: string; workspaceId: string; operationId?: string }) { const [workspace] = await db.select().from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, input.companyId), eq(executionWorkspaces.id, input.workspaceId))); @@ -40,21 +45,43 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri const marker = object(workspace.metadata?.remoteWorkspaceMigration); const allowedOperation = input.operationId ?? (marker.schema === SCHEMA && typeof marker.operationId === "string" ? marker.operationId : undefined); const reasons: string[] = []; - if (authority) return { workspace, reasons, authority, marker, source: authority.source }; + if (authority) return { workspace, reasons, authority, marker, source: authority.source, issueBinding: null, retainedDirectory: null, archivedGitSource: null }; if (!["local_fs", "git_worktree"].includes(workspace.providerType)) reasons.push("workspace_provider_unsupported"); - if (!["active", "idle", "in_review"].includes(workspace.status)) reasons.push("workspace_not_active"); + if (!["active", "idle", "in_review", "archived"].includes(workspace.status)) reasons.push("workspace_not_active"); if (!workspace.sourceIssueId) reasons.push("task_binding_required"); const nestedMigration = workspace.metadata?.nestedCloneMigrationPending; if (nestedMigration != null && (typeof nestedMigration !== "object" || Array.isArray(nestedMigration) || Object.keys(nestedMigration).length > 0)) reasons.push("nested_clone_migration_unresolved"); const cwd = workspace.cwd ?? workspace.providerRef; + let retainedDirectory = null as ReturnType; + let archivedGitSource: ArchivedGitMigrationSource | null = null; if (!cwd || !path.isAbsolute(cwd)) reasons.push("local_source_missing"); else { - const entry = await fs.lstat(cwd).catch(() => null); - if (!entry?.isDirectory() || entry.isSymbolicLink()) reasons.push("local_source_unsafe_or_missing"); + const entry = await fs.lstat(cwd).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!entry && workspace.status === "archived" && mayBeIsolatedTaskDirectoryWorkspace(workspace)) { + const retained = readTaskDirectoryRetention(workspace); + if (retained?.removed === true && retained.restorePoint && retained.generation === retained.restorePoint.generation) { + if (retained.restorePoint.expiredAt) reasons.push("restore_point_expired"); + else { + try { validateTaskDirectoryRestorePointRef(retained.restorePoint, workspace); retainedDirectory = retained; } + catch { reasons.push("restore_point_invalid"); } + } + } + } + if (!entry && workspace.status === "archived" && !retainedDirectory) { + try { archivedGitSource = await inspectArchivedGitMigrationSource(db, workspace); } + catch (error) { reasons.push(error instanceof Error && error.message === "remote_workspace_migration_restore_point_expired" + ? "restore_point_expired" : "archived_git_source_unavailable"); } + } + if (!retainedDirectory && !archivedGitSource && (!entry?.isDirectory() || entry.isSymbolicLink())) reasons.push("local_source_unsafe_or_missing"); } const [issue] = workspace.sourceIssueId ? await db.select().from(issues).where(and(eq(issues.companyId, workspace.companyId), eq(issues.id, workspace.sourceIssueId))) : []; - if (workspace.sourceIssueId && (!issue || issue.executionWorkspaceId !== workspace.id)) reasons.push("task_binding_changed"); + // Historical records still own their saved files after the task selects a + // different workspace. Importing them must not reopen or rebind that task. + if (workspace.sourceIssueId && !issue) reasons.push("task_binding_changed"); const [project] = workspace.projectId ? await db.select({ policy: projects.executionWorkspacePolicy }).from(projects) .where(and(eq(projects.companyId, workspace.companyId), eq(projects.id, workspace.projectId))) : []; const issuePolicy = object(issue?.executionWorkspaceSettings), projectPolicy = object(project?.policy); @@ -109,10 +136,10 @@ async function inspect(db: ReadDb, input: { companyId: string; workspaceId: stri eq(executionWorkspaceRepositories.executionWorkspaceId, workspace.id))); if (repositories.some(repository => repository.state !== "ready")) reasons.push("repository_preparation_unresolved"); const source: RemoteWorkspaceSource = workspace.repoUrl ? { kind: "git", repoUrl: workspace.repoUrl, - ref: workspace.baseRef, branchName: workspace.branchName } : { kind: "empty" }; + ref: archivedGitSource?.commit ?? workspace.baseRef, branchName: workspace.branchName } : { kind: "empty" }; if (repositories.length) source.repositories = repositories.map(repository => ({ id: repository.id, relativePath: repository.relativePath, repoUrl: repository.repoUrl, ref: repository.requestedRef, ...(repository.pinnedCommit ? { pinnedCommit: repository.pinnedCommit } : {}) })); - return { workspace, reasons: [...new Set(reasons)], authority, marker, source }; + return { workspace, reasons: [...new Set(reasons)], authority, marker, source, issueBinding: issue?.executionWorkspaceId ?? null, retainedDirectory, archivedGitSource }; } export async function inventoryRemoteWorkspaceMigration(db: Db, input: { companyId: string; workspaceId: string }) { @@ -134,7 +161,7 @@ export async function migrateRemoteWorkspace(input: { if (!workspace) throw new Error("remote_workspace_migration_workspace_missing"); if (workspace.sourceIssueId) await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, workspace.companyId), eq(issues.id, workspace.sourceIssueId))).for("update"); const current = await inspect(tx as unknown as Db, input); - if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "" }; + if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null }; if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`); const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source); const sourceFingerprint = fingerprint(workspace); @@ -144,16 +171,26 @@ export async function migrateRemoteWorkspace(input: { const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration"))); if (!operation) throw new Error("remote_workspace_migration_operation_missing"); - return { workspace, authority: null, operation, source, sourceFingerprint }; + // Older operations only admitted the task's current workspace. Preserve + // that binding when resuming an import journaled before this field existed. + const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding") + ? operation.metadata!.sourceIssueBinding : workspace.id; + if (issueBinding !== current.issueBinding) throw new Error("remote_workspace_migration_source_changed"); + if (Boolean(operation.metadata?.retainedDirectory) !== Boolean(current.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed"); + if (canonical(operation.metadata?.archivedGitSource ?? null) !== canonical(gitSourceProof(current.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed"); + return { workspace, authority: null, operation, source, sourceFingerprint, issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource }; } const [operation] = await tx.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id, issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running", - metadata: { schema: SCHEMA, sourceFingerprint, source } }).returning(); + metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding, + ...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}), + ...(current.retainedDirectory ? { retainedDirectory: { restorePointId: current.retainedDirectory.restorePoint.id, + generation: current.retainedDirectory.generation, digest: current.retainedDirectory.restorePoint.digest } } : {}) } }).returning(); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); await tx.update(executionWorkspaces).set({ metadata: { ...workspace.metadata, remoteWorkspaceMigration: { schema: SCHEMA, operationId: operation!.id, state: "running", sourceFingerprint, source, } }, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id)); - return { workspace, authority: null, operation: operation!, source, sourceFingerprint }; + return { workspace, authority: null, operation: operation!, source, sourceFingerprint, issueBinding: current.issueBinding, retainedDirectory: current.retainedDirectory, archivedGitSource: current.archivedGitSource }; }); if (reserved.authority) return { ...await inventoryRemoteWorkspaceMigration(input.db, input), applied: true, replayed: true }; const operation = reserved.operation!; @@ -161,15 +198,21 @@ export async function migrateRemoteWorkspace(input: { operationId: operation.id, generation: 1, epoch: 1 }); const scratch = await fs.mkdtemp(path.join(os.tmpdir(), "pc-workspace-migration-")); try { + const retained = reserved.retainedDirectory?.restorePoint; + const sourceRoot = retained ? await materializeTaskDirectorySnapshot({ + payload: await loadTaskDirectoryRestorePoint({ storage: input.storage, workspace: reserved.workspace, ref: retained }), + target: path.join(scratch, "retained-source"), expectedDigest: retained.digest, + }) : reserved.archivedGitSource ? await materializeArchivedGitMigrationSource({ db: input.db, storage: input.storage, + workspace: reserved.workspace, source: reserved.archivedGitSource, scratch }) : reserved.workspace.cwd ?? reserved.workspace.providerRef!; const archivePath = path.join(scratch, "workspace.pcws"); - const captured = await captureRemoteWorkspaceDirectory({ cwd: reserved.workspace.cwd ?? reserved.workspace.providerRef!, archivePath }); + const captured = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot, archivePath }); const ref = await uploadWorkspaceSnapshot({ db: input.db, storage: input.storage, snapshotId: snapshot.id, companyId: input.companyId, workspaceId: input.workspaceId, archivePath, manifestDigest: captured.manifestDigest }); const restoredArchive = path.join(scratch, "verified.pcws"), restored = path.join(scratch, "restored"); await loadWorkspaceSnapshotArchive({ storage: input.storage, reference: ref, targetPath: restoredArchive }); await restoreRemoteWorkspaceArchive({ archivePath: restoredArchive, targetDir: restored, expectedManifestDigest: ref.manifestDigest }); const currentArchive = path.join(scratch, "current.pcws"); - const current = await captureRemoteWorkspaceDirectory({ cwd: reserved.workspace.cwd ?? reserved.workspace.providerRef!, archivePath: currentArchive }); + const current = await captureRemoteWorkspaceDirectory({ cwd: sourceRoot, archivePath: currentArchive }); if (current.manifestDigest !== captured.manifestDigest) throw new Error("remote_workspace_migration_source_changed"); await lock.execute(sql`select 1`); const publication = await input.db.transaction(async tx => { @@ -181,6 +224,9 @@ export async function migrateRemoteWorkspace(input: { || object(workspace.metadata?.remoteWorkspaceMigration).operationId !== operation.id) throw new Error("remote_workspace_migration_publication_changed"); if (workspace.sourceIssueId) await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, workspace.sourceIssueId)).for("update"); const validation = await inspect(tx as unknown as Db, { ...input, operationId: operation.id }); + if (validation.issueBinding !== reserved.issueBinding) throw new Error("remote_workspace_migration_source_changed"); + if (Boolean(validation.retainedDirectory) !== Boolean(reserved.retainedDirectory)) throw new Error("remote_workspace_migration_source_changed"); + if (canonical(gitSourceProof(validation.archivedGitSource)) !== canonical(gitSourceProof(reserved.archivedGitSource))) throw new Error("remote_workspace_migration_source_changed"); if (validation.reasons.length) throw new Error(`remote_workspace_migration_blocked:${validation.reasons.join(",")}`); const [uploaded] = await tx.select().from(executionWorkspaceSnapshots).where(and(eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.id, ref.id))).for("update"); From e491b23eb5d66b0cf81b0ae9988ef3c4d8788325 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:42:21 -0600 Subject: [PATCH 06/16] fix(runner): retire sandbox suspension receipts after admission (#18829) ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Native Runner sessions preserve a conversation across task turns. > - Remote suspension records a local receipt for the stopped Runner identity. > - The built-in sandbox path now writes the same receipt as plugin tasks. > - Admission cleared that receipt only for plugin tasks. > - A later warm turn could reject a clean journal because the old receipt named an earlier run. > - This pull request clears the receipt after successful admission for both sandbox paths. ## Linked Issues or Issue Description Refs #18343 and #18554 for the shared suspension receipt producer. Refs #18718 for the related controller projection refresh. **What happened?** After a suspended built-in sandbox session resumed, its next warm turn could fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Its journal named the completed resumed run. Its local suspension receipt still named the older run. **Expected behavior** Successful admission must retire the prior suspension receipt. The next compatible warm turn must reuse the admitted session and provider history once. A failed admission must retain recovery evidence. **Steps to reproduce** 1. Complete and suspend remote turn A. 2. Resume turn B through the built-in sandbox transport. 3. Keep the controller warm, with full Runner state owned by the sandbox. 4. Admit turn C. The old local receipt conflicts with B's journal identity. **Paperclip version or commit** Reproduced in the executor regression on base `367f29478d`. **Deployment mode** Native Runner with the built-in remote sandbox transport and a warm lifecycle. ## What Changed - Clear the local suspension receipt after a remote sandbox session passes provider recovery and durable admission. Apply this to built-in runners and plugin tasks. - Extend the existing three-turn warm admission regression to cover a suspended receipt and remote-only Runner state. - Keep an active-owner control that rejects admission and retains its receipt. Keep the existing foreign-owner, retirement, configuration, plugin, and unchanged-session controls. - Add a real Linux Runner continuity case that suspends A, resumes B, and keeps B warm for C. Register the test fixture's owned local API listener. - Document receipt ownership and the new continuity case. ## Verification - The new behavioral regression failed before the production change. Turn C raised the exact identity mismatch above. The active-owner control passed and retained its receipt. - All 1,087 native executor tests pass after the fix. - Both opt-in Linux process cases pass at the final commit: ordinary warm reuse and suspension followed by warm reuse. They use PostgreSQL, TLS, current-source Rust binaries, and a deterministic Codex fixture. Each preserves one conversation, starts exactly three turns, and reuses the warm Runner and provider on turn C. - Full repository typecheck passes with `node scripts/typecheck-low-memory.mjs`. Full production build passes with `pnpm build`. - All current-head CI checks pass at `a95a6fdfe8`, including the full test shards, browser shards, Rust checks, build, and typecheck. The policy job needed one retry after repository checkout timed out before checks ran. - Greptile and Ponytail each give 5/5 at the final head. There are no open review threads or findings. - The local full `pnpm test:run` attempt passed 351 cases in its first chat shard. The session interruption stopped it during the second shard, so it has no complete local verdict. The complete CI test run passes. - Module boundary checks and `git diff --check` pass. ## Risks - A receipt removed before admission could discard recovery evidence. Removal stays in the successful `onSession` callback. The negative admission control checks retention. - A stale owner must not gain authority. Journal, run, lease, and scope checks remain in place before the callback. - Historical contradictory or quarantined state still requires the existing verified recovery path. This change does not delete those records. - No migration or provider configuration change is required. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific model ID or context limit. Used reasoning, repository tools, code execution, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 5 ++-- doc/testing-native-remote-continuity.md | 4 +++ .../src/__tests__/helpers/native-restart.ts | 2 ++ ...unner-restart-recovery.integration.test.ts | 13 +++++---- .../native-session-executor.test.ts | 29 ++++++++++++++----- .../native-runtime/native-session-executor.ts | 3 +- 6 files changed, 41 insertions(+), 15 deletions(-) diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index d93ae631b6c..7d50d479125 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -1602,8 +1602,9 @@ agent, workspace, retained session, and termination of prior execution. The checkpoint must match the exact Runner, lease, run, session, turn, and item. A missing restart claim does not remove these checks or authorize a fresh session. -After a plugin session passes provider recovery and durable run admission, its -local suspension receipt is removed. The running session then owns authority; +After a remote sandbox session passes provider recovery and durable run admission, +its local suspension receipt is removed. This applies to built-in sandbox runners +and plugin tasks. The running session then owns authority; the old receipt must not conflict with a later warm run's journal identity. Failed admission retains the receipt. Normal suspension writes a new receipt. diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 65b00378d69..0f06c319e23 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -19,6 +19,10 @@ same Runner and provider history through another healthy follow-up. This is a controlled cache fault, not a reconstruction of a historical incident. Each recovery case includes another healthy turn after the recovered turn and asserts that it does not start another runner or provider process. +The `warm-suspended` case suspends the first turn, resumes the second turn, +then keeps that controller warm for the third turn. It must retire the first +turn's local suspension receipt so the third turn can reuse the resumed Runner +and provider history without a second process or repeated turn. The same-server case also starts preparation lease renewal during recovery and holds instruction collection open past a renewal tick after coordinator release. The executor must acquire renewal before preparation releases its reference, then diff --git a/server/src/__tests__/helpers/native-restart.ts b/server/src/__tests__/helpers/native-restart.ts index a05f5149f05..deccd592af1 100644 --- a/server/src/__tests__/helpers/native-restart.ts +++ b/server/src/__tests__/helpers/native-restart.ts @@ -16,6 +16,7 @@ import { prepareNativeHeartbeatRun } from "../../services/native-runtime/prepare import { createIsolatedTestHome } from "./isolated-test-home.js"; import { type NativeControllerIdentity } from "../../services/native-runtime/native-restart-recovery.js"; import { waitForPendingRunFailureReports } from "../../services/run-failure-report.js"; +import { registerLocalRuntimeApiServer } from "../../runtime-api.js"; // An explicitly supplied disposable database also permits qualification on hosts // where native Postgres cannot allocate semaphores. Never fall back silently. @@ -106,6 +107,7 @@ export function nativeRestartFixture() { : new Date(), }; server = createServer(); + registerLocalRuntimeApiServer(server); await new Promise((resolveListen) => server!.listen(0, "127.0.0.1", resolveListen), ); diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index aa23a2d35b0..b7d3403eb9a 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -150,12 +150,13 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( expect(next.kind).toBe("bootstrap_incomplete"); }); - it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "stale-projection", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { - const hardRestart = !["warm", "same-server"].includes(scenario); + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "warm-suspended", "stale-projection", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry", "diskless-warm-suspended", "diskless-warm-suspended-read-retry", "diskless-live-inspection"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { + const hardRestart = !["warm", "same-server", "warm-suspended"].includes(scenario); const diskless = scenario.startsWith("diskless"); const perTurn = scenario.startsWith("diskless-per-turn"); - const warmSuspended = scenario.startsWith("diskless-warm-suspended"); + const warmSuspended = scenario === "warm-suspended" || scenario.startsWith("diskless-warm-suspended"); const resumedAfterSuspension = perTurn || warmSuspended; + const suspendedBeforeThird = perTurn || (diskless && warmSuspended); const container = process.env.PAPERCLIP_TEST_RUNNER_CONTAINER!; const tunnel = await createRunnerTestTlsTunnel({ port: Number(process.env.PAPERCLIP_TEST_RUNNER_PORT), key: process.env.PAPERCLIP_TEST_RUNNER_TLS_KEY!, cert: process.env.NODE_EXTRA_CA_CERTS! }); @@ -310,6 +311,8 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }; const first = hardRestart ? await executeInController(firstOptions) : await executeFirst(); expect(first.exitCode).toBe(0); + if (scenario === "warm-suspended") expect(await closeIdleWarmNativeSessionsForRestart()) + .toEqual({ closed: 1, busy: 0, failed: 0 }); await finalizeNativeRun({ db: fixture.db, runId: fixture.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); const snapshot = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); expect(snapshot.exitCode).toBe(0); @@ -476,9 +479,9 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( const finalCalls = await runner.execute({ command: "cat", args: [`${remoteCwd}/provider-calls.log`] }); expect(finalCalls.stdout.split("\n").filter(line => line === "turn/start")).toHaveLength(3); expect(finalCalls.stdout.split("\n").filter(line => line === "thread/start")).toHaveLength(1); - expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (resumedAfterSuspension ? 1 : 0)); + expect(finalCalls.stdout.split("\n").filter(line => line === "process-start")).toHaveLength(providerStarts + (suspendedBeforeThird ? 1 : 0)); const finalProcess = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-process.identity`] }); - if (resumedAfterSuspension) expect(finalProcess.stdout).not.toBe(processAfter.stdout); + if (suspendedBeforeThird) expect(finalProcess.stdout).not.toBe(processAfter.stdout); else expect(finalProcess.stdout).toBe(processAfter.stdout); expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: perTurn ? 0 : 1, busy: 0, failed: 0 }); const suspended = await runner.execute({ command: "sh", args: ["-c", `cat '${remoteCwd}'/.paperclip-runtime/paperclip-runner/sessions/*/runner/runner-state.json`] }); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 725f07f2ea0..94fa60242b5 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -12967,9 +12967,10 @@ describe("runnerd provider runtime wiring", () => { }, ); - it.each((["codex", "claude"] as const).flatMap(provider => (["unchanged", "external-checkpoint", "changed", "runtime-context", "retirement-failed", "stale-owner", "active-owner", "missing-owner", "foreign-environment", "plugin-replaced", "plugin-managed-github"] as const).filter(scenario => provider === "codex" || scenario.startsWith("plugin-")).map(scenario => [provider, scenario] as const)))("rotates an owned warm %s remote runner before checking changed configuration: %s", async (provider, scenario) => { + it.each((["codex", "claude"] as const).flatMap(provider => (["unchanged", "external-checkpoint", "suspension-receipt", "suspension-active-owner", "changed", "runtime-context", "retirement-failed", "stale-owner", "active-owner", "missing-owner", "foreign-environment", "plugin-replaced", "plugin-managed-github"] as const).filter(scenario => provider === "codex" || scenario.startsWith("plugin-")).map(scenario => [provider, scenario] as const)))("rotates an owned warm %s remote runner before checking changed configuration: %s", async (provider, scenario) => { const plugin = scenario === "plugin-replaced" || scenario === "plugin-managed-github"; - const reusable = ["unchanged", "external-checkpoint", "plugin-managed-github"].includes(scenario); + const suspensionReceipt = scenario.startsWith("suspension-"); + const reusable = suspensionReceipt || ["unchanged", "external-checkpoint", "plugin-managed-github"].includes(scenario); const { permissionPolicy: _permissionPolicy, ...claudeProfile } = QUALIFIED_ACPX_PROFILES.claude; const first = { ...scopedNativeExecution(`warm-admission-${provider}-${scenario}`), schema: "paperclip.native-execution-input.v5" as const, @@ -12978,7 +12979,7 @@ describe("runnerd provider runtime wiring", () => { if (provider === "claude") first.session.driverKind = "acpx_runtime"; first.session.lifecyclePolicy = { mode: "warm", idleTimeoutMs: 60_000 }; const next = { ...first, binding: { ...first.binding, runId: `${first.binding.runId}-next` }, - session: { ...first.session, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: ["unchanged", "external-checkpoint", "runtime-context", "stale-owner", "missing-owner", "plugin-replaced", "plugin-managed-github"].includes(scenario) ? 60_000 : 90_000 } } }; + session: { ...first.session, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: suspensionReceipt || ["unchanged", "external-checkpoint", "runtime-context", "stale-owner", "missing-owner", "plugin-replaced", "plugin-managed-github"].includes(scenario) ? 60_000 : 90_000 } } }; if (scenario === "runtime-context") { next.runtimeContext = { ...first.runtimeContext, mcp: { ...first.runtimeContext.mcp, digest: "1".repeat(64) } }; @@ -13084,11 +13085,17 @@ describe("runnerd provider runtime wiring", () => { const ordinarySelect = nextDb.select.bind(nextDb); nextDb.select = ((fields?: Record) => fields?.status === heartbeatRuns.status && fields?.runnerProfileJson === heartbeatRuns.runnerProfileJson - ? priorNativeRunDb(scenario === "stale-owner" ? older : first, scenario === "active-owner" ? "running" : "succeeded").select(fields as never) + ? priorNativeRunDb(scenario === "stale-owner" ? older : first, ["active-owner", "suspension-active-owner"].includes(scenario) ? "running" : "succeeded").select(fields as never) : ordinarySelect(fields as never)) as Db["select"]; try { await executePaperclipNativeSession({ db: firstDb, execution: first, runnerInstanceId: identity.runnerInstanceId, useRunnerd: true, runnerExecutionTarget: pluginTarget as never,...pluginOptions }); + if (suspensionReceipt) { + // Remote suspension leaves a local identity receipt, while the Runner + // owns its full state in the sandbox. A resumed turn must retire it. + await writeFile(join(root, "plugin-runner-state.json"), JSON.stringify(durableRunnerState(identity, "suspended"))); + await rm(join(root, "runner/runner-state.json")); + } if (scenario === "stale-owner") await writeRunnerAuthority(root, { ...identity, runId: older.binding.runId }, "ready"); if (scenario === "missing-owner") { await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "fixture owner loss" }); @@ -13101,12 +13108,14 @@ describe("runnerd provider runtime wiring", () => { runnerInstanceId: `runner-next-${scenario}`, useRunnerd: true, runnerExecutionTarget: (scenario === "foreign-environment" ? { ...target, environmentId: `${environmentId}-other` } : plugin ? {...pluginTarget,leaseId:"next-core-lease"} : target) as never,...pluginOptions }); - if (["stale-owner", "active-owner", "missing-owner", "foreign-environment"].includes(scenario)) { - await expect(continuation).rejects.toThrow(scenario === "active-owner" + if (["stale-owner", "active-owner", "suspension-active-owner", "missing-owner", "foreign-environment"].includes(scenario)) { + await expect(continuation).rejects.toThrow(["active-owner", "suspension-active-owner"].includes(scenario) ? "prior_owner_active" : "prior_owner_terminal_state_indeterminate"); expect(state.execute).toHaveBeenCalledOnce(); expect(firstSession.close).not.toHaveBeenCalled(); expect(order).toEqual([]); + if (suspensionReceipt) expect(JSON.parse(await readFile(join(root, "plugin-runner-state.json"), "utf8"))) + .toMatchObject({ ...identity, lifecycle: "suspended" }); } else if (scenario === "retirement-failed") { await expect(continuation).rejects.toThrow("retirement unconfirmed"); expect(state.execute).toHaveBeenCalledOnce(); @@ -13120,8 +13129,9 @@ describe("runnerd provider runtime wiring", () => { } if (scenario === "plugin-managed-github") expect(rebind).toHaveBeenCalledOnce(); expect(order).toEqual(reusable ? ["execute-next"] : ["retire", "execute-next"]); - if (["unchanged", "external-checkpoint"].includes(scenario)) { + if (["unchanged", "external-checkpoint", "suspension-receipt"].includes(scenario)) { if (!persistedController) await writeRunnerAuthority(root, { ...identity, runId: next.binding.runId }, "ready"); + if (suspensionReceipt) await rm(join(root, "runner/runner-state.json")); const third = { ...next, binding: { ...next.binding, runId: `${next.binding.runId}-third` } }; const thirdDb = leaseDb(third); const thirdSelect = thirdDb.select.bind(thirdDb); @@ -13139,6 +13149,11 @@ describe("runnerd provider runtime wiring", () => { runnerExecutionTarget: pluginTarget as never,...pluginOptions })).resolves.toMatchObject({ exitCode: 0 }); expect(firstSession.close).not.toHaveBeenCalled(); expect(state.execute).toHaveBeenCalledTimes(3); + if (suspensionReceipt) { + expect(existsSync(join(root, "plugin-runner-state.json"))).toBe(false); + expect(JSON.parse(await readFile(join(root, "control-plane/control-plane-state.json"), "utf8")).identity) + .toMatchObject({ ...identity, runId: next.binding.runId }); + } if (restore) { expect(restore.mock.calls.map(([, binding]) => binding.runId)).toEqual([first.binding.runId, next.binding.runId]); expect(existsSync(root)).toBe(true); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c68407087e2..7264f55e6f1 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -9844,7 +9844,8 @@ async function executePaperclipNativeSessionWithinScope( ); }, onSession: async (session) => { - if (session && usesPluginTask) { + if (session && input.runnerExecutionTarget?.kind === "remote" + && input.runnerExecutionTarget.transport === "sandbox") { rmSync(resolve(scopedRunnerdStateRoot(input.execution), "plugin-runner-state.json"), { force: true }); } if (session && session === existingWarmSession) remoteWarmRecoveryPending = false; From f836fec59ebaa879010a1c92a4fb2761e97c5da5 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:48:56 -0600 Subject: [PATCH 07/16] perf(budgets): avoid company lock for routine eligibility checks (#18830) ## Thinking Path > - Paperclip helps people manage AI agents at work. > - The budget service checks whether an agent can start work. > - Preliminary checks acquired the company accounting lock even with no budget policy. > - Independent checks then waited behind accounting writes for the same company. > - This pull request reads one consistent snapshot before it decides whether reconciliation is needed. > - Healthy checks avoid the writer lock. State changes and final paid admission retain their existing authority. > - The benefit is less company-wide contention during routine eligibility checks. ## Linked Issues or Issue Description Refs #17697 (the preliminary-check portion). Refs #17817 (related serialization in agent configuration). Related approach: #17966. That PR adds a precheck for policies that cannot block. This PR also avoids the lock for healthy active hard-stop policies and manual pause reads. It uses a repeatable-read snapshot and leaves public transactional admission locked. It does not add the inactive-policy spend filtering proposed in #17966. **Current behavior** Every public `getInvocationBlock` call takes the company writer lock. Healthy checks can wait behind an unrelated receipt or settlement operation. **Proposed behavior** Read current scope state, policies, exposure and relevant incidents in a read-only snapshot. Return directly when no transition is needed. Otherwise discard the snapshot and reconcile current state under the company lock. **Reason and benefit** Independent preliminary checks can finish while another transaction owns the company row. Final reservations and paid-work admission still serialize against current accounting state. ## What Changed - Separate preliminary eligibility inspection from mutating reconciliation in the budget service. - Share scope loading, pause decisions and soft-incident resolution conditions between the two paths. - Keep no-policy checks free of exposure scans and incident reads. - Add inspection and reconciliation spans to the existing operator-gated Observability path. - Add PostgreSQL contention tests and regressions for pauses, holds, incidents, recovering native accounting and company boundaries. - Use the shared fixture startup timeout. Per-test deadlines and the 500 ms lock timeout stay intact. - Update the implementation contract and tracing documentation. Add dated investigation and future-design notes. The future accounting architecture remains a proposal. ## Verification - 73 targeted tests passed across eight files on implementation commit `244db565`. No tests were skipped. The files cover budget service, enforcement scoping and delivery, live receipt attempts, budget history, run settlement, heartbeat accounting and budget tracing. - Real PostgreSQL tests hold the company accounting lock while routine checks finish. The same reader's authoritative admission instead fails with PostgreSQL lock timeout `55P03`. - Full workspace typecheck passed before the rebase: `node scripts/typecheck-low-memory.mjs`. - Full workspace build passed before the rebase: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r --workspace-concurrency=1 build`. The Runner release binary was rebuilt from source. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Initial local setup failed before assertions: cold fixture setup exceeded 20 seconds, then a missing SDK build was exposed. The fixture now uses the shared 90-second startup allowance. The SDK was built before the successful test run. An incomplete local Copilot artifact also blocked the first build; the pinned installer repaired it before the successful canonical retry. - The branch rebased cleanly onto master `83b5f7d486`. All 73 targeted tests then passed again on clean head `244db565ae206502b100c3cdf5cd86a0ae91b372` (eight files, no skips). The full local Vitest suite was not run; remote CI must verify the wider suite. - Initial remote CI failed three monitor-scheduler cases at the suite's three-second heartbeat-idle wait. The unchanged suite then passed all 21 tests locally (`pnpm exec vitest run src/__tests__/issue-monitor-scheduler.test.ts`, 70.13 seconds). The one failed-job rerun was cancelled after the review follow-up was pushed; it produced no pass/fail result. No scheduler assertions or timeouts were changed. On updated head `ba2ab2a0`, the unchanged monitor suite passed all 21 tests in 38.381 seconds in [server shard 10/12](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126151059714). That shard passed 115 files and 2,677 tests, with no failures (one file and 20 tests skipped by existing suite configuration). - The independent review identified missing recovering-run coverage. The added contention case requires the locked fallback while another accounting writer holds the company row, then verifies the recovery block without pausing or cancelling recovery. The regression is in follow-up commit `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`. All 19 budget-service tests passed locally (no skips, 48.89 seconds), and `pnpm --filter @paperclipai/server typecheck` passed after the follow-up. - The updated CI run then hit a separate plugin-refresh fixture timeout at its final re-enable settle. The unchanged `external-object-refresh-work.test.ts` suite passed all 24 tests locally (52.02 seconds). Inspection identified a possible fake-clock race: its quiet helper can observe no scheduled deadline while reconstruction is still active, then encounter the existing 30-second reconstruction cooldown. The one same-head failed-job retry [passed](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126158227876): refresh suite 24/24, shard 115 files and 4,125 tests passed (one file and four tests skipped by existing configuration). No assertions, timeouts, or product code were changed. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41702687492) passed on exact head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: build, typecheck, Runner checks, all server/workspace suites and E2E. Both required aggregates are successful. All 59 checks are terminal: 54 successful, two advisory neutral and three expected skips. The PR is mergeable with a clean merge state. - Deployment latency and CPU effects have not been measured. - Greptile reviewed updated head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: 5/5, no actionable findings or unresolved Greptile threads. - Independent Paperclip review on the updated head: advisory 4/5, no blocker, recovery gap confirmed resolved. One canonical maintainability advisory remains visible and is documented below. ## Risks - Preliminary inspection can precede a concurrent policy or pause change. It grants no paid dispatch authority. The final locked admission and reservation checks remain required. - Healthy configured policies add an incident lookup. An unheld blocking scope can compute exposure during both inspection and locked revalidation. Already-held scopes skip the first exposure computation. - Holds and incidents must retain their existing transition effects. The fallback reads them again under the company lock, rather than mutating from the earlier snapshot. - Eligibility inspection and reconciliation still maintain separate transition lists. Future reconciliation changes must update the inspection predicate and its contention coverage together; consolidating the effects model remains a nonblocking review follow-up. - No schema or API response change is required. This change does not relax the current budget enforcement contract. ## Model Used OpenAI Codex with model `gpt-6.1-sol`. Used reasoning, repository inspection, code execution and delegated review. The session did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/SPEC-implementation.md | 5 +- doc/observability.md | 5 + ...nting-contention-and-budget-enforcement.md | 400 ++++++++++++++++ .../2026-11-03-lock-free-accounting-design.md | 432 ++++++++++++++++++ server/src/__tests__/budgets-service.test.ts | 136 +++++- server/src/services/budget-tracing.ts | 2 +- server/src/services/budgets.ts | 87 +++- 7 files changed, 1043 insertions(+), 24 deletions(-) create mode 100644 doc/plans/2026-11-03-accounting-contention-and-budget-enforcement.md create mode 100644 doc/plans/2026-11-03-lock-free-accounting-design.md diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index c9455c9de14..dd49412c784 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -1729,7 +1729,10 @@ Scheduler must skip invocation when: - an existing run is active - hard budget limit has been hit -Budget admission and accounting serialize under the company accounting lock. +Authoritative paid-work admission and accounting mutations serialize under the +company accounting lock. Preliminary invocation checks use a read-only snapshot +and take that lock only when a hold, incident, or enforcement transition needs +reconciliation; reconciliation rereads current state under the lock. The dispatch boundary rechecks current company, agent, and project policies and reserves capacity before paid work starts. Accounting commits budget blocks and durable stop intent without waiting for provider shutdown or reconciling unrelated diff --git a/doc/observability.md b/doc/observability.md index abadb0e6538..c04aef97125 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -45,10 +45,15 @@ Keep credential values out of logs and diagnostic reports. ## Budget admission, accounting, and enforcement The operator-gated `paperclip.budgets` tracer records +`budget.admission.inspect`, `budget.admission.reconcile`, `budget.admission.evaluate`, `budget.admission.reserve`, `budget.accounting.transaction`, `budget.accounting.settlement`, `budget.accounting.lock_wait`, `budget.exposure.evaluate`, `budget.enforcement.deliver`, `budget.enforcement.scope`, `budget.cancellation.deliver`, and `budget.accounting.recover`. +Inspection spans cover the preliminary read-only snapshot. Reconciliation spans +exist only when that inspection finds a transition requiring the company lock; +compare their counts to measure how often preliminary checks bypass the lock. +Authoritative reservation and transactional evaluation retain their own spans. Transaction spans include commit time. Lock-wait spans cover acquisition of the company accounting lock. Settlement spans cover the ledger and reservation transaction. Enforcement spans measure asynchronous delivery; cancellation diff --git a/doc/plans/2026-11-03-accounting-contention-and-budget-enforcement.md b/doc/plans/2026-11-03-accounting-contention-and-budget-enforcement.md new file mode 100644 index 00000000000..5ab838d2f97 --- /dev/null +++ b/doc/plans/2026-11-03-accounting-contention-and-budget-enforcement.md @@ -0,0 +1,400 @@ +# Accounting contention and best-effort budget enforcement + +Date: 2026-11-03 +Status: Investigation complete; first admission optimization implemented locally, remaining stages proposed +Source revision: `367f29478d14003cc5c0f161b7ce8f3164993634` + +Follow-up: [Lock-free accounting design](2026-11-03-lock-free-accounting-design.md) +defines the requested target architecture. Its append-only receipt path replaces +the run-lock proposal below; this document retains the investigation evidence +and the possible behavior-preserving first migration step. + +## Recommendation + +Keep receipt persistence, deduplication, final settlement, and audit transactional. +Make budget detection and automatic stopping a measured, short-lag control. Move +frequent receipt writes to transactions scoped to their run or paid operation. +Evaluate budget policies in coalesced batches outside those transactions. + +First remove avoidable admission transactions and measure the remaining critical +sections. Do not begin with distributed spending allocations or a new accounting +service. A short lock at an infrequent boundary can be acceptable; serializing +every company's receipt stream and admission preflight is the scaling concern. + +The first implementation now inspects preliminary eligibility in a repeatable-read, +read-only snapshot. Healthy checks and checks with no policies bypass the company +writer lock. Holds, hard blocks, recovering accounting and incidents needing +resolution fall back to the existing mutating owner, which revalidates current +state under the lock. Transactional admission, reservations and receipt accounting +retain their authority. Separate inspection/reconciliation spans expose the bypass +ratio. This does not implement the later append-only accounting architecture. + +Configured-policy tradeoffs: a healthy check adds an incident lookup, and an +unheld blocking/recovering scope can compute exposure once during inspection +and again during locked revalidation. Already-held scopes request reconciliation +before computing exposure. Qualify configured-policy workloads as well as the +predominant no-policy case; the change is not a claim of lower CPU everywhere. + +The first milestone's budget-service suite passes 18 tests against real +PostgreSQL. A controlled writer holds the company accounting lock while routine +checks finish; authoritative admission on the same reader instead reaches its +500 ms lock timeout. Coverage includes healthy/warning policies, manual and +orphaned pauses, hold release, incident cleanup and cross-company rejection. +This establishes the lock-bypass contract, not deployed latency or CPU savings. +Seven additional accounting/enforcement/replay/tracing suites pass 55 tests, +for 73 passing targeted tests across eight files with no skips. +The full workspace typecheck passes via `node scripts/typecheck-low-memory.mjs`, +and `node scripts/check-module-boundaries.mjs` passes. +The full workspace build passes via `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r +--workspace-concurrency=1 build`, including the current-source Runner release +binary, server, UI and CLI. The initial incomplete local Copilot build artifact +was repaired with the pinned installer before the successful canonical retry. + +This proposes a product-contract change for the later stages. It does not change +the current spec or silently redefine existing configured budgets. The first +implementation stage below preserves existing enforcement semantics. + +## What the investigation established on the source revision above + +### The lock protects several different responsibilities + +[`withAccountingTransaction`](../../server/src/services/accounting-transaction.ts) +takes `FOR NO KEY UPDATE` on the company row before invoking its callback. It +holds that lock until the transaction ends. Different companies can proceed +independently; writers for one company serialize. Ordinary report reads already +use a read-only repeatable-read snapshot and do not take this writer lock. + +| Path | Work sharing the company lock before this change | +| --- | --- | +| Usage receipt | Lock run, validate source/sequence and replay hash, insert immutable receipt, update running usage, evaluate affected budgets if monetary exposure changed | +| Admission | Read company/agent/project state and policies, calculate exposure, reconcile budget holds/incidents, reserve estimated capacity before dispatch | +| Final settlement | Recover durable receipt files, validate completion, settle reservation, insert ledger entries, update projections and runtime totals, acknowledge the run, evaluate budgets | +| Other mutations | Direct cost reports, billing imports/corrections, accounting repairs, paid browser/decision/fast-response/classifier operations, and agent lifecycle operations | + +The wrapper was unconditional. Having no configured budget did not bypass the +lock. The public `getInvocationBlock` method also used this mutating wrapper. +The implementation above separates its preliminary reads while preserving +the reconciliation and resume effects that some callers need. + +Primary owners are [usage receipts](../../server/src/services/usage-receipts.ts), +[budgets](../../server/src/services/budgets.ts), +[reservations](../../server/src/services/budget-reservations.ts), and +[run settlement](../../server/src/services/run-cost-accounting.ts). +Queue, dispatch, and lifecycle code should consume their contracts rather than +implement another budgeting system. + +### The current guarantee is already narrower than a dollar cap + +Reservations serialize admission against known exposure and estimated holds. +They do not restrict how much an already-dispatched provider can charge. +`reservationCents` defaults to zero. A zero reservation still prevents duplicate +dispatch, but does not reserve a positive allowance for future model spend. + +The [implementation spec](../SPEC-implementation.md) explicitly acknowledges that +mid-run enforcement depends on available provider reports or supported prices. +Providers reporting only at completion cannot provide exact mid-call detection. +Browser work can pass a provider cost cap; preserve such supported limits. + +The current rules are approximately: + +- Admission counts ledger spend plus the maximum of each unfinished operation's + reservation and observed spend, plus separate browser exposure. +- Threshold stopping uses ledger plus observed running spend, rather than + treating reservation estimates as money already spent. +- Pending terminal accounting and configured unknown-price restrictions can + independently block work. +- Final settlement replaces live exposure with the ledger charge atomically. + +Accurate recording of received evidence and instantaneous control of external +spend are separate contracts. Some prices remain estimates or unknown even when +their receipt is recorded correctly. + +### The critical section can do substantial work + +When applicable policies exist, [exposure evaluation](../../server/src/services/budget-exposure.ts) +executes four sequential aggregate statements over ledger entries, held +reservations joined to runs, pending/live runs, and browser work. Policies are +batched into those statements; it is not four statements per policy. With no +billable policies, these four queries are skipped. + +A changed monetary receipt can invoke this evaluation synchronously. Duplicate, +stale, unchanged, and subscription receipts take different branches; not every +receipt performs the aggregate scans. Native delivery already suppresses +accepted source-event replays in its receiver, so duplicate suppression is a +measurement question rather than an assumed large win. + +The hot inputs are usage and turn-lifecycle events, not every streamed text +token. Native capture handles usage reports and turn start/terminal events. +Legacy capture also durably spools before its database attempt. Recovery can +still read receipt files while the settlement transaction holds the company +lock, although it builds an initial file-identity index outside the lock. + +[Monthly spend projections](../../server/src/services/costs.ts) update +`companies.spentMonthlyCents` for each ledger insert, and the applicable agent +counter. Merely deleting the explicit company lock would leave an implicit +shared-row write bottleneck and break assumptions around settlement and repair. +PostgreSQL updates acquire conflicting row locks too. See the +[PostgreSQL locking contract](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS). + +### Live traces justify a focused optimization, not a CPU diagnosis + +Read-only observability queries examined one live host. The matched one-hour +sample is **2026-11-03 16:45–17:45 UTC**, service revision +`3910352c923afda320b2a5134e523932ee1d0633`. +The aggregate measurements are reproduced below. Private trace links and host +identifiers remain outside the repository. + +| Observed span | Count | p50 | p95 | p99 | Maximum | +| --- | ---: | ---: | ---: | ---: | ---: | +| Company lock acquisition | 9,421 | 13.00 ms | 206.85 ms | 725.14 ms | 4,301.53 ms | +| Accounting transaction | 9,421 | 87.16 ms | 486.95 ms | 1,147.10 ms | 5,594.19 ms | +| Admission evaluation | 8,082 | 43.45 ms | 166.84 ms | 342.13 ms | 1,212.92 ms | +| Final settlement | 216 | 355.90 ms | 1,894.05 ms | 3,149.67 ms | 5,594.33 ms | + +These are span counts; nested spans must not be added as independent operations. +There were 23 `budget.admission.reserve` spans in the same window. The much larger +admission-evaluation count warrants caller attribution, not an assumption that +all evaluations represent new paid dispatches. + +Of 8,292 exposure evaluations on that host in that hour, **8,280 had zero policies** +and 12 had one. That is 99.86% with zero policies at this call boundary; it does +not prove that every company has no budgets. These empty evaluations are cheap, +but their surrounding admission/accounting work still takes the company lock. + +One inspected admission transaction took 4.183 seconds, with 4.090 seconds in +lock acquisition. Its exposure child had zero policies and took about 0.026 ms. +A separate settlement transaction took 5.594 seconds, including 4.302 seconds +in acquisition and a zero-policy exposure evaluation. +Both inspected traces report mean sample rate 1. + +Limits of this evidence: + +- The acquisition span measures the complete awaited lock query. It includes + database/network round trips and Node scheduling; it is not a PostgreSQL + lock-wait measurement. Transaction duration also includes transaction setup + and commit. Do not subtract unrelated percentiles to estimate hold time. +- The spans do not identify the locking company, blocker, receipt frequency, + or operation caller. They do not establish one company's throughput ceiling + or the cause of the previously observed CPU utilization. +- The sample identifies one host. It does not establish fleet-wide behavior or + independently qualify the public instance serving each request. +- The local accounting wrapper, receipt, exposure, budget service, and run-cost + files match the traced revision in Git. Other deployed components and database + workload have not been qualified against this checkout. +- The existing [budget benchmark](../../scripts/benchmark-budget-enforcement.ts) + varies unrelated idle-agent count. It does not test concurrent receipt writers, + 10,000 active runs, or growing ledger history. It was inspected, not run here. + +## Proposed behavior + +| Responsibility | Proposed contract | +| --- | --- | +| Receipt acceptance | Durable commit before acknowledgement; source ownership, sequence, hash conflicts and company scope remain enforced | +| Ledger and settlement | Idempotent transaction scoped to the run/invocation; no lost charges, duplicate application, or transient live-plus-final double counting | +| Budget detection | Coalesced evaluation from committed evidence; a measurable delay is allowed | +| New work | Check current authoritative pause/Stop and known budget-block state at dispatch; concurrent admission against a not-yet-reflected spend increase can overshoot | +| Automatic cancellation | Commit durable stop intent, then use the existing scoped cancellation worker; provider shutdown remains a separate obligation | +| Manual Stop and authorization | Preserve current execution-authority checks; budget approximation does not weaken them | +| Unknown accounting | Retain unresolved obligations and existing configured unknown-usage restrictions; missing evidence is never a zero-cost receipt | + +Suggested initial service targets, to validate before adopting as promises: + +- Coalesce dirty budget work over approximately one second while work exists. +- p99 from committed priced receipt to durable threshold-stop intent at most + five seconds under the qualified workload. Measure provider reporting delay + and cancellation latency separately. +- For applicable spend limits, stop admitting additional paid work when the + oldest unevaluated accounting change exceeds fifteen seconds. This is a + proposed protection against a failed evaluator, not a hard spending cap. + An idle company's old evaluation timestamp alone must never trigger it. +- With no dirty changes, unresolved retries, or actual budget deadlines, the + evaluator performs no recurring database scans. + +Best effort needs a stated overspend model. As an illustration, 10,000 runs each +spending $0.001/second consume $10/second; five seconds of extra detection delay +adds about $50, before unreported and uncancellable in-flight work. Those are +hypothetical rates, not a measured cost estimate or maximum. + +In general, additional spend depends on fleet burn rate multiplied by reporting, +evaluation, and stop delay, plus already-committed work. Without bounds on those +quantities, do not advertise a bounded dollar overshoot. A strict end-to-end cap +would require provider-enforced limits or preauthorized bounded work; distributed +allocations alone cannot cap a provider that can exceed its allocation. + +## Implementation stages + +### 1. Attribute the cost and eliminate unnecessary admission transactions + +Owners: accounting transaction/tracing services, budget admission API, and its +queue/dispatch/lifecycle callers. + +Add a closed operation-kind label to accounting spans so receipt, finalization, +preflight, reservation, policy edit, correction, and lifecycle work can be +distinguished. Capture acquisition-query time, time from acquisition return to +transaction completion, query count, and replay/outcome categories. Keep the +operator-gated Observability contract and content-free attributes; this is not +first-party Telemetry. Label the client-observed held interval honestly because +it includes commit/return latency. Use sampled PostgreSQL wait/blocker evidence +to establish actual contention, with SQL values and credentials excluded. + +Separate read-only preliminary eligibility from mutating reconciliation. +Preflights should not take a company writer lock just to discover no applicable +policy. Keep the existing authoritative reservation and policy check at final +dispatch in this stage. Budget changes, rollover, settlement and explicit +reconciliation continue to own their necessary effects. Do not introduce a +stale cached "no budgets" result that can bypass a newly configured limit. + +Measure identical receipt replay and unchanged snapshots before prioritizing +their optimization. Any early replay return must prove the immutable receipt +and relevant projection state agree. Native event identity and lifecycle events +cannot be discarded just because their monetary amount is unchanged. + +Deliver a real-PostgreSQL benchmark using the service boundaries below, and the +small behavior-preserving optimization justified by the caller measurements. +If this removes today's significant delay, ship it independently; later stages +remain a deliberate scale/product decision. + +### 2. Persist receipts per run and evaluate budgets asynchronously + +Owners: usage receipts, native usage delivery, budget exposure, budgets, and the +existing accounting/enforcement delivery lifecycle. + +In one run-scoped transaction: + +1. Validate current source/ownership and lock the run. +2. Insert or verify the receipt with its unique identity and fingerprint. +3. Replace the run's accepted cumulative usage/exposure projection. +4. Persist a versioned pending-evaluation marker and register the normal + transaction-aware wake. +5. Commit before acknowledging native delivery or removing legacy spool evidence. + +Do not take the company accounting writer lock or evaluate all affected policies +inside this receipt transaction. Keep final and partial receipts durable; batch +derived budget evaluation rather than dropping intermediate accounting evidence. + +Use the shared delivery coordinator for coalescing, retry and deadlines. Pending +state must be scoped to the changed run/operation, or an append-only change; +updating one company dirty counter for every receipt would recreate the lock. +Consumption must compare revisions or claim durable work so a concurrent change +cannot be cleared by an older evaluator. A timestamp/sequence high-water mark +alone cannot safely assume database transactions commit in allocation order. + +Read ledger and live contributions in one consistent database snapshot, or use +transactionally consistent replacement contributions. The present four separate +read-committed aggregates are not safe to reuse unchanged once other operations +in the same company can settle concurrently: live-to-ledger +replacement must not be missed or counted twice across queries. + +Evaluate each affected policy once per batch, revalidate its revision before +committing block/resume state, and retain durable cancellation generations. +Updates arriving during evaluation remain pending. Recovery reconstructs pending +work after restart; rollback must not publish a false wake or acknowledgement. +Use short policy-state transactions where necessary. Preserve manual holds and +committed stop obligations during policy changes and price corrections. + +This is the first stage that permits an additional detection/admission race +window. Update the product contract together with implementation; removing only +the wrapper is not a valid intermediate change. + +### 3. Remove remaining company writes from frequent settlement + +Owners: run-cost accounting, costs, billing reconciliation/import, accounting +integrity, paid-service invocation owners, and DB/shared contracts. + +Settle ledger entries, reservation state, final receipt acknowledgement, and +run-specific accounting together under the operation's lock. Use database +uniqueness plus conflict/hash verification for concurrent idempotency; a +read-before-insert check previously protected by the company lock is insufficient. +Retain narrow agent coordination for shared runtime totals where needed. + +Move company monthly display-counter updates to versioned, coalesced projection +work. Update integrity inspection/repair and all counter consumers to understand +projection freshness and checkpoint identity. Reporting from ledger snapshots +remains available; intentional projection lag must not look like lost money. +Handle backdated events, credits/corrections, UTC rollover and lifetime budgets. + +For receipt-file recovery, stage immutable candidate envelopes outside the broad +transaction, then validate source/version and closure under the operation lock. +Retain disk evidence until the final owning transaction commits. A concurrent +late receipt or replaced source must still prevent premature acknowledgement. + +Apply the same boundary to browser runs, decision-model invocations, fast +responses, prompt-injection classification, and externally reported charges. +Inventory administrative users of `withAccountingTransaction` separately; +agent deletion, repair, and lifecycle safety must not depend on a lock removed +from receipt writers. Specify and test the new lock order, including issue/run +ownership and foreign-key interactions. Keep supported provider spending caps. + +Start with bounded batches and indexed aggregates. If history growth or batch +time misses the target, maintain incremental scope/window contributions. If +those counters become the next measured hot row, partition their writes and +sum a bounded set of partitions. Do not synchronously increment a single new +company total on every receipt and call the bottleneck solved. + +### 4. Qualify workload and roll out + +Extend the existing benchmark rather than using idle-agent count as scale proof. +Use real PostgreSQL with at least two independent controller processes and +explicitly recorded connection-pool limits. Simulated provider events should +exercise real receipt ingestion and settlement, followed by a smaller real +provider qualification. Keep load tests off the active staging instance. + +| Dimension | Required cases | +| --- | --- | +| Distribution | 100, 1,000, then 10,000 active runs; one company and many companies | +| Receipt cadence | One update per run per 60 seconds, 10 seconds and 1 second; record actual target-provider cadence before choosing the release load | +| Policies | None; company; agent/project only; overlapping scopes and windows | +| History | Increasing ledger history independently of active-run count | +| Event shape | Cost-changing, unchanged, subscription, duplicate, stale, conflicting and out-of-order reports; reconnect bursts | +| Races/failures | Simultaneous settlement and receipt, worker crash, failed/ambiguous commit, source replacement, policy change, manual Stop, late final charge, unknown price and UTC rollover | + +At 10,000 runs the illustrative receipt cadences imply about 167, 1,000, and +10,000 receipts/second before admission or settlement. This defines a workload, +not a claim that current or proposed code supports it. A shared critical section +held for an average of H seconds has an ideal ceiling near 1/H operations/second +for that company; do not estimate H from the unrelated percentile summaries. + +Acceptance requires durable receipt/ledger equality after quiescence, no duplicate +effect or acknowledgement loss, stable pending-work depth, correct policy/manual +hold behavior, and measured detection/cancellation/freshness targets. Record +p50/p95/p99 ingestion, admission and settlement latency, actual PostgreSQL wait +events, query count/rows/bytes, CPU/event-loop delay, and reconnect recovery time. +Prove unrelated companies remain responsive and writers for unrelated runs no +longer contend on a shared company update in the frequent path. + +Initially compute the proposed evaluation in shadow mode without stop/resume +effects and compare it against the authoritative ledger and existing decisions. +Then canary the new enforcement contract on an explicitly selected environment, +with its versioned pending work rebuilt and validated before switching. Do not +run old and new enforcement authorities simultaneously for the same company. +Rollback must drain/reconcile accepted receipts and rebuild authoritative state +before restoring the older path; flipping a flag alone is insufficient. + +Relevant existing coverage includes scoped budget enforcement, live attempts, +native usage delivery, run-cost accounting, budget cancellation, worker recovery, +and idle-accounting work. Add focused concurrency/failure tests for the changed +boundaries; run the repository's full checks before a PR-ready implementation +handoff. Browser qualification is needed if dispatch/Stop or budget UI semantics +change, not to validate this plan file. + +## Contract and delivery boundaries + +Before enabling the new behavior, update `doc/SPEC.md`, +`doc/SPEC-implementation.md` (especially scheduler and budget enforcement), the +applicable contributor invariants, shared budget contracts, and user-facing +budget explanations. The UI should say what auto-stop does and disclose possible +in-flight overspend; it should not expose internal partition mechanics. Decide +how configured existing reservations retain their estimate/visibility role +without promising atomic aggregate admission in the new mode. + +Distributed bounded spending allocations are deferred. Revisit them only if +customers need stricter aggregate admission or qualified batch evaluation cannot +meet latency/overspend needs. They require provider cost bounds as well as +replenishment, revocation and policy-version coordination to become a monetary +guarantee. They are not required for the recommended first implementation. + +Investigation validation: inspected the source and existing test/benchmark +contracts, queried live spans and two individual traces, and checked the relevant +deployed-revision source diff. No service, configuration, database, or deployed +code was changed. Runtime tests and load tests were not run for this planning +change; dependency installation and implementation belong to the next stage. diff --git a/doc/plans/2026-11-03-lock-free-accounting-design.md b/doc/plans/2026-11-03-lock-free-accounting-design.md new file mode 100644 index 00000000000..4c0ed915adc --- /dev/null +++ b/doc/plans/2026-11-03-lock-free-accounting-design.md @@ -0,0 +1,432 @@ +# Lock-free accounting and best-effort budgets + +Date: 2026-11-03 +Status: Target architecture proposed; preliminary admission optimization implemented locally +Scope: Accounting ingestion, settlement, budget projections and budget admission +Evidence: [Accounting contention investigation](2026-11-03-accounting-contention-and-budget-enforcement.md) + +## Decision + +Make the frequent accounting write path append-only. A receipt writes its own +immutable fact and commits. It does not lock or update the company, agent, run, +budget, aggregate counter, or worker checkpoint. Independent runs use independent +keys, including when they belong to the same company or agent. + +Asynchronous workers derive operation state, ledger entries and partitioned spend +totals. Budget admission reads a recent view without making a synchronized debit +against a company balance. Threshold detection and cancellation may lag; durable +evidence, deterministic accounting and execution authorization remain required. + +"Lock-free" here means no application mutex, advisory lock, or `SELECT FOR +UPDATE` in the receipt or budget-check path, and no shared accounting row written +for each receipt. It is not a formal nonblocking progress guarantee: PostgreSQL +still uses internal locks, unique-index arbitration, foreign-key checks and +transaction commit. In particular, two inserts with the same unique key can +wait for each other. See [PostgreSQL INSERT](https://www.postgresql.org/docs/current/sql-insert.html). + +Background batches and infrequent policy changes use optimistic conditional +writes. Those also take database row locks internally. Moving a compare-and-swap +onto the same company counter for every receipt would fail this design. + +Existing issue ownership, provider-dispatch fencing, manual Stop and resource +ownership are outside this lock-removal scope. Accounting cannot replace those +contracts with an eventually consistent spend view. + +## Problem fit and first milestone + +This design removes the identified company-wide accounting serialization point +if receipt ingestion, admission, settlement and shared counters all migrate. +It does not establish that accounting is the current main-loop CPU culprit or +that the resulting system supports 10,000 active agents. Those remain separate +claims requiring measurements. + +The matched investigation sample contained 8,280 zero-policy exposure evaluations +out of 8,292. Those exposure evaluations returned almost immediately; the +surrounding admission path still read state and policies inside the accounting +transaction. This makes separating eligibility reads from reconciliation the +most directly supported first change. It does not show that large exposure +scans or a high receipt rate caused the sampled delays. + +The 9,421 accounting transactions in that hour average about 2.6 per second +across the observed host. Burst rate, company distribution and actual lock hold +time are missing. The acquisition spans include database/network time and Node +scheduling, so their long tails alone do not prove PostgreSQL row-lock saturation. + +| Problem | What this design changes | Remaining condition | +| --- | --- | --- | +| Eligibility checks enter a mutating company transaction | Read-only admission, separate reconciliation | Classify callers and preserve manual holds, resume behavior and execution fencing | +| Independent receipts serialize by company | Independent immutable inserts | Qualify actual receipt rate, index/FK waits and durable ingestion capacity | +| Repeated exposure work and shared counters limit growth | Coalesced evaluation and partitioned projections | Show less work per receipt and sufficient projection throughput | +| Main-loop CPU/event-loop contention | Some synchronous work leaves the request path | Same-process reducers still consume CPU; batching must reduce work or processing must move to another runtime | + +The immutable identity model, closure changes and partition publication protocol +are target-architecture work, not prerequisites for the first latency improvement. +An asynchronous pipeline also adds backlog and recovery obligations: its stale +decision gate can become an admission outage if reducers cannot keep up. Merely +moving identical work behind an ACK would conceal latency without solving load. + +Start with an independently shippable milestone: attribute admission calls and +transaction work, provide a pure eligibility read (including a no-policy fast +path), and move required reconciliation to its explicit owner. Retain the current +final admission/reservation authority during this initial behavior-preserving +step. Having no configured policy must not bypass manual company/project pauses, +existing holds, unknown-accounting restrictions or duplicate-dispatch protection. + +Compare end-to-end dispatch latency, accounting transactions and queries per +dispatch, database waits, CPU and event-loop delay under the same workload. +Then qualify receipt coalescing and the remaining single-company bottleneck. +Use the full migration below when that evidence or the agreed capacity target +justifies it. The accounting rewrite does not cover the other independently +identified work in run-event persistence, recovery, snapshots or fanout. + +## Architecture + +```mermaid +flowchart LR + P["Runner / paid service"] --> I["Validate and append receipt"] + I --> J[("Immutable receipt journal")] + J --> A["ACK after durable commit"] + J --> R["Independent partition reducers"] + R --> L[("Ledger + operation views")] + R --> T[("Spend totals by scope / window / partition")] + T --> B["Coalesced budget evaluator"] + B --> V[("Versioned budget decisions")] + V --> D["Dispatch: read decision, record own admission"] + B --> S[("Durable stop intent")] + S --> C["Existing scoped cancellation path"] +``` + +The journal, reducers and evaluator can start in the existing server and +PostgreSQL deployment. Independent scaling is a module/runtime boundary, not a +requirement for Kafka, Redis, another service, or a separate database today. + +## 1. Give every paid operation a stable identity + +Use one accounting operation for each heartbeat run, browser invocation, +decision-model invocation, fast response, classifier invocation, or separately +reported charge. Record its company and attribution before paid dispatch. +Retries of the same external operation reuse its identity; a genuinely new paid +attempt has a distinct attempt identity beneath that operation. + +An accounting operation's immutable identity is separate from mutable run state. +Receipts reference this identity, rather than acquiring foreign-key protection +on a busy mutable `heartbeat_runs` row. The existing journal directly references +that row, so removing the explicit lock alone does not eliminate this possible +wait. Use a company-scoped composite reference to the immutable identity; keep +company-boundary validation and the existing deletion/retention obligations. +Separate any nullable live-run pointer from retained original attribution. + +Admitted accounting sources have durable identities and provenance. Controller +reconnection does not manufacture a new paid attempt. Source replacement must +retain the existing verified handoff/baseline semantics. A higher timestamp, +random UUID, or newly generated controller source ID is not proof of authority +over earlier billing evidence. + +Source handoff remains an infrequent execution-authority operation. Ordinary +checkpoints do not mutate its ownership record. Existing authentication and +company access checks still apply. Authority to report a closing receipt after +Stop is accounting-only; it never grants authority to continue execution. + +## 2. Receipt ingestion only appends evidence + +Extend/generalize the existing `run_usage_receipts` contract. The conceptual +receipt contains: + +| Field | Purpose | +| --- | --- | +| Company and operation IDs | Tenant boundary and accounting unit | +| Attempt and admitted source identity | Separate paid attempts from reconnect/replay | +| Stable source event ID and source sequence | Replay identity and ordering within that source | +| Receipt kind and normalization version | Partial usage, final usage, source handoff, closure, correction | +| Cumulative usage, price and provenance | Evidence; unknown price stays unknown | +| Payload fingerprint | Detect conflicting reuse of an identity | +| Server-assigned ingestion time and deterministic partition | Recovery, pending-work age and routing | + +Provider event time remains separate. Freshness protection never trusts a caller's +timestamp. External charge callers need a stable request identity to obtain +idempotent retries; matching dollar amounts or payload similarity cannot prove +that two otherwise unidentified charges are the same purchase. + +Within a small transaction, validate the admitted identity and input, register +the existing transaction-aware work signal, and insert the immutable receipt. +Enforce the logical identity with a unique constraint. On conflict, fetch the +committed existing receipt and compare its fingerprint; equal means replay, +different means a conflict requiring reconciliation. Do not use `DO UPDATE` +to replace receipt evidence. At read committed isolation, a conflict check may +need a subsequent statement to see the winning committed row. + +Commit before ACK. An ACK means the evidence is durable, not that the spend view +has already incorporated it. Preserve legacy durable spooling and native +redelivery when the database write or its acknowledgement is uncertain. + +There is no read-modify-write of `usageJson`, `spentMonthlyCents`, reservation +capacity, or a company-wide dirty flag here. The new receipt is itself durable +pending work. Mutable delivery metadata may be attached to it by the consumer; +its evidence fields are immutable. + +Batch inserts with count/byte bounds when useful. Keep lifecycle, final, unknown +and conflicting evidence intact. Do not drop a receipt solely because its dollar +amount matches the previous one: it may change completeness or provenance. + +## 3. Reduce independent partitions, then publish batches atomically + +Route by a stable hash of **operation ID**, not company ID. For example, a +starting benchmark can use 64 virtual partitions per company. That number is a +test parameter, not a capacity claim. One company with 10,000 active runs spreads +over all partitions; one agent with many runs spreads too. + +Persist the layout version and partition with the operation's admission. All its +receipts, closure and corrections follow that assignment. Changing partition +count requires an explicit migration; do not silently rehash active operations. + +The minimal conceptual storage is: + +| Record | Write frequency and ownership | +| --- | --- | +| Operation identity | Once per admitted operation; immutable attribution | +| Receipt journal | Append per receipt; unique identity; durable pending metadata | +| Operation view | Replaced by its partition reducer, at most once per operation per batch | +| Scope/window/partition totals | Updated in a bounded batch, never by receipt ingestion | +| Partition publication head | One conditional generation advance per batch | +| Budget policy and decision | Policy edits and coalesced evaluations; independent of receipt acceptance | +| Ledger, adjustments, incidents, stop intent | Existing domain records, adapted to operation identities and idempotent writes | + +Some records can extend existing tables. Separate immutable identities from +mutable projections; do not add a table for each timer or delivery stage. + +For a reducer batch: + +1. In one repeatable-read snapshot, capture partition generation `g`, a bounded + set of unprocessed receipt IDs, and their prior operation views. +2. Reduce those operations and compute replacement contributions outside the + publishing transaction. No provider calls or filesystem work occurs during + publication. +3. In one short transaction, conditionally advance the partition head from `g` + to `g + 1`, bulk-write operation views and aggregate deltas, insert any + qualified ledger settlement, and mark exactly the consumed receipt IDs. +4. If the conditional advance loses, roll back and retry from a fresh snapshot. + Use bounded retries with scheduler backoff under contention. + +All writers of those views, including repairs, follow this publication protocol. +A normally assigned reducer avoids duplicate work; overlapping workers or an +old worker after failover remain safe because publication requires the captured +generation. No durable exclusive lease is needed for correctness of this batch. + +An input arriving during computation remains pending. Never advance a global +`last_seen_id` or timestamp past all smaller values: sequence allocation and +transaction commit order differ. An earlier allocated receipt that commits +later must still be discoverable. Retain an indexed pending predicate. + +This has bounded coordination per partition **per batch**. It cannot stall +receipt ACKs behind the calculation, and different partitions do not write the +same aggregate rows. Shared company/agent/project totals are sums of these +partition contributions, not counters incremented by every producer. + +## 4. Define canonical money without depending on arrival order + +The reducer preserves current normalization and source-ownership rules as a +versioned pure function over admitted evidence. Within a source, order by the +producer sequence, not database arrival time. Across sources, follow verified +handoff/continuation provenance rather than arbitrarily choosing a winner. + +Cumulative reports replace earlier reports for the same accounting component. +Reports of $2, $3 and $5 represent $5, not $10. Distinct paid attempts contribute +separately; session-cumulative counters require their established baseline. +An incomplete/unknown report cannot erase known partial exposure. Final prices +and audited corrections may legitimately reduce earlier estimates. + +Each operation view owns mutually exclusive contribution categories: + +```text +unsettled: ledger = 0; live = known observed amount; hold = max(estimate, live) +settled: ledger = final; live = 0; hold = 0 +``` + +This is a contribution model, not a declaration that unknown exposure is zero. +Carry unknown-price, pending-terminal and unresolved-dispatch counts separately +and apply their configured restrictions. Hosted browser work remains a separate +paid operation from its parent run. + +Settlement and that operation's replacement contribution publish atomically. +Company/agent/project views sum one contribution per operation. A reader sees +the old or new contribution, never both the live estimate and its final charge. +UTC-month and lifetime attribution, unsettled carryover, credits, backdated +charges and corrections need explicit reducer rules matching the selected +product contract; do not assign costs to the month in which a worker ran. + +### Finalization requires closure, not merely a "complete" checkpoint + +The governing execution coordinator supplies immutable closure evidence only +after the operation cannot start another ordinary paid attempt. It references +the admitted attempts/sources and the final usage evidence or unresolved debt. +An adapter's `complete` usage flag alone does not establish that boundary. + +A closure record must prove the final cumulative coverage it names. The current +native `sourceSeq` includes non-accounting events, so gaps between usage receipts +are not automatically lost usage. Use explicit final references/coverage, or a +dedicated accounting sequence if the producer contract is extended. Do not +invent a contiguous usage sequence from today's sparse source sequence. + +Publish a settlement certificate and its ledger components in one transaction, +with one base settlement per `(company_id, operation_id)` and a verified content +fingerprint. A different closure for an already-settled operation is a conflict +or an explicit correction, not a second base settlement. +Only the transaction that inserts that certificate inserts its components; a +replay verifies the existing certificate. Ledger IDs and component keys are +stable across retries. Unknown external outcomes remain unresolved. + +Late evidence within the declared closure coverage is replay/recovery. Evidence +beyond it remains durable and creates a reconciliation obligation or an explicit +correction; it cannot silently reopen the same final charge or disappear because +the run was marked terminal. Corrections use immutable adjustment identities. +Existing mutable correction/reporting APIs need an adapter to this ledger model. + +## 5. Budget checks read; they do not reserve a shared balance + +For dispatch, read current company/agent/project controls and applicable budget +decisions in one bounded read from the authoritative database, rather than an +unbounded-lag replica or cache. A policy without a current decision must +be evaluated or treated as pending; it is not implicitly unlimited. The decision +records policy revision, budget window, projection generations, status and +freshness evidence. + +If applicable decisions allow work, record this operation's own admission with +the decision versions and estimated cost. Preserve duplicate-dispatch fencing +in the execution subsystem. An admission receipt is not proof that an external +provider call occurred exactly once; existing unknown-outcome recovery remains. + +There is no atomic decrement of company remaining budget. Estimate-fit checks +can use the same snapshot, and admitted estimates can enter the asynchronous +exposure view. Two dispatches can see the same remaining money and both start. +That race is accepted by this best-effort contract. Reservations become recorded +estimates/obligations, not exclusive claims on a shared spending allowance. + +New snapshots after a committed budget block deny additional work. Admissions +already in flight can still race that block. Manual Stop, pause, permissions, +issue ownership and dispatch authority retain their existing final gates. +Keep provider-enforced cost caps where supported. + +## 6. Coalesce budget decisions and preserve stop authority + +The evaluator reads the applicable scope/window partition totals in one +consistent snapshot, rather than scanning ledger and receipt history on each +checkpoint. Capture the policy's configuration revision and prior decision +revision in that same snapshot. + +Publish with a conditional update against both captured revisions. A policy +edit advances its configuration revision and invalidates the previous decision. +An older evaluator cannot overwrite a newer decision or a changed policy. A +losing evaluator recomputes; it never retries publication using a fresh version +number attached to stale calculated totals. + +Publish the decision, any incident/approval transition, and durable stop intent +in the same transaction. Generate idempotency identities from the winning +transition; losing or repeated evaluations produce no duplicate side effects. +Only actual state transitions need notification/audit effects. Cancellation +delivery remains at least once, scoped and recoverable. Already-committed +per-run Stop obligations survive later policy changes. Resuming a budget hold +cannot clear a manual hold. + +These occasional policy-row updates can contend with one another. Producers +and admission readers do not wait on that row to record receipts or debit money. +Do not replace them with per-receipt policy updates. + +## 7. Bound lag and recover work without an idle polling loop + +Retain the proposed healthy-load targets from the investigation: roughly +one-second coalescing and a five-second p99 target from durable priced receipt +to stop intent. Provider reporting and shutdown delay are additional. Targets +require a workload qualification; none is a guarantee on final billed dollars. + +For scopes with spending limits, reject additional paid admissions if an +unrepresented accounting change has been waiting over fifteen seconds. This +check considers pending journal inputs **and** reduced contributions not yet +covered by the decision's projection generations. Carry their oldest pending +time through publication. Consuming the journal must not hide evaluator lag. +An old decision with no subsequent relevant changes remains fresh while idle. +Policy edits and UTC-window changes are explicit new evaluation obligations. + +Durable pending rows are truth; notifications are hints. Reuse the existing +delivery coordinator for local coalescing, finite retries, actual deadlines, +drain and restart. No unconditional per-company timer scans empty queues. + +The current DB work signals are process-local by default, with optional +commit-bound PostgreSQL notifications. Separate reducers therefore require +qualified cross-process wake delivery, subscribed before the startup scan and +reconciled after reconnect. `NOTIFY` alone is not durable storage or a way to +wake stopped compute. See the [existing signal implementation](../../packages/db/src/work-signals.ts) +and [PostgreSQL notification behavior](https://www.postgresql.org/docs/current/sql-notify.html). +Do not claim distributed recovery or host sleep until the actual notification, +startup, disconnect and external-wake paths are qualified. + +Receipt acceptance continues while budgets are blocked. Otherwise stopping spend +would also stop the evidence needed to settle it. + +## Failure cases the design must pass + +| Event/race | Required outcome | +| --- | --- | +| Same receipt delivered twice | One logical receipt; equal fingerprint accepted as replay | +| Same identity, different content | Conflict retained/reported; no silent overwrite or second charge | +| Sequence 8 arrives before 7 | Correct cumulative state; 7 cannot rewind it | +| Earlier allocated receipt commits after a batch | It remains discoverable as pending work | +| Producer loses ACK after commit | Safe redelivery with the same identity | +| Reducer dies before or during publication | All batch effects commit together or remain replayable | +| Two reducers compute from generation 12 | One publishes generation 13; the other recomputes | +| Final receipt races partial receipt | Closure and canonical reduction determine one contribution, independent of arrival order | +| Final bill is lower than live estimate | Correct signed adjustment/replacement; no double counting | +| Source changes during recovery | Verified handoff retains prior debt; no guessed new baseline | +| Policy is lowered during evaluation | Old policy revision cannot publish a decision | +| Manual Stop races budget resume | Manual execution hold remains authoritative | +| Evaluator stops while receipts continue | Evidence accumulates durably; stale-view admission protection activates | +| No new accounting work | Reducers and evaluator become quiet | + +## Implementation and migration + +These steps implement the full target after the first milestone above; they are +not required as one change to improve today's eligibility path. + +1. Add immutable operation identities and the journal acceptance boundary, + initially alongside the existing authoritative path. Preserve existing native + and legacy receipts; adapt source/closure evidence before removing their + current completion fences. +2. Implement deterministic operation reduction, idempotent settlement and + partition publication. Run it in shadow mode with isolated projections and + no production ledger, stop, or incident side effects. Compare at a common + receipt/closure boundary after draining, not at arbitrary wall times. +3. Implement versioned budget decisions, freshness, read-only admission and + transition-based stop delivery. Replace shared monthly/runtime counters with + partitioned projections and update all report, repair and integrity consumers. +4. At a controlled cutover, fence the old settlement/evaluation authority, drain + committed inputs, seed verified ledger/adjustment baselines, and select one + authoritative writer path. Resume admission only with current decisions. + Continue accepting/replaying durable receipts through this transition. +5. Rollback similarly requires reconciliation and an authority handoff. Do not + let old and new settlement writers charge the same operation independently. + +Owning modules are usage receipts/native usage delivery, run-cost accounting, +costs/reconciliation/integrity, budgets and the existing enforcement worker. +Dispatch consumes a narrow decision contract; provider normalization remains +with adapters. Update DB/shared/API/UI contracts and the budget guarantees in +`doc/SPEC.md`, `doc/SPEC-implementation.md` and contributor guidance before enabling +the change. The earlier per-run-lock design is not this target's hot path. + +Qualification uses real PostgreSQL and multiple independent processes, both one +large company and many companies, overlapping scope budgets, ledger-history +growth, duplicate/reconnect bursts and the races above. Verify producer SQL has +no shared counter/head/policy writes and no company/run `FOR UPDATE`. Audit +foreign-key, trigger and uniqueness behavior too; string searches cannot prove +absence of contention. + +Measure sustained receipts and bytes per second, ACK/admission/settlement latency, +projection backlog age, stop latency, database waits and event-loop load. Ten +thousand agents at one receipt per second would create 864 million receipts per +day before batching/retention effects. Eliminating serialization does not remove +database, WAL, index, storage or provider-capacity limits. Start with bounded +batches and measured retention; introduce a different log substrate only when +its need is demonstrated. + +Design validation in this chat: inspected the existing journal, budget incident, +enforcement and work-signal contracts and checked PostgreSQL concurrency and +notification semantics. This is an architecture document, not a prototype or +load-test result. Implementation details and capacity remain to be qualified. diff --git a/server/src/__tests__/budgets-service.test.ts b/server/src/__tests__/budgets-service.test.ts index 1b2495cf7b6..0d1580064ef 100644 --- a/server/src/__tests__/budgets-service.test.ts +++ b/server/src/__tests__/budgets-service.test.ts @@ -1,7 +1,7 @@ import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { randomUUID } from "node:crypto"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { agents, @@ -11,10 +11,14 @@ import { companies, costEvents, createDb, + heartbeatRuns, + issues, + nativeRunFinalizations, projects, } from "@paperclipai/db"; import { budgetService, deliverBudgetEnforcement, hasPendingBudgetEnforcement } from "../services/budgets.ts"; import { + EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; @@ -40,13 +44,16 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => { await import("../services/agent-lifecycle.js"); if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-budgets-service-"); db = createDb(externalUrl ?? tempDb!.connectionString); - }, 20_000); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterEach(async () => { await db.delete(budgetIncidents); await db.delete(approvals); await db.delete(budgetPolicies); await db.delete(costEvents); + await db.delete(nativeRunFinalizations); + await db.delete(heartbeatRuns); + await db.delete(issues); await db.delete(projects); await db.delete(agents); await db.delete(companies); @@ -118,6 +125,131 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => { return event!; } + it.each(["no policy", "healthy policy", "warning policy", "manual company pause", "manual project pause", "orphan budget pause", "recovering run"])( + "checks %s while another accounting writer holds the company lock", + async state => { + const { companyId, agentId, projectId } = await createBudgetFixture(); + if (state === "healthy policy" || state === "warning policy" || state === "recovering run") { + const [policy] = await db.insert(budgetPolicies).values({ companyId, scopeType: "company", scopeId: companyId, + windowKind: "calendar_month_utc", amount: 100 }).returning(); + if (state !== "recovering run") await insertCostEvent({ companyId, agentId, costCents: state === "warning policy" ? 80 : 20 }); + if (state === "warning policy") { + const now = new Date(); + await db.insert(budgetIncidents).values({ companyId, policyId: policy.id, scopeType: "company", scopeId: companyId, + metric: "billed_cents", windowKind: "calendar_month_utc", + windowStart: new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 1)), + windowEnd: new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() + 1, 1)), + thresholdType: "soft", amountLimit: 100, amountObserved: 80 }); + } + } + if (state === "recovering run") { + const [issue] = await db.insert(issues).values({ companyId, projectId, title: "Recover accounting" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, scopeKind: "issue", issueId: issue.id, + nativeIssueId: issue.id, runtimeMode: "native", status: "failed", finishedAt: new Date(), costAccountingPending: true }).returning(); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId: run.id, phase: "observed" }); + } + if (state === "manual company pause" || state === "orphan budget pause") { + await db.update(companies).set({ status: "paused", pauseReason: state === "orphan budget pause" ? "budget" : "manual" }) + .where(eq(companies.id, companyId)); + } + if (state === "manual project pause") { + await db.update(projects).set({ pausedAt: new Date(), pauseReason: "manual" }).where(eq(projects.id, projectId)); + } + + const reader = createDb(externalUrl ?? tempDb!.connectionString, { maxConnections: 1 }); + let acquired!: () => void, release!: () => void; + const lockAcquired = new Promise(resolve => { acquired = resolve; }); + const holdLock = new Promise(resolve => { release = resolve; }); + const writer = db.transaction(async tx => { + await tx.select({ id: companies.id }).from(companies).where(eq(companies.id, companyId)).for("no key update"); + acquired(); + await holdLock; + }); + try { + await Promise.race([lockAcquired, writer]); + await reader.execute(sql`set lock_timeout = '500ms'`); + if (state === "recovering run") { + await expect(budgetService(reader).getInvocationBlock(companyId, agentId, { projectId })) + .rejects.toMatchObject({ cause: { code: "55P03" } }); + } else { + const block = await budgetService(reader).getInvocationBlock(companyId, agentId, { projectId }); + if (state === "manual company pause") { + expect(block).toMatchObject({ scopeType: "company", scopeId: companyId, + reason: "Company is paused and cannot start new work." }); + } else if (state === "manual project pause") { + expect(block).toMatchObject({ scopeType: "project", scopeId: projectId, + reason: "Project is paused and cannot start new work." }); + } else if (state === "orphan budget pause") { + expect(block).toMatchObject({ scopeType: "company", scopeId: companyId, + reason: "Budget pause requires a policy or an explicit operator resume." }); + } else expect(block).toBeNull(); + if (state === "warning policy") { + expect(await reader.select().from(budgetIncidents).where(eq(budgetIncidents.companyId, companyId))) + .toMatchObject([{ thresholdType: "soft", status: "open", resolvedAt: null }]); + } + if (state === "no policy") { + await expect(budgetService(reader).evaluateAdmission(companyId, agentId, { projectId })) + .rejects.toMatchObject({ cause: { code: "55P03" } }); + } + } + } finally { + release(); + try { await writer; } + finally { await reader.$client.end({ timeout: 1 }); } + } + if (state === "recovering run") { + const cancelWorkForScope = vi.fn(); + expect(await budgetService(db, { cancelWorkForScope }).getInvocationBlock(companyId, agentId, { projectId })) + .toMatchObject({ scopeType: "company", scopeId: companyId, + reason: "New work must wait while a native run recovers its unfinished accounting." }); + expect(cancelWorkForScope).not.toHaveBeenCalled(); + expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0]) + .toMatchObject({ status: "active", pauseReason: null }); + expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]) + .toMatchObject({ status: "active", pauseReason: null }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))) + .toMatchObject([{ status: "failed", costAccountingPending: true }]); + expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId))) + .toMatchObject([{ phase: "observed", resultId: null }]); + } + }, + ); + + it.each([true, false])("releases an agent budget hold when its policy is healthy (active: %s)", async isActive => { + const { companyId, agentId } = await createBudgetFixture(); + await db.insert(budgetPolicies).values({ companyId, scopeType: "agent", scopeId: agentId, + windowKind: "calendar_month_utc", amount: 100, isActive }); + await db.update(agents).set({ status: "paused", lifecycleState: "paused", pauseReason: "budget", pausedAt: new Date(), lifecycleHolds: ["budget"] }) + .where(eq(agents.id, agentId)); + + expect(await budgetService(db).getInvocationBlock(companyId, agentId)).toBeNull(); + expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]).toMatchObject({ + lifecycleState: "resuming", pauseReason: null, lifecycleHolds: [], + }); + }); + + it.each(["hard", "soft"])("resolves an obsolete %s incident even when the scope no longer has a budget hold", async thresholdType => { + const { companyId, agentId } = await createBudgetFixture(); + const [policy] = await db.insert(budgetPolicies).values({ companyId, scopeType: "agent", scopeId: agentId, + windowKind: "calendar_month_utc", amount: 100 }).returning(); + const [incident] = await db.insert(budgetIncidents).values({ companyId, policyId: policy.id, scopeType: "agent", scopeId: agentId, + metric: "billed_cents", windowKind: "calendar_month_utc", windowStart: new Date(0), windowEnd: new Date(1), + thresholdType, amountLimit: 100, amountObserved: 100 }).returning(); + + expect(await budgetService(db).getInvocationBlock(companyId, agentId)).toBeNull(); + expect((await db.select().from(budgetIncidents).where(eq(budgetIncidents.id, incident.id)))[0]) + .toMatchObject({ status: "resolved", resolvedAt: expect.any(Date) }); + }); + + it("rejects another company's agent or project before allowing a no-policy invocation", async () => { + const own = await createBudgetFixture(); + const other = await createBudgetFixture(); + const service = budgetService(db); + await expect(service.getInvocationBlock(own.companyId, other.agentId)).rejects.toThrow("Budget scope not found"); + await expect(service.getInvocationBlock(own.companyId, own.agentId, { projectId: other.projectId })) + .rejects.toThrow("Budget scope not found"); + }); + it("raises one soft incident per window before hard-stopping and preserving safe local audit details", async () => { const { companyId, agentId } = await createBudgetFixture(); const cancelWorkForScope = vi.fn().mockResolvedValue(undefined); diff --git a/server/src/services/budget-tracing.ts b/server/src/services/budget-tracing.ts index f4b9212ca8f..535ef5617e1 100644 --- a/server/src/services/budget-tracing.ts +++ b/server/src/services/budget-tracing.ts @@ -3,7 +3,7 @@ import { withDatabaseOperation } from "@paperclipai/db"; type BudgetOperation = "budget.accounting.lock_wait" | "budget.accounting.recover" | "budget.accounting.settlement" | "budget.accounting.transaction" - | "budget.admission.evaluate" | "budget.admission.reserve" + | "budget.admission.evaluate" | "budget.admission.inspect" | "budget.admission.reconcile" | "budget.admission.reserve" | "budget.cancellation.deliver" | "budget.enforcement.deliver" | "budget.enforcement.scope" | "budget.exposure.evaluate"; diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index 3d5c2c00352..da4b8da1d2b 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -34,7 +34,7 @@ import type { } from "@paperclipai/shared"; import { HttpError, notFound, unprocessable } from "../errors.js"; import { logger } from "../middleware/logger.js"; -import { withAccountingTransaction } from "./accounting-transaction.js"; +import { withAccountingReadSnapshot, withAccountingTransaction } from "./accounting-transaction.js"; import { logActivity, type LogActivityInput, type ActivityPublication } from "./activity-log.js"; import { computeObservedSpend, computeObservedSpends, type ObservedSpend } from "./budget-exposure.js"; import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js"; @@ -198,11 +198,66 @@ export function budgetPoliciesForRun(companyId: string, agentId: string | null, ); } +type AdmissionContext = { issueId?: string | null; projectId?: string | null }; + +async function readAdmissionScopes(db: Db, companyId: string, agentId: string | null, context?: AdmissionContext) { + const scopes: Array<{ scopeType: BudgetScopeType; scopeId: string }> = [ + { scopeType: "company", scopeId: companyId }, ...(agentId ? [{ scopeType: "agent" as const, scopeId: agentId }] : []), + ...(context?.projectId ? [{ scopeType: "project" as const, scopeId: context.projectId }] : []), + ]; + const records = await Promise.all(scopes.map(scope => resolveScopeRecord(db, scope.scopeType, scope.scopeId))); + if (records.some(scope => scope.companyId !== companyId)) throw notFound("Budget scope not found"); + const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, companyId), + budgetPoliciesForRun(companyId, agentId, context?.projectId ?? null))); + return { scopes, records, policies }; +} + +function scopePauseBlock(scopeType: BudgetScopeType, scopeId: string, scope: ScopeRecord, hasPolicies: boolean) { + if (scopeType !== "agent" && scope.paused && scope.pauseReason !== "budget") { + return { scopeType, scopeId, scopeName: scope.name, reason: `${scopeType === "company" ? "Company" : "Project"} is paused and cannot start new work.` }; + } + if (scope.paused && scope.pauseReason === "budget" && !hasPolicies) { + return { scopeType, scopeId, scopeName: scope.name, reason: "Budget pause requires a policy or an explicit operator resume." }; + } + return null; +} + +async function inspectInvocationEligibility(db: Db, companyId: string, agentId: string | null, context?: AdmissionContext) { + const { scopes, records, policies } = await readAdmissionScopes(db, companyId, agentId, context); + if (policies.length > 0) { + if (records.some(scope => scope.budgetHeld)) return { needsReconciliation: true } as const; + const observations = await computeObservedSpends(db, policies); + if (policies.some((policy, index) => observedBlocks(policy, observations[index]) + || (policy.isActive && policy.hardStopEnabled && policy.amount > 0 && observations[index].recoveringRunCount > 0))) { + return { needsReconciliation: true } as const; + } + const incidents = await db.select({ policyId: budgetIncidents.policyId, thresholdType: budgetIncidents.thresholdType }) + .from(budgetIncidents).where(and(eq(budgetIncidents.companyId, companyId), + inArray(budgetIncidents.policyId, policies.map(policy => policy.id)), eq(budgetIncidents.status, "open"))); + const byId = new Map(policies.map((policy, index) => [policy.id, { policy, observed: observations[index] }])); + if (incidents.some(incident => { + const { policy, observed } = byId.get(incident.policyId)!; + return incident.thresholdType === "hard" || shouldResolveSoftIncident(policy, observed); + })) return { needsReconciliation: true } as const; + } + for (const [index, { scopeType, scopeId }] of scopes.entries()) { + const block = scopePauseBlock(scopeType, scopeId, records[index], + policies.some(policy => policy.scopeType === scopeType && policy.scopeId === scopeId)); + if (block) return { needsReconciliation: false, block } as const; + } + return { needsReconciliation: false, block: null } as const; +} + function unpricedUsageBlocks(policy: PolicyRow, observed: ObservedSpend) { return policy.unpricedUsagePolicy !== "allow" && (observed.unpricedEventCount > 0 || observed.unknownBrowserRunCount > 0); } +function shouldResolveSoftIncident(policy: PolicyRow, observed: ObservedSpend) { + return !policy.isActive || !policy.notifyEnabled + || compareCents(observed.totalExact, Math.ceil(policy.amount * policy.warnPercent / 100)) < 0; +} + function observedBlocks(policy: PolicyRow, observed: Awaited>) { if (!policy.isActive || !policy.hardStopEnabled || policy.amount <= 0) return false; return compareCents(observed.totalExact, policy.amount) >= 0 || (observed.pendingRunCount - observed.recoveringRunCount) > 0 || unpricedUsageBlocks(policy, observed); @@ -654,18 +709,12 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica // A closed UTC window, inactive policy, or budget raise releases only // budget-owned pauses. Manual pauses remain operator decisions. await resolveOpenIncidentsForPolicy(policy.id, policy.companyId, "approved", "budget_service", "hard"); - if (!policy.isActive || !policy.notifyEnabled || compareCents(observed.totalExact, Math.ceil(policy.amount * policy.warnPercent / 100)) < 0) { + if (shouldResolveSoftIncident(policy, observed)) { await resolveOpenIncidentsForPolicy(policy.id, policy.companyId, "approved", "budget_service", "soft"); } } - // Agent status has its own admission gate and error code. Preserve a - // manual pause without misclassifying it as a budget failure. - if (scopeType !== "agent" && scope.paused && scope.pauseReason !== "budget") { - return { scopeType, scopeId, scopeName: scope.name, reason: `${scopeType === "company" ? "Company" : "Project"} is paused and cannot start new work.` }; - } - if (scope.paused && scope.pauseReason === "budget" && !policies.length) { - return { scopeType, scopeId, scopeName: scope.name, reason: "Budget pause requires a policy or an explicit operator resume." }; - } + const pauseBlock = scopePauseBlock(scopeType, scopeId, scope, policies.length > 0); + if (pauseBlock) return pauseBlock; if (scope.budgetHeld && policies[0]) await resumeScopeFromBudget(policies[0]); if (recoveringPolicy) return { scopeType, scopeId, scopeName: scope.name, reason: "New work must wait while a native run recovers its unfinished accounting." }; @@ -748,14 +797,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica async function evaluateAdmissionInTransaction(companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) { - const scopes: Array<{ scopeType: BudgetScopeType; scopeId: string }> = [ - { scopeType: "company", scopeId: companyId }, ...(agentId ? [{ scopeType: "agent" as const, scopeId: agentId }] : []), - ...(context?.projectId ? [{ scopeType: "project" as const, scopeId: context.projectId }] : []), - ]; - const records = await Promise.all(scopes.map(scope => resolveScopeRecord(db, scope.scopeType, scope.scopeId))); - if (records.some(scope => scope.companyId !== companyId)) throw notFound("Budget scope not found"); - const policies = await db.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, companyId), - budgetPoliciesForRun(companyId, agentId, context?.projectId ?? null))); + const { scopes, records, policies } = await readAdmissionScopes(db, companyId, agentId, context); const observations = await computeObservedSpends(db, policies); const reservationIndexes = policies.map((policy, i) => policy.isActive && policy.hardStopEnabled && policy.metric === "billed_cents" && policy.amount > 0 ? i : -1).filter(i => i >= 0); const admission = (block: Awaited>) => ({ block, @@ -1188,8 +1230,13 @@ export function budgetService(db: Db, _hooks: BudgetServiceHooks = {}) { upsertPolicy: (companyId: string, input: BudgetPolicyUpsertInput, actorUserId: string | null) => mutate(companyId, service => service.upsertPolicy(companyId, input, actorUserId)), evaluateCostEvent: (event: typeof costEvents.$inferSelect) => mutate(event.companyId, service => service.evaluateCostEvent(event)), - getInvocationBlock: (companyId: string, agentId: string | null, context?: { issueId?: string | null; projectId?: string | null }) => - mutate(companyId, service => service.getInvocationBlock(companyId, agentId, context)), + getInvocationBlock: async (companyId: string, agentId: string | null, context?: AdmissionContext) => { + const eligibility = await withBudgetSpan("budget.admission.inspect", () => + withAccountingReadSnapshot(db, companyId, tx => inspectInvocationEligibility(tx, companyId, agentId, context))); + if (!eligibility.needsReconciliation) return eligibility.block; + return withBudgetSpan("budget.admission.reconcile", () => + mutate(companyId, service => service.getInvocationBlock(companyId, agentId, context))); + }, resolveIncident: (companyId: string, incidentId: string, input: BudgetIncidentResolutionInput, actorUserId: string) => mutate(companyId, service => service.resolveIncident(companyId, incidentId, input, actorUserId)), }; From 4d31588dea135a50326b470a3167f06fdbe68088 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:50:15 -0600 Subject: [PATCH 08/16] test(server): load model route setup before test deadlines (#18828) ## Thinking Path > - Paperclip uses native agents and local provider adapters to do work. > - The server model route must keep provider catalogs separate. > - Its tests use the real HTTP route with controlled service and adapter fixtures. > - Each test reset the module graph and imported the route again inside short test deadlines. > - Slow imports timed out. Unfinished work could then change mock call counts in a later case. > - This pull request loads the stable route graph during collection and keeps each case's fixture resets. ## Linked Issues or Issue Description Refs #18816 (affected required CI run). **What happened?** The pending Kimi model-route test exceeded its 15-second deadline in required Linux CI. The unchanged test file also failed locally: the first setup hook exceeded 30 seconds, later tests exceeded 15 seconds, and the last OpenCode case saw two calls instead of one after unfinished work reached the shared mock. **Expected behavior** Route import and transformation must finish before the case deadline starts. Each case must still test the real HTTP route with its own mock state, environment and profile values, and adapter registry cleanup. **Steps to reproduce** Run `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts` with a cold module graph. On a loaded worker, repeated module resets and imports can exceed the hook and test deadlines. ## What Changed - Register stable service mocks once and load the real route and middleware during test collection. - Remove per-case module invalidation. Keep all five original HTTP assertions, fixture resets, profile and environment restoration, and adapter unregister operations. - Keep the existing hook and test deadlines. Product source code does not change. No user-facing documentation changes are needed. ## Verification - The unchanged file reproduced the hook timeout, test timeouts, and leaked call-count failure locally. - All five original cases pass with the fix. The cold graph takes about 47 seconds during collection; the cases complete within the unchanged deadlines. - All five cases also pass at the same head in a shuffled order with seed `20261103`. - Module-boundary and diff checks pass. - Full repository typecheck, tests, and build run in required CI. No hosted product behavior is claimed from this test-only change. - Current-head CI attempt 2 failed in the typecheck harness because the trusted master workflow called `scripts/typecheck-test-projects.mjs`, which the older source head did not contain. The log is retained. A normal merge of master supplies the helper from #18738. All five original HTTP cases pass again on that merged baseline. Fresh CI and exact-head reviews passed at `86f0bf27267fe3f44ab813fb0a57ebbb94bae7fe`. - The refreshed run passed all server checks but one browser shard failed: `task-reload.spec.ts:63` waited five seconds for a task heading after the intentional bundle failure. The trace showed the saved task breadcrumb and properties, with the conversation still loading. The exact failure log and trace were retained. One targeted retry of that shard and its aggregate passed without source changes. This establishes recovery of the check; the original browser loading delay remains unexplained. - Final exact-head checks: 52 success, two skipped, and two neutral. Both Greptile and Paperclip reviews are 5/5. No review threads remain. ## Risks - Stable imports share module state across cases. The test retains explicit mock resets, adapter unregister, and profile/environment restoration. - The affected model route reads profile qualification and environment fixtures when it handles the request. These cases do not require a new module instance for each environment change. - No runtime or schema changes. No test deadline increase. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used repository tools, code execution, and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../adapter-model-refresh-routes.test.ts | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/server/src/__tests__/adapter-model-refresh-routes.test.ts b/server/src/__tests__/adapter-model-refresh-routes.test.ts index 6d94b9b6ed9..f65f12b52f1 100644 --- a/server/src/__tests__/adapter-model-refresh-routes.test.ts +++ b/server/src/__tests__/adapter-model-refresh-routes.test.ts @@ -101,13 +101,15 @@ function registerModuleMocks() { })); } +registerModuleMocks(); +const [{ agentRoutes }, { errorHandler }] = await Promise.all([ + vi.importActual("../routes/agents.js"), + vi.importActual("../middleware/index.js"), +]); + const refreshableAdapterType = "refreshable_adapter_route_test"; async function createApp() { - const [{ agentRoutes }, { errorHandler }] = await Promise.all([ - vi.importActual("../routes/agents.js"), - vi.importActual("../middleware/index.js"), - ]); const app = express(); app.use(express.json()); app.use((req, _res, next) => { @@ -159,11 +161,6 @@ async function unregisterTestAdapter(type: string) { describe("adapter model refresh route", () => { beforeEach(async () => { - vi.resetModules(); - vi.doUnmock("../routes/agents.js"); - vi.doUnmock("../routes/authz.js"); - vi.doUnmock("../middleware/index.js"); - registerModuleMocks(); vi.clearAllMocks(); mockCompanySkillService.listRuntimeSkillEntries.mockResolvedValue([]); mockCompanySkillService.resolveRequestedSkillKeys.mockResolvedValue([]); From 1f55073f96b2485f49f98ee8c261aa14ec2f79ac Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:55:39 -0600 Subject: [PATCH 09/16] fix(cursor-cloud): support API-only controllers on managed stacks (#18310) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-28-cursor-cloud-native-runner.md | 30 +++ maps/agent-access-map/agents.md | 19 ++ maps/feature-map/agents.md | 16 ++ server/src/__tests__/activity-service.test.ts | 40 ++++ server/src/__tests__/adapter-routes.test.ts | 25 +++ .../agent-adapter-validation-routes.test.ts | 50 ++++- ...-harness-verification-provider-api.test.ts | 164 +++++++++++++++++ .../agent-test-environment-routes.test.ts | 171 ++++++++++++++++++ .../src/__tests__/inbox-mine-routes.test.ts | 11 +- .../src/__tests__/live-recovery-work.test.ts | 117 +++++++++++- .../agent-lifecycle/adapters/configuration.ts | 4 +- .../agent-lifecycle/adapters/effects.ts | 3 +- .../agent-lifecycle/adapters/postgres.ts | 2 +- .../agent-lifecycle/adapters/records.ts | 2 +- server/src/routes/adapters.ts | 15 +- server/src/routes/agents.ts | 27 ++- .../src/services/acknowledged-native-stop.ts | 52 +++++- server/src/services/activity.ts | 15 ++ .../adapter-settings-postgres.test.ts | 57 +++++- .../services/agent-harness-verification.ts | 10 +- server/src/services/agent-hiring.ts | 8 +- server/src/services/agent-lifecycle.ts | 4 +- .../services/agent-runner-selection.test.ts | 86 ++++++++- server/src/services/agent-runner-selection.ts | 72 ++++++-- server/src/services/heartbeat-run-events.ts | 5 +- server/src/services/heartbeat.ts | 17 +- .../heartbeat/environment-selection.ts | 37 ++-- .../services/heartbeat/external-provider.ts | 2 +- .../services/heartbeat/live-recovery-work.ts | 11 +- server/src/services/heartbeat/recovery.ts | 3 +- .../services/heartbeat/run-control.test.ts | 155 +++++++++++++++- server/src/services/heartbeat/run-control.ts | 79 +++++--- .../services/heartbeat/runtime-execution.ts | 3 +- .../heartbeat/runtime-selection.test.ts | 69 ++++++- .../services/heartbeat/runtime-selection.ts | 21 ++- .../heartbeat/workspace-preparation.test.ts | 56 +++++- .../heartbeat/workspace-preparation.ts | 62 ++++--- .../native-runtime/cursor-cloud-runtime.ts | 15 ++ .../provider-api-placement.test.ts | 56 ++++++ .../native-runtime/provider-api-placement.ts | 60 ++++++ .../native-runtime/provider-profile.ts | 2 +- tests/runner-e2e/catalog.test.ts | 3 + tests/runner-e2e/catalog.ts | 2 +- .../cursor-cloud-replay-fault.test.ts | 5 +- tests/runner-e2e/cursor-cloud-replay-fault.ts | 2 +- ui/src/adapters/codex-local/config-fields.tsx | 2 +- .../AgentConfigForm.render.test.tsx | 43 +++++ ui/src/components/AgentConfigForm.tsx | 22 ++- ui/src/components/NewAgentDialog.test.tsx | 48 +++++ ui/src/components/TaskChatThread.test.tsx | 32 ++++ ui/src/components/TaskChatThread.tsx | 7 + .../new-agent/AgentBasicsDialog.tsx | 16 +- ui/src/components/new-agent/NewAgentSetup.tsx | 26 +-- .../task-chat/TaskChatComposer.test.tsx | 55 ++++++ .../components/task-chat/TaskChatComposer.tsx | 4 +- ui/src/hooks/useComposerStop.ts | 22 ++- ui/src/lib/new-agent-adapters.ts | 1 + ui/src/lib/wait-for-stopped-runs.test.ts | 8 + ui/src/lib/wait-for-stopped-runs.ts | 34 +++- ui/src/pages/IssueDetail.test.tsx | 16 ++ 60 files changed, 1825 insertions(+), 176 deletions(-) create mode 100644 server/src/__tests__/agent-harness-verification-provider-api.test.ts create mode 100644 server/src/services/native-runtime/provider-api-placement.test.ts create mode 100644 server/src/services/native-runtime/provider-api-placement.ts diff --git a/doc/plans/2026-10-28-cursor-cloud-native-runner.md b/doc/plans/2026-10-28-cursor-cloud-native-runner.md index fbdecbf71c0..6264582fb0c 100644 --- a/doc/plans/2026-10-28-cursor-cloud-native-runner.md +++ b/doc/plans/2026-10-28-cursor-cloud-native-runner.md @@ -15,6 +15,36 @@ and dependent [product PR #18069](https://github.com/paperclipai/paperclip/pull/ Neither PR is merged. Product PR #18069's Verification section owns the latest candidate, check status and final acceptance decision. +## Hosted placement correction — 2026-10-30 + +Native Cursor Cloud uses a provider API controller. Cursor hosts the model and +commands; Paperclip runs only its qualified API client and bounded private artifact +stager. This placement has no Paperclip Computer, lease, task checkout or host +provisioning command. It is derived from the validated native Cursor profile and +persisted with the admitted run; recovery retains that backend and placement. +Earlier Cursor profiles retain their admitted workspace binding as identity; +recovery does not reopen that workspace or allocate a Computer. Known backend +and driver contradictions fail closed, while historical missing driver values +remain recoverable. +The generic managed-sandbox-only policy and local Computer denial stay intact. +Kubernetes-only execution remains unsupported in save, Test, inventory and run +admission. An unavailable Computer does not advertise local Legacy execution. +Legacy direct agents retain their existing selection and execution behavior. + +Ordinary native setup clears the Computer binding and explains Cursor's hosted +environment. Save/reopen and Test use the same account/configuration contract as +run admission. The Cloud front door must pass the exact authenticated native-run +callback to the tenant; Core still enforces live run authority and per-turn scope. + +The ordinary Cloud card is available only with native rollout and a qualified +controller. A server-held new setup draft can use that managed API default before +model selection. It keeps its setup hold and does not inherit another harness's +model or permissions. Unsaved automatic Test uses the advertised native default; +saved Legacy agents still require an explicit conversion. + +This correction requires fresh hosted acceptance on its final image. Local and +synthetic checks do not qualify that deployed journey or erase prior failures. + ## Review correction checkpoint — 2026-10-29 20:25 UTC All six original acceptance gates have passed on their recorded candidates; diff --git a/maps/agent-access-map/agents.md b/maps/agent-access-map/agents.md index 75ef16b9af0..c9d591c1b71 100644 --- a/maps/agent-access-map/agents.md +++ b/maps/agent-access-map/agents.md @@ -19,6 +19,25 @@ approval activation, built-in provisioning and import preservation described below. The original baseline commit above still governs unrelated entries; this update does not claim operation-complete review or live agent verification. +Native Cursor Cloud uses a server-derived `provider_api` placement after its +complete native profile is validated. Create/update with `runner: paperclip` +stores `defaultEnvironmentId: null`; the caller cannot supply placement, commands, +working directories or arbitrary environment values. Only the bound +`CURSOR_API_KEY` is passed to the qualified API proxy. Test does not acquire a +Computer lease. Finish setup uses the same validated placement and account check; +it does not inherit a default Computer or its warm-task lifecycle. Incomplete +held drafts stay inactive until the complete profile passes activation validation. +Task/run authority, company boundaries, budgets and per-turn callback scope remain enforced. Local Computer execution stays denied under +managed-sandbox-only policy. Verify the actual saved agent and live run; a board +Test result does not prove agent task-tool access. + +On managed-only instances with native Runner enabled, a server-held Cursor setup +draft can resolve an automatic runner to the API controller before it has a +model. Unsaved automatic Test requests use the same advertised default and still +validate the full profile. Existing active Legacy agents and explicit Legacy +choices do not convert automatically. A client-supplied setup flag does not grant +draft status or bypass active model validation. + ## `hire` Capability ID: `agents/hire`. diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index 206f4544abb..1fac593afb4 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -37,6 +37,22 @@ Save & exit keeps its configuration; Finish setup on the agent returns to the same record. Final submission requests approval when required, then follows the existing preparation and verification lifecycle. +For **Cursor Cloud**, choose its card in Cloud when native Runner is enabled and +the qualified controller is available. Its held setup draft uses the API +controller without a Computer, before model or credential selection. Local +operators can choose **Paperclip Runner** in Advanced. Bind a Cursor Cloud +API key, enter the exact model and select a repository/ref or a named Cursor +cloud environment. Cursor hosts the agent's environment; this native setup has +no Paperclip Computer selection. Save, reopen, and run Test. Test checks the +account catalog and callback authentication gate without a prompt; the effective-model +warning remains visible for this experimental integration. Finish setup repeats +that account check and reaches Ready without preparing or leasing a Computer, +even when the instance has a default plugin Computer. The Cursor lifecycle stays +per-turn. Assign a real task +and verify its output, question continuation, Plan approval and Stop separately. +Managed-sandbox-only policy stays enabled. It continues to deny local agent code. +Kubernetes-only execution is not supported by this API controller. + When the native Runner adapter is available, **Google Antigravity** appears in the ordinary adapter picker. Before runtime acceptance, its card says **Not available yet**. Selecting it opens Google account setup and model configuration without saving diff --git a/server/src/__tests__/activity-service.test.ts b/server/src/__tests__/activity-service.test.ts index cf613e0b59e..d2ed798d5c7 100644 --- a/server/src/__tests__/activity-service.test.ts +++ b/server/src/__tests__/activity-service.test.ts @@ -325,6 +325,46 @@ describeEmbeddedPostgres("activity service", () => { expect(runs[0]).not.toHaveProperty("contextSnapshot"); }); + it.each(["requested", "acknowledged", "pending_native", "missing_native", "wrong_run", "wrong_company", + "malformed_execution", "null_execution", "malformed_native"] as const)( + "projects only bounded, owner-matched Stop evidence in issue history: %s", async proof => { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Stop history", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Stopped runner", adapterType: "process" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Stop the current run", assigneeAgentId: agentId }); + const native = { schema: "paperclip.native-cancellation.v1", runId, companyId, + dispatchState: proof === "pending_native" ? "pending" : "acknowledged", + intentAuditId: "private-stop-proof", effects: ["private-stop-proof"], secret: "private-stop-proof" }; + const result: Record = { + executionCancellation: { state: proof === "requested" ? "requested" : "acknowledged", + proof: "private-stop-proof", secret: "private-stop-proof" }, + nativeCancellation: native, unrelated: "private-stop-proof", + }; + if (proof === "missing_native") delete result.nativeCancellation; + if (proof === "wrong_run") native.runId = randomUUID(); + if (proof === "wrong_company") native.companyId = randomUUID(); + if (proof === "malformed_native") result.nativeCancellation = { ...native, dispatchState: { secret: "private-stop-proof" } }; + if (proof === "malformed_execution") result.executionCancellation = { state: { secret: "private-stop-proof" } }; + if (proof === "null_execution") result.executionCancellation = null; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, issueId, scopeKind: "issue", + nativeIssueId: issueId, runtimeMode: "native", status: "cancelled", finishedAt: new Date(), + contextSnapshot: { issueId }, livenessState: "advanced", resultJson: result }); + + const rows = await activityService(db).runsForIssue(companyId, issueId); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ runId, runtimeMode: "native", status: "cancelled" }); + expect(rows[0]!.resultJson).toEqual({ + executionCancellation: { state: proof === "requested" ? "requested" + : proof === "malformed_execution" || proof === "null_execution" ? "unknown" : "acknowledged" }, + ...(["missing_native", "wrong_run", "wrong_company", "malformed_native"].includes(proof) ? {} : { + nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId, companyId, + dispatchState: proof === "pending_native" ? "pending" : "acknowledged" }, + }), + }); + expect(JSON.stringify(rows)).not.toContain("private-stop-proof"); + }, + ); + it("backfills missing liveness for completed issue runs before returning the ledger", async () => { const companyId = randomUUID(); const agentId = randomUUID(); diff --git a/server/src/__tests__/adapter-routes.test.ts b/server/src/__tests__/adapter-routes.test.ts index 7a29bc1d1a0..75dccc36eac 100644 --- a/server/src/__tests__/adapter-routes.test.ts +++ b/server/src/__tests__/adapter-routes.test.ts @@ -1,4 +1,5 @@ import express from "express"; +import { unprocessable, forbidden } from "../errors.js"; import request from "supertest"; import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; import { vi } from "vitest"; @@ -259,6 +260,30 @@ describe("adapter routes", () => { expect(resolveRunnerTarget).not.toHaveBeenCalled(); }); + it("keeps Cursor API discovery available without granting local support when the implicit managed Computer is unavailable", async () => { + const unavailable = unprocessable("Managed Computer unavailable", { code: "managed_sandbox_unavailable" }); + const app = createApp({}, { resolveRunnerTarget: async () => { throw unavailable; } }); + const response = await request(app).get("/api/adapters?companyId=company-1"); + expect(response.status).toBe(200); + expect(response.body.find((adapter: any) => adapter.type === "cursor_cloud")).toMatchObject({ supportedRunners: ["paperclip"], defaultRunner: "paperclip" }); + expect(response.body.find((adapter: any) => adapter.type === "codex_local").supportedRunners).toEqual([]); + const explicit = await request(app).get("/api/adapters?companyId=company-1&environmentId=selected"); + expect(explicit.status).toBe(422); + const foreign = await request(createApp({}, { resolveRunnerTarget: async () => { throw forbidden("Foreign Computer", { code: "environment_company_mismatch" }); } })).get("/api/adapters?companyId=company-1"); + expect(foreign.status).toBe(403); + }); + + it("does not offer Cursor execution under Kubernetes-only policy, including the unavailable managed Computer fallback", async () => { + for (const resolveRunnerTarget of [ + async () => ({ driver: "sandbox", executionMode: "kubernetes" as const }), + async () => { throw unprocessable("Managed Computer unavailable", { code: "managed_sandbox_unavailable", executionMode: "kubernetes" }); }, + ]) { + const response = await request(createApp({}, { resolveRunnerTarget })).get("/api/adapters?companyId=company-1"); + expect(response.status).toBe(200); + expect(response.body.find((adapter: any) => adapter.type === "cursor_cloud").supportedRunners).toEqual([]); + } + }); + it.each(["environmentId=selected", "companyId=", "companyId=one&companyId=two", "companyId=one&environmentId="])("rejects malformed discovery scope before resolution (%s)", async query => { const resolveRunnerTarget = vi.fn(); const response = await request(createApp({}, { resolveRunnerTarget })).get(`/api/adapters?${query}`); diff --git a/server/src/__tests__/agent-adapter-validation-routes.test.ts b/server/src/__tests__/agent-adapter-validation-routes.test.ts index 75c8f730ed7..1606070bb74 100644 --- a/server/src/__tests__/agent-adapter-validation-routes.test.ts +++ b/server/src/__tests__/agent-adapter-validation-routes.test.ts @@ -68,7 +68,7 @@ const mockApprovalService = vi.hoisted(() => ({ const mockInstanceSettingsService = vi.hoisted(() => ({ get: vi.fn(async () => ({ defaultEnvironmentId: null })), getGeneral: vi.fn(async () => ({ censorUsernameInLogs: false })), - getExperimental: vi.fn(async (): Promise<{ enableNativeRunner: boolean; enableOpenAiDot?: boolean }> => ({ enableNativeRunner: false })), + getExperimental: vi.fn(async (): Promise<{ enableNativeRunner: boolean; enableOpenAiDot?: boolean; enableManagedSandboxOnly?: boolean }> => ({ enableNativeRunner: false })), })); const mockManagedAgentProfileService = vi.hoisted(() => ({ @@ -268,6 +268,7 @@ describe("agent routes adapter validation", () => { mockAccessService.setPrincipalPermission.mockResolvedValue(undefined); mockLogActivity.mockResolvedValue(undefined); mockSecretService.syncEnvBindingsForTarget.mockResolvedValue(undefined); + mockInstanceSettingsService.get.mockResolvedValue({ defaultEnvironmentId: null }); mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableNativeRunner: false }); mockManagedAgentProfileService.requireQualified.mockResolvedValue({ id: "managed-primary", @@ -959,6 +960,46 @@ describe("agent routes adapter validation", () => { expect(mockAgentService.create).not.toHaveBeenCalled(); }); + it("selects the native Cursor API controller from the ordinary held Cloud setup card without a Computer", async () => { + const settings = { defaultEnvironmentId: null, experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + mockInstanceSettingsService.get.mockResolvedValue(settings); + mockInstanceSettingsService.getExperimental.mockResolvedValue(settings.experimental); + const environments = await import("../services/environments.js"); + const findManaged = vi.fn(async () => null); + const getEnvironment = vi.fn(async () => null); + vi.spyOn(environments, "environmentService").mockReturnValue({ findManagedSandboxEnvironment: findManaged, + getById: getEnvironment, listBoundCompanyIds: vi.fn(async () => []) } as never); + const existing = { ...(await mockAgentService.getById()), adapterType: "process", adapterConfig: {}, + status: "paused", lifecycleState: "paused", pauseReason: "setup", lifecycleHolds: ["setup"] }; + mockAgentService.getById.mockResolvedValue(existing); + const app = await createApp(); + const response = await requestApp(app, base => request(base).patch(`/api/agents/${existing.id}`).send({ + name: "Cursor setup", adapterType: "cursor_cloud", runner: "auto", adapterConfig: {}, metadata: { setupAdapterPending: false }, + })); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toMatchObject({ lifecycleState: "paused", pauseReason: "setup", lifecycleHolds: ["setup"] }); + expect(mockAgentService.update).toHaveBeenCalledWith(existing.id, expect.objectContaining({ + adapterType: "paperclip_runner", defaultEnvironmentId: null, + adapterConfig: expect.objectContaining({ provider: "cursor_cloud", lifecycleMode: "per_turn" }), + }), expect.anything()); + expect(findManaged).not.toHaveBeenCalled(); + expect(getEnvironment).not.toHaveBeenCalled(); + mockAgentService.update.mockClear(); + mockAgentService.getById.mockResolvedValue({ ...existing, status: "idle", lifecycleState: "ready", pauseReason: null, lifecycleHolds: [] }); + const active = await requestApp(app, base => request(base).patch(`/api/agents/${existing.id}`).send({ + adapterType: "cursor_cloud", runner: "auto", adapterConfig: {}, setupOnly: true, + })); + expect(active.status).toBe(422); + expect(active.body.details?.code).toBe("managed_sandbox_unavailable"); + expect(mockAgentService.update).not.toHaveBeenCalled(); + const incompleteActive = await requestApp(app, base => request(base).patch(`/api/agents/${existing.id}`).send({ + adapterType: "paperclip_runner", runner: "paperclip", adapterConfig: { provider: "cursor_cloud" }, setupOnly: true, + })); + expect(incompleteActive.status).toBe(422); + expect(incompleteActive.body.details?.code).toBe("paperclip_runner_cursor_cloud_model_required"); + expect(mockAgentService.update).not.toHaveBeenCalled(); + }); + it.each(["auto", "legacy"] as const)("creates Pi through the public route with %s selection and the experimental flag off", async runner => { const selection = await import("../services/agent-runner-selection.js"); vi.spyOn(selection, "resolveNewAgentRunnerForCompany").mockImplementation(async (_db, _companyId, input) => @@ -1107,9 +1148,9 @@ describe("agent routes adapter validation", () => { ); }); - it("saves and reopens an explicit native Cursor Cloud setup without Codex defaults", async () => { - mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableNativeRunner: true }); - let existing = { ...(await mockAgentService.getById()), adapterType: "cursor_cloud", adapterConfig: {}, + it("saves and reopens native Cursor API setup under managed-only policy without a Computer", async () => { + mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableNativeRunner: true, enableManagedSandboxOnly: true }); + let existing = { ...(await mockAgentService.getById()), adapterType: "cursor_cloud", adapterConfig: {}, defaultEnvironmentId: "33333333-3333-4333-8333-333333333333", status: "paused", lifecycleState: "paused", pauseReason: "setup", lifecycleHolds: ["setup"] }; mockAgentService.getById.mockImplementation(async () => existing); mockAgentService.update.mockImplementation(async (_id, patch) => { existing = { ...existing, ...patch }; return existing; }); @@ -1123,6 +1164,7 @@ describe("agent routes adapter validation", () => { provider: "cursor_cloud", model: config.model, repoUrl: config.repoUrl, repoRef: "feature", env: config.env, lifecycleMode: "per_turn", } }); expect(saved.body.adapterConfig).not.toHaveProperty("codexPermissionMode"); + expect(saved.body.defaultEnvironmentId).toBeNull(); const reopened = await requestApp(app, baseUrl => request(baseUrl).get(`/api/agents/${existing.id}/configuration`)); expect(reopened.status, JSON.stringify(reopened.body)).toBe(200); expect(reopened.body.adapterConfig).toEqual(saved.body.adapterConfig); diff --git a/server/src/__tests__/agent-harness-verification-provider-api.test.ts b/server/src/__tests__/agent-harness-verification-provider-api.test.ts new file mode 100644 index 00000000000..b117ae7eaf2 --- /dev/null +++ b/server/src/__tests__/agent-harness-verification-provider-api.test.ts @@ -0,0 +1,164 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { eq, sql } from "drizzle-orm"; +import { agents, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, + environments, environmentLeases, executionWorkspaces, heartbeatRuns, plugins, type Db } from "@paperclipai/db"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { requireServerAdapter } from "../adapters/index.js"; +import { agentHarnessVerificationService } from "../services/agent-harness-verification.js"; +import { createAgentLifecycle, configureAgentLifecycle } from "../services/agent-lifecycle.js"; +import { createLifecycleDriver } from "../services/agent-lifecycle-driver.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { environmentService } from "../services/environments.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_PROVIDER_API_LIFECYCLE_TEST_DATABASE_URL"); +const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); + +describe.skipIf(!support.supported)("Cursor Cloud API controller activation", () => { + let database: Awaited>; + let db: Db; + let home: string; + let companyId: string; + const workers: ReturnType[] = []; + const config = { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", + lifecycleMode: "per_turn", env: { CURSOR_API_KEY: "synthetic-cursor-key" } }; + const fetch = vi.fn(async (input: unknown, init?: RequestInit) => { + const url = String(input); + if (url.startsWith("https://callback.example.test/mcp/native-runs/") && init?.method === "POST") return new Response(null, { status: 401 }); + if (url === "https://api.cursor.com/v1/models" && init?.method === "GET") return new Response(JSON.stringify({ items: [{ id: config.model, displayName: "Caller model" }] }), { headers: { "Content-Type": "application/json" } }); + throw new Error(`Unexpected provider request: ${init?.method} ${url}`); + }); + + beforeAll(async () => { + home = await mkdtemp(join(tmpdir(), "paperclip-provider-api-lifecycle-")); + vi.stubEnv("PAPERCLIP_HOME", home); + vi.stubEnv("PAPERCLIP_MANAGED_CONFIG", undefined); + vi.stubEnv("PAPERCLIP_SETTING_DEFAULTS", undefined); + vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://callback.example.test"); + if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl); + database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} } + : await startEmbeddedPostgresTestDatabase("paperclip-provider-api-lifecycle-"); + db = createDb(database.connectionString); + await db.execute(sql`set client_min_messages = warning`); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + + beforeEach(async () => { + for (const worker of workers) await worker.stop(); + await instanceSettingsService(db).update({ defaultEnvironmentId: null }); + await db.execute(sql`truncate table companies, plugins, environments restart identity cascade`); + await instanceSettingsService(db).updateGeneral({ executionMode: "any" }); + await instanceSettingsService(db).updateExperimental({ enableManagedSandboxOnly: true, enableNativeRunner: true }); + [companyId] = await db.insert(companies).values({ name: "API controller activation", issuePrefix: "API" }).returning({ id: companies.id }).then(rows => rows.map(row => row.id)); + fetch.mockClear(); + vi.stubGlobal("fetch", fetch); + }); + afterEach(() => { vi.restoreAllMocks(); vi.unstubAllGlobals(); }); + afterAll(async () => { + for (const worker of workers) await worker.stop(); + await db?.$client.end(); + await database?.cleanup(); + vi.unstubAllEnvs(); + if (home) await rm(home, { recursive: true, force: true }); + }); + + const current = (id: string) => db.select().from(agents).where(eq(agents.id, id)).then(rows => rows[0]!); + async function saved(adapterType = "paperclip_runner", adapterConfig = config) { + return (await db.insert(agents).values({ companyId, name: "Saved controller", adapterType, adapterConfig, + status: "paused", lifecycleState: "verifying" }).returning())[0]!; + } + async function noComputerWork() { + expect(await db.select().from(environmentLeases)).toHaveLength(0); + expect(await db.select().from(executionWorkspaces)).toHaveLength(0); + expect(await db.select().from(heartbeatRuns)).toHaveLength(0); + } + + it.each([false, true])("finishes the same held draft and reaches Ready with an experimental account WARN, inherited plugin Computer=%s", async defaultComputer => { + if (defaultComputer) { + const pluginId = randomUUID(); + await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "unneeded-computer", version: "1.0.0", status: "ready", + manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", requiresAgent: true, supportsTasks: true, warmTaskVersion: 1 }] } as never }); + const [computer] = await db.insert(environments).values({ name: "Inherited Computer", driver: "plugin", status: "active", + config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } }).returning(); + await instanceSettingsService(db).update({ defaultEnvironmentId: computer.id }); + } + expect(await environmentService(db).findManagedSandboxEnvironment(companyId)).toBeNull(); + const lifecycle = createAgentLifecycle(db); + const draft = await lifecycle.requestHire(companyId, { name: "Cursor setup", adapterType: "paperclip_runner", + adapterConfig: { provider: "cursor_cloud" } }, { setupOnly: true, runnerResolved: true }); + expect(draft).toMatchObject({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["setup"], defaultEnvironmentId: null }); + await lifecycle.updateConfiguration(draft.id, { adapterConfig: config, runtimeConfig: { heartbeat: { enabled: false } } }); + expect(await current(draft.id)).toMatchObject({ lifecycleState: "paused", lifecycleHolds: ["setup"] }); + const finished = await lifecycle.requestHire(companyId, { id: draft.id, name: draft.name, adapterType: "paperclip_runner", + adapterConfig: config, runtimeConfig: { heartbeat: { enabled: false } }, defaultEnvironmentId: null }, { setupAgentId: draft.id, runnerResolved: true }); + expect(finished.id).toBe(draft.id); + expect(finished.adapterConfig.lifecycleMode).toBe("per_turn"); + const probe = vi.spyOn(requireServerAdapter("paperclip_runner"), "testEnvironment"); + const driver = createLifecycleDriver(db, {} as never); + const worker = configureAgentLifecycle(db, driver); + workers.push(worker); + await worker.process(draft.id); + expect(await current(draft.id)).toMatchObject({ status: "idle", lifecycleState: "ready", lifecycleHolds: [], defaultEnvironmentId: null }); + expect(probe).toHaveBeenCalledOnce(); + expect(await probe.mock.results[0]!.value).toMatchObject({ status: "warn", checks: [ + { code: "cursor_cloud_account_ready", level: "info" }, + { code: "cursor_cloud_model_listed", level: "info" }, + { code: "cursor_cloud_model_identity_unverified", level: "warn" }, + ] }); + expect(fetch.mock.calls.map(([url, input]) => [String(url), input?.method])).toEqual([ + ["https://callback.example.test/mcp/native-runs/00000000-0000-4000-8000-000000000000", "POST"], + ["https://api.cursor.com/v1/models", "GET"], + ]); + await noComputerWork(); + }); + + it("keeps native Computer-backed setup preparation and warm-task defaults", async () => { + const pluginId = randomUUID(); + await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "required-computer", version: "1.0.0", status: "ready", + manifestJson: { capabilities: ["environment.drivers.register"], environmentDrivers: [{ driverKey: "tasks", requiresAgent: true, supportsTasks: true, warmTaskVersion: 1 }] } as never }); + const [computer] = await db.insert(environments).values({ name: "Required Computer", driver: "plugin", status: "active", + config: { pluginKey: pluginId, driverKey: "tasks", driverConfig: {} } }).returning(); + await instanceSettingsService(db).update({ defaultEnvironmentId: computer.id }); + const draft = await createAgentLifecycle(db).requestHire(companyId, { name: "Codex setup", adapterType: "paperclip_runner", + adapterConfig: { provider: "codex", model: "gpt-5.6-luna", lifecycleMode: "per_turn" } }, { setupOnly: true, runnerResolved: true }); + expect(draft).toMatchObject({ status: "paused", lifecycleState: "preparing", lifecycleHolds: ["setup"], adapterConfig: { lifecycleMode: "warm" } }); + expect(fetch).not.toHaveBeenCalled(); + await noComputerWork(); + }); + + it("verifies a saved native controller without resolving an unavailable managed Computer", async () => { + const agent = await saved(); + await expect(agentHarnessVerificationService(db, {} as never).verify(agent)).resolves.toBe("complete"); + expect(fetch).toHaveBeenCalledTimes(2); + await noComputerWork(); + }); + + it("rejects Kubernetes-only activation before any provider probe", async () => { + const agent = await saved(); + await instanceSettingsService(db).updateGeneral({ executionMode: "kubernetes" }); + await expect(agentHarnessVerificationService(db, {} as never).verify(agent)).rejects.toMatchObject({ + status: 422, details: { code: "provider_api_kubernetes_execution_unsupported" }, + }); + await expect(createAgentLifecycle(db).requestHire(companyId, { name: "Forbidden hire", adapterType: "paperclip_runner", adapterConfig: config }, { runnerResolved: true })) + .rejects.toMatchObject({ status: 422, details: { code: "provider_api_kubernetes_execution_unsupported" } }); + expect(fetch).not.toHaveBeenCalled(); + await noComputerWork(); + }); + + it("does not grant API placement to an incomplete active profile", async () => { + const agent = await saved("paperclip_runner", { ...config, model: "" }); + await expect(agentHarnessVerificationService(db, {} as never).verify(agent)).rejects.toMatchObject({ code: "paperclip_runner_cursor_cloud_model_required" }); + expect(fetch).not.toHaveBeenCalled(); + await noComputerWork(); + }); + + it.each(["opencode_local", "cursor_cloud"])("preserves the managed Computer requirement for %s", async adapterType => { + const agent = await saved(adapterType); + await expect(agentHarnessVerificationService(db, {} as never).verify(agent)).rejects.toMatchObject({ details: { code: "managed_sandbox_unavailable" } }); + expect(fetch).not.toHaveBeenCalled(); + await noComputerWork(); + }); +}); diff --git a/server/src/__tests__/agent-test-environment-routes.test.ts b/server/src/__tests__/agent-test-environment-routes.test.ts index 7a9088665ed..9a93d6badb2 100644 --- a/server/src/__tests__/agent-test-environment-routes.test.ts +++ b/server/src/__tests__/agent-test-environment-routes.test.ts @@ -293,6 +293,177 @@ describe("agent test-environment route", () => { await unregisterTestAdapter("external_test"); }); + it("tests the saved Cursor API controller without allocating a Computer under managed-only policy", async () => { + const { registerServerAdapter, unregisterServerAdapter, getServerAdapter } = await import("../adapters/index.js"); + const previous = getServerAdapter("paperclip_runner"); + const probe = vi.fn(async () => ({ adapterType: "paperclip_runner", status: "warn" as const, testedAt: new Date().toISOString(), + checks: [{ code: "cursor_cloud_account_ready", level: "info" as const, message: "Synthetic account probe; no prompt." }] })); + registerServerAdapter({ ...externalAdapter, type: "paperclip_runner", testEnvironment: probe }); + try { + const id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; + const adapterConfig = { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", env: { CURSOR_API_KEY: "synthetic" } }; + mockAgentService.getById.mockResolvedValue({ id, companyId: "company-1", adapterType: "paperclip_runner", adapterConfig, defaultEnvironmentId: "11111111-1111-4111-8111-111111111111" }); + mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableManagedSandboxOnly: true }); + mockInstanceSettingsService.get.mockResolvedValue({ defaultEnvironmentId: "11111111-1111-4111-8111-111111111111" }); + const app = await createApp(); + const response = await request(app).post("/api/companies/company-1/adapters/paperclip_runner/test-environment").send({ agentId: id, adapterConfig }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toMatchObject({ status: "warn", checks: [{ code: "cursor_cloud_account_ready" }] }); + expect(probe).toHaveBeenCalledWith(expect.objectContaining({ executionTarget: null, environmentName: null, config: adapterConfig })); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.realizeWorkspace).not.toHaveBeenCalled(); + expect(mockEnvironmentService.findManagedSandboxEnvironment).not.toHaveBeenCalled(); + mockInstanceSettingsService.getGeneral.mockResolvedValue({ censorUsernameInLogs: false, executionMode: "kubernetes" }); + const denied = await request(app).post("/api/companies/company-1/adapters/paperclip_runner/test-environment").send({ agentId: id, adapterConfig }); + expect(denied.status).toBe(422); + expect(denied.body.details?.code ?? denied.body.code).toBe("provider_api_kubernetes_execution_unsupported"); + expect(probe).toHaveBeenCalledOnce(); + } finally { unregisterServerAdapter("paperclip_runner"); if (previous) registerServerAdapter(previous); } + }); + + async function mockCursorSwitchProbes() { + const { getServerAdapter } = await import("../adapters/index.js"); + const legacy = vi.spyOn(getServerAdapter("cursor_cloud"), "testEnvironment").mockImplementation(async () => ({ + adapterType: "cursor_cloud", status: "pass", checks: [], testedAt: new Date().toISOString(), + })); + const native = vi.spyOn(getServerAdapter("paperclip_runner"), "testEnvironment").mockImplementation(async () => ({ + adapterType: "paperclip_runner", status: "warn", testedAt: new Date().toISOString(), + checks: [{ code: "cursor_cloud_account_ready", level: "info", message: "Synthetic no-prompt probe" }], + })); + return { legacy, native, restore: () => { legacy.mockRestore(); native.mockRestore(); } }; + } + + function savedCursorSwitchFixture() { + const id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; + const environmentId = "22222222-2222-4222-8222-222222222222"; + const adapterConfig = { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", env: {} }; + mockAgentService.getById.mockResolvedValue({ id, companyId: "company-1", adapterType: "paperclip_runner", + adapterConfig, runtimeConfig: {}, defaultEnvironmentId: "11111111-1111-4111-8111-111111111111" }); + const envVars = { CURSOR_API_KEY: { type: "secret_ref", secretId: "computer-key" } }; + mockEnvironmentService.getById.mockResolvedValue({ id: environmentId, name: "Chosen Computer", + driver: "local", config: {}, envVars }); + mockSecretService.resolveEnvBindings.mockResolvedValue({ env: { CURSOR_API_KEY: "chosen-computer-key" }, + secretKeys: new Set(["CURSOR_API_KEY"]), manifest: [] }); + return { id, environmentId, adapterConfig, envVars }; + } + + it.each([false, true])("tests a saved native Cursor's explicit Legacy switch on the selected Computer (legacyPrompt=%s)", async legacyPrompt => { + const probes = await mockCursorSwitchProbes(); + const probe = probes.legacy; + try { + const f = savedCursorSwitchFixture(); + const config = { ...f.adapterConfig, ...(legacyPrompt ? { promptTemplate: "Legacy prompt" } : {}) }; + const response = await request(await createApp()).post("/api/companies/company-1/adapters/paperclip_runner/test-environment") + .send({ agentId: f.id, runner: "legacy", environmentId: f.environmentId, adapterConfig: config }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(probe).toHaveBeenCalledWith(expect.objectContaining({ environmentName: "Chosen Computer", executionTarget: null, + config: expect.objectContaining({ model: "caller-model", env: { CURSOR_API_KEY: "chosen-computer-key" }, + ...(legacyPrompt ? { promptTemplate: "Legacy prompt" } : {}) }) })); + expect(probe.mock.calls[0][0].config).not.toHaveProperty("provider"); + expect(mockSecretService.resolveEnvBindings).toHaveBeenCalledWith("company-1", f.envVars, + expect.objectContaining({ consumerType: "environment", consumerId: f.environmentId })); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(probes.native).not.toHaveBeenCalled(); + } finally { probes.restore(); } + }); + + it("fails closed on a foreign Computer before credentials for an explicit Legacy switch", async () => { + const probes = await mockCursorSwitchProbes(); + try { + const f = savedCursorSwitchFixture(); + mockEnvironmentService.listBoundCompanyIds.mockResolvedValue(["company-2"]); + const response = await request(await createApp()).post("/api/companies/company-1/adapters/paperclip_runner/test-environment") + .send({ agentId: f.id, runner: "legacy", environmentId: f.environmentId, + adapterConfig: { ...f.adapterConfig, promptTemplate: "Legacy prompt" } }); + expect(response.status).toBe(403); + expect(mockSecretService.resolveAdapterConfigForRuntime).not.toHaveBeenCalled(); + expect(mockSecretService.resolveEnvBindings).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(probes.legacy).not.toHaveBeenCalled(); + expect(probes.native).not.toHaveBeenCalled(); + } finally { probes.restore(); } + }); + + it("keeps managed-only policy for a Legacy switch rather than testing its API controller", async () => { + const probes = await mockCursorSwitchProbes(); + const probe = probes.legacy; + try { + const f = savedCursorSwitchFixture(); + mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableManagedSandboxOnly: true }); + const response = await request(await createApp()).post("/api/companies/company-1/adapters/paperclip_runner/test-environment") + .send({ agentId: f.id, runner: "legacy", environmentId: f.environmentId, adapterConfig: f.adapterConfig }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toMatchObject({ status: "fail", checks: expect.arrayContaining([ + expect.objectContaining({ code: "managed_sandbox_unavailable", level: "error" }), + ]) }); + expect(probe).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(probes.native).not.toHaveBeenCalled(); + } finally { probes.restore(); } + }); + + it("keeps an explicit Paperclip Cursor test API-only despite a selected Computer", async () => { + const probes = await mockCursorSwitchProbes(); + const probe = probes.native; + try { + const f = savedCursorSwitchFixture(); + const settings = { defaultEnvironmentId: null, experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + mockInstanceSettingsService.get.mockResolvedValue(settings); + mockInstanceSettingsService.getExperimental.mockResolvedValue(settings.experimental); + const response = await request(await createApp()).post("/api/companies/company-1/adapters/paperclip_runner/test-environment") + .send({ agentId: f.id, runner: "paperclip", environmentId: f.environmentId, adapterConfig: f.adapterConfig }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(probe).toHaveBeenCalledWith(expect.objectContaining({ executionTarget: null, environmentName: null, + config: expect.objectContaining({ provider: "cursor_cloud" }) })); + expect(mockEnvironmentService.getById).not.toHaveBeenCalled(); + expect(mockSecretService.resolveEnvBindings).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(probes.legacy).not.toHaveBeenCalled(); + } finally { probes.restore(); } + }); + + it.each([ + { patch: { model: "" }, code: "paperclip_runner_cursor_cloud_model_required" }, + { patch: { command: "untrusted" }, code: "paperclip_runner_cursor_cloud_config_invalid" }, + ])("returns a useful validation error before Computer or credential work for invalid native Cursor input ($code)", async ({ patch, code }) => { + const app = await createApp(); + const response = await request(app).post("/api/companies/company-1/adapters/paperclip_runner/test-environment").send({ + adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", ...patch }, + }); + expect(response.status, JSON.stringify(response.body)).toBe(422); + expect(response.body.details?.code).toBe(code); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.realizeWorkspace).not.toHaveBeenCalled(); + expect(mockEnvironmentService.findManagedSandboxEnvironment).not.toHaveBeenCalled(); + }); + + it.each([undefined, "auto"] as const)("tests the prospective managed Cursor API default without converting saved Legacy agents (%s)", async runner => { + const { getServerAdapter } = await import("../adapters/index.js"); + const native = getServerAdapter("paperclip_runner"); + const probe = vi.spyOn(native, "testEnvironment").mockImplementation(async () => ({ adapterType: "paperclip_runner", status: "warn" as const, testedAt: new Date().toISOString(), + checks: [{ code: "cursor_cloud_account_ready", level: "info" as const, message: "Synthetic account probe; no prompt." }] })); + try { + const settings = { defaultEnvironmentId: null, experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + mockInstanceSettingsService.get.mockResolvedValue(settings); + mockInstanceSettingsService.getExperimental.mockResolvedValue(settings.experimental); + const adapterConfig = { model: "caller-model", repoUrl: "https://github.com/example/qualification", env: { CURSOR_API_KEY: "synthetic" } }; + const app = await createApp(); + const response = await request(app).post("/api/companies/company-1/adapters/cursor_cloud/test-environment").send({ runner, adapterConfig }); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toMatchObject({ adapterType: "paperclip_runner", checks: [{ code: "cursor_cloud_account_ready" }] }); + expect(probe).toHaveBeenCalledWith(expect.objectContaining({ executionTarget: null, environmentName: null, + config: expect.objectContaining({ provider: "cursor_cloud", model: "caller-model" }) })); + expect(mockEnvironmentService.findManagedSandboxEnvironment).not.toHaveBeenCalled(); + expect(mockEnvironmentRuntime.acquireRunLease).not.toHaveBeenCalled(); + const id = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa"; + mockAgentService.getById.mockResolvedValue({ id, companyId: "company-1", adapterType: "cursor_cloud", adapterConfig, defaultEnvironmentId: null }); + const legacy = await request(app).post("/api/companies/company-1/adapters/cursor_cloud/test-environment").send({ agentId: id, adapterConfig }); + expect(legacy.status).toBe(422); + expect(legacy.body.details?.code).toBe("managed_sandbox_unavailable"); + expect(probe).toHaveBeenCalledOnce(); + } finally { probe.mockRestore(); } + }); + it("accepts a managed Dot environment without allocating an idle sandbox", async () => { vi.stubEnv("PAPERCLIP_PUBLIC_URL", "https://paperclip.example"); mockInstanceSettingsService.getExperimental.mockResolvedValue({ diff --git a/server/src/__tests__/inbox-mine-routes.test.ts b/server/src/__tests__/inbox-mine-routes.test.ts index 527e661c42f..5a029b4ada8 100644 --- a/server/src/__tests__/inbox-mine-routes.test.ts +++ b/server/src/__tests__/inbox-mine-routes.test.ts @@ -285,11 +285,14 @@ describeEmbeddedPostgres("Mine inbox snapshot routes", () => { const responses = await Promise.all(requests); expect(responses.every(response => response.status === 200)).toBe(true); if (coalesced) expect((await coalesced).headers["x-paperclip-request-cache"]).toBe("coalesced"); - await request(app).get(snapshotPath).expect(200); - const computesBeforeFreshRead = started.mock.calls.length; - const fresh = await request(app).get(snapshotPath).expect(200); + // The aborted snapshot can outlive the seven ordinary reads. A coalesced + // response waits for that compute to settle before we require a fresh one. + const settled = await request(app).get(snapshotPath).expect(200); + const fresh = settled.headers["x-paperclip-request-cache"] === "coalesced" + ? await request(app).get(snapshotPath).expect(200) + : settled; expect(fresh.headers["x-paperclip-request-cache"]).toBe("miss"); - expect(started).toHaveBeenCalledTimes(computesBeforeFreshRead + 1); + expect(started).toHaveBeenCalledTimes(9); } finally { release(); await aborted; diff --git a/server/src/__tests__/live-recovery-work.test.ts b/server/src/__tests__/live-recovery-work.test.ts index ae79cab0a1a..25f3eeb9984 100644 --- a/server/src/__tests__/live-recovery-work.test.ts +++ b/server/src/__tests__/live-recovery-work.test.ts @@ -32,8 +32,14 @@ import { terminalizeLegacyExecution } from "../services/legacy-execution-recover import { issueService } from "../services/issues.js"; import { nativeRetryCancellationEligible, pendingFailedNativeRetryCancellationCondition } from "../services/native-runtime/native-cancellation-request.js"; import { requestNativeQuestionRunCancellation } from "../services/native-runtime/native-question-bridge.js"; -import { readNativeLocalProcessStop } from "../services/native-local-process-stop.js"; +import { readNativeLocalProcessStop, releaseSettledNativeRunProcess, settleNativeRunProcess } from "../services/native-local-process-stop.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { buildNativeExecutionInput } from "../services/native-runtime/native-execution-input.js"; +import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js"; +import * as nativeStopEvidence from "../services/acknowledged-native-stop.js"; import { remoteTerminationReceipt } from "../services/remote-execution-termination.js"; +import { subscribeCompanyLiveEvents } from "../services/live-events.js"; +import type { LiveEvent } from "@paperclipai/shared"; import { logger } from "../middleware/logger.js"; vi.setConfig({ testTimeout: 90_000 }); @@ -295,6 +301,115 @@ describe("residual recovery production matrix", () => { }, { timeout: 45_000 }); await competing.stop(); extraStops.pop(); await quiet(); }); + it.each(["confirmed", "already_retired", "missing_terminal", "unproven_terminal", "ack_failure", "late_terminal"] as const)( + "settles an API-only Stop after deferred controller retirement without inventing proof: %s", async mode => { + const f = await fixture(), runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID(); + const statusEvents: LiveEvent[] = []; + const unsubscribe = subscribeCompanyLiveEvents(f.companyId, event => { + if (event.type === "heartbeat.run.status" && event.payload.runId === runId) statusEvents.push(event); + }); + extraStops.push(async () => unsubscribe()); + const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { detached: true, stdio: "ignore" }); + await once(child, "spawn"); + const stopChild = async () => { + if (child.exitCode !== null || child.signalCode !== null) return; + const exited = once(child, "exit"); child.kill("SIGTERM"); await exited; + }; + extraStops.push(stopChild); + const recordedAt = new Date().toISOString(); + const nativeExecutionInput = buildNativeExecutionInput({ companyId: f.companyId, agentId: f.agent.id, runId, + provider: "cursor_cloud", model: "explicit-model", cursorCloud: { schema: "paperclip.cursor-cloud.config.v1", + repos: [{ url: "https://github.com/example/qualification" }] }, + issue: { id: f.issue.id, identifier: "API-1", title: f.issue.title, description: null, workMode: "standard" }, + taskPrompt: "Stop the foreground command", normalizedSessionId: sessionId, + workspace: { id: randomUUID(), cwd: "/private/controller", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Stop the foreground command", criteria: [{ id: "stop", requirement: "Stop the foreground command" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); + await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agent.id, + runtimeMode: "native", driverKind: "cursor_cloud_api", nativeIssueId: f.issue.id, nativeSessionId: sessionId, + runnerInstanceId: runnerId, status: "cancelled", finishedAt: new Date(), + runnerProfileJson: { backend: "cursor_cloud_api", nativeExecutionInput, sessionCheckpoint: { providerSessionId: "cloud-agent" }, + executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } }, + resultJson: { executionCancellation: { state: "requested", requestedAt: recordedAt }, + cancelledByActorType: "user", cancelledByUserId: f.actorId, nativeCancellation: { + schema: "paperclip.native-cancellation.v1", companyId: f.companyId, runId, issueId: f.issue.id, + scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true, + effects: ["release_run_resources"], recordedAt, acknowledgedAt: recordedAt, + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID() } } }); + await persistHeartbeatRunProcessMetadata(db, runId, { pid: child.pid!, processGroupId: child.pid!, startedAt: recordedAt, targetKind: "local" }); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, runId, phase: "committed" }); + const appendTerminal = async (target: typeof db = db) => { + const terminal = { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: "turn.interrupted", + sourceKind: "runner", sourceInstanceId: runnerId, sourceEventId: `${runnerId}:${runId}:1`, sourceSeq: 1, + runId, turnId: "stopped-turn", normalizedSessionId: sessionId, emittedAt: new Date().toISOString(), priority: 1, + payload: { status: "interrupted", error: null, ...(mode === "unproven_terminal" ? { providerTerminalObserved: false } : {}) } }; + await appendHeartbeatRunEvent(target, { companyId: f.companyId, agentId: f.agent.id, runId, + eventType: terminal.eventType, payload: { prpEvent: terminal }, nativeSource: { + sourceInstanceId: runnerId, sourceEventId: terminal.sourceEventId, sourceSeq: 1, + protocolSchemaVersion: 1, canonicalPayload: terminal } }); + }; + if (mode !== "missing_terminal" && mode !== "late_terminal") await appendTerminal(); + expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, runId))).toEqual([]); + expect(await settleNativeRunProcess(db, runId)).toBe("pending"); + expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" }); + expect(statusEvents).toEqual([]); + if (mode === "already_retired") { await stopChild(); expect(await releaseSettledNativeRunProcess(db, runId)).toBe("released"); } + await start(); + if (mode !== "already_retired") { + await vi.waitFor(() => expect(deadlines.get(DELIVERY_QUEUES.settledNativeProcesses)?.()).toBeGreaterThan(Date.now())); + expect((await getRun(runId)).processPid).toBe(child.pid); + await stopChild(); + } + const failure = new Error("synthetic Stop proof read failure after controller retirement"); + if (mode === "ack_failure") vi.spyOn(nativeStopEvidence, "acknowledgedNativeApiStopExecutionHasStopped").mockRejectedValueOnce(failure); + await advance(31_000); + if (mode === "ack_failure") { + expect((await getRun(runId)).processPid).toBeNull(); + expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" }); + expect(errors).toHaveLength(1); + expect(errors[0]).toBeInstanceOf(AggregateError); + expect((errors[0] as AggregateError).errors).toEqual([failure]); errors.length = 0; + expect(statusEvents).toEqual([]); + await advance(31_000); + } + const stopped = await getRun(runId); + expect(stopped).toMatchObject({ processPid: null, processGroupId: null, processStartedAt: null }); + expect(await readNativeLocalProcessStop(db, f.companyId, runId)).toMatchObject({ processPid: child.pid, processGroupId: child.pid }); + expect(stopped.resultJson?.executionCancellation).toMatchObject(["missing_terminal", "unproven_terminal", "late_terminal"].includes(mode) + ? { state: "requested" } : { state: "acknowledged", proof: "provider_api_stop_receipt" }); + expect(statusEvents).toHaveLength(["missing_terminal", "unproven_terminal", "late_terminal"].includes(mode) ? 0 : 1); + await quiet(); + if (mode === "late_terminal") { + const rollback = new Error("rollback the uncommitted terminal receipt"); + await expect(db.transaction(async tx => { await appendTerminal(tx as unknown as typeof db); throw rollback; })).rejects.toBe(rollback); + await advance(1_000); + expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" }); + expect(statusEvents).toEqual([]); + await quiet(); + let commit!: () => void, entered!: () => void; + const gate = new Promise(resolve => { commit = resolve; }); + const recorded = new Promise(resolve => { entered = resolve; }); + const transaction = db.transaction(async tx => { await appendTerminal(tx as unknown as typeof db); entered(); await gate; }); + await recorded; + try { + await vi.advanceTimersByTimeAsync(500); await immediate(); + expect((await getRun(runId)).resultJson?.executionCancellation).toMatchObject({ state: "requested" }); + expect(statusEvents).toEqual([]); + expect(await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId), + eq(heartbeatRunEvents.eventType, "turn.interrupted")))).toEqual([]); + } finally { commit(); await transaction; } + await vi.waitFor(async () => expect((await getRun(runId)).resultJson?.executionCancellation) + .toMatchObject({ state: "acknowledged", proof: "provider_api_stop_receipt" })); + await quiet(); + expect(statusEvents).toHaveLength(1); + expect(statusEvents[0]).toMatchObject({ companyId: f.companyId, type: "heartbeat.run.status", + payload: { runId, agentId: f.agent.id, status: "cancelled" } }); + expect(statusEvents[0]!.payload).not.toHaveProperty("resultJson"); + } + }, + ); it("releases a settled native process only after matching host and birth evidence arrives", async () => { const f = await fixture(), startedAt = new Date(); const child = spawn(process.execPath, ["-e", "console.log('identity-proof')"], { stdio: "ignore" }); diff --git a/server/src/modules/agent-lifecycle/adapters/configuration.ts b/server/src/modules/agent-lifecycle/adapters/configuration.ts index 80e90dbd71d..7acebf46d54 100644 --- a/server/src/modules/agent-lifecycle/adapters/configuration.ts +++ b/server/src/modules/agent-lifecycle/adapters/configuration.ts @@ -28,7 +28,9 @@ export async function updateAgentConfiguration(tx: Db, effects: LifecycleEffects const prepareSetup = changed && current.lifecycleHolds.length === 1 && current.lifecycleHolds[0] === "setup" && ["paused", "preparing", "verifying"].includes(current.lifecycleState) && await effects.setupRequiresPreparation(tx, current.companyId, - Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId") ? patch.defaultEnvironmentId : current.defaultEnvironmentId); + Object.prototype.hasOwnProperty.call(patch, "defaultEnvironmentId") ? patch.defaultEnvironmentId : current.defaultEnvironmentId, + { adapterType: patch.adapterType ?? current.adapterType, + adapterConfig: Object.prototype.hasOwnProperty.call(patch, "adapterConfig") ? patch.adapterConfig : current.adapterConfig }); const invalidate = changed && (["preparing", "verifying", "resuming"].includes(current.lifecycleState) || prepareSetup); if (invalidate) await effects.notifyWork(tx); const [updated] = await tx.update(agents).set({ ...patch, updatedAt: new Date(), ...(invalidate ? { diff --git a/server/src/modules/agent-lifecycle/adapters/effects.ts b/server/src/modules/agent-lifecycle/adapters/effects.ts index 57de2cc5468..e1787124063 100644 --- a/server/src/modules/agent-lifecycle/adapters/effects.ts +++ b/server/src/modules/agent-lifecycle/adapters/effects.ts @@ -6,7 +6,8 @@ import type { ActivityPublication } from "../../../types/activity-publication.js /** Services supply integrations; lifecycle owns the transaction and state writes. */ export interface LifecycleEffects { - setupRequiresPreparation(db: Db, companyId: string, environmentId?: string | null): Promise; + setupRequiresPreparation(db: Db, companyId: string, environmentId: string | null | undefined, + profile: Pick): Promise; notifyWork(db: Db): Promise; beforeStatusChange(db: Db, companyId: string, agentId: string, before: string, after: string): Promise; reportFailure(context: LifecycleFailure, error: unknown): void; diff --git a/server/src/modules/agent-lifecycle/adapters/postgres.ts b/server/src/modules/agent-lifecycle/adapters/postgres.ts index d7f8e77c553..2ce6580ef41 100644 --- a/server/src/modules/agent-lifecycle/adapters/postgres.ts +++ b/server/src/modules/agent-lifecycle/adapters/postgres.ts @@ -64,7 +64,7 @@ async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: Lifec if (command === "complete" && next === "ready" && holds.length) next = "pausing"; if (command === "complete" && next === "paused" && !holds.length) next = "resuming"; if (["complete", "reconcile"].includes(command) && next === "paused" && holds.length === 1 && holds[0] === "setup" - && await effects.setupRequiresPreparation(tx, agent.companyId, agent.defaultEnvironmentId)) next = "preparing"; + && await effects.setupRequiresPreparation(tx, agent.companyId, agent.defaultEnvironmentId, agent)) next = "preparing"; if (next === agent.lifecycleState) { if (JSON.stringify(holds) !== JSON.stringify(agent.lifecycleHolds) || agent.pauseReason !== (holds[0] ?? null)) { const [updated] = await tx.update(agents).set({ lifecycleHolds: holds, pauseReason: holds[0] ?? null }).where(eq(agents.id, agent.id)).returning(); diff --git a/server/src/modules/agent-lifecycle/adapters/records.ts b/server/src/modules/agent-lifecycle/adapters/records.ts index 96892987b01..64e10144c01 100644 --- a/server/src/modules/agent-lifecycle/adapters/records.ts +++ b/server/src/modules/agent-lifecycle/adapters/records.ts @@ -36,7 +36,7 @@ export function agentRecords(db: Db, effects: LifecycleEffects) { const [company] = await tx.select({ status: companies.status }).from(companies).where(eq(companies.id, companyId)); const companyHolds = company?.status === "paused" ? ["company_paused"] : company?.status === "archived" ? ["company_archived"] : []; const holds = [...(status === "paused" ? [pauseReason ?? "manual"] : []), ...companyHolds]; - const prepareSetup = options?.setupOnly && !companyHolds.length && await effects.setupRequiresPreparation(txDb, companyId, prepared.defaultEnvironmentId); + const prepareSetup = options?.setupOnly && !companyHolds.length && await effects.setupRequiresPreparation(txDb, companyId, prepared.defaultEnvironmentId, prepared); const lifecycleState = status === "pending_approval" ? "pending_approval" : status === "terminated" ? "terminated" : prepareSetup ? "preparing" : status === "paused" ? "paused" : companyHolds.length ? "pausing" : "preparing"; if (lifecycleState === "preparing" || lifecycleState === "pausing") await effects.notifyWork(tx); diff --git a/server/src/routes/adapters.ts b/server/src/routes/adapters.ts index 7a02043644b..94e0ab10c97 100644 --- a/server/src/routes/adapters.ts +++ b/server/src/routes/adapters.ts @@ -48,7 +48,8 @@ import type { } from "@paperclipai/adapter-utils"; import { loadExternalAdapterPackage, getUiParserSource, getOrExtractUiParserSource, reloadExternalAdapter } from "../adapters/plugin-loader.js"; import { logger } from "../middleware/logger.js"; -import { badRequest, forbidden } from "../errors.js"; +import { badRequest, forbidden, HttpError } from "../errors.js"; +import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { assertExternalAdapterCodeAllowed } from "../services/adapter-code-policy.js"; import { getHiddenSettings } from "../services/settings-visibility.js"; import { assertBoardOrgAccess, assertCompanyAccess, assertInstanceAdmin, getActorInfo } from "./authz.js"; @@ -281,7 +282,17 @@ export function adapterRoutes(options: { if (typeof companyId === "string") { assertCompanyAccess(_req, companyId); if (!options.resolveRunnerTarget) throw badRequest("Scoped adapter discovery is unavailable."); - target = await options.resolveRunnerTarget(companyId, typeof environmentId === "string" ? environmentId : undefined); + try { + target = await options.resolveRunnerTarget(companyId, typeof environmentId === "string" ? environmentId : undefined); + } catch (error) { + // API controllers do not need a Computer. Keep their catalog visible + // when only the implicit managed Computer is unavailable. This target + // grants no local execution support to any workspace harness. + if (environmentId !== undefined || !(error instanceof HttpError) || error.status !== 422 + || (error.details as { code?: unknown } | undefined)?.code !== "managed_sandbox_unavailable") throw error; + target = { driver: "unavailable", ...((error.details as { executionMode?: unknown }).executionMode === "kubernetes" + ? { executionMode: "kubernetes" as const } : {}) }; + } } await waitForExternalAdapters(); diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 756fed8a50e..ad455fc6dd8 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -1,3 +1,4 @@ +import { assertProviderApiPlacementPolicy, managedCursorApiDefaultEnabled, resolveProviderApiPlacement } from "../services/native-runtime/provider-api-placement.js"; import { readTaskPanelActivations } from "../services/plugin-task-panels.js"; import { enqueueNativeRuntimeResponse } from "../services/native-runtime/native-runtime-response-delivery.js"; import { latestFailedRunsByAgent } from "@paperclipai/shared"; @@ -1960,6 +1961,9 @@ export function agentRoutes( return input.nextAdapterConfig; } if (["openai_dot", "muse"].includes(String(input.nextAdapterConfig.provider))) return input.nextAdapterConfig; + // A provisional setup harness supplies no Cursor model or permissions. + // Legacy Cursor conversion still retains its provider-specific defaults. + if (input.nextAdapterConfig.provider === "cursor_cloud" && input.previousAdapterType !== "cursor_cloud") return input.nextAdapterConfig; const defaults = paperclipRunnerTransitionConfig(input.previousAdapterType, input.previousAdapterConfig.model, input.nextAdapterConfig.provider); if (!["claude_local", "codex_local", "opencode_local", "pi_local", "grok_local"].includes(input.previousAdapterType) && !isPaperclipRunnerProvider(input.nextAdapterConfig.provider)) { @@ -2205,6 +2209,7 @@ export function agentRoutes( ); const incompleteModelSetup = input.setupOnly && input.adapterType === "paperclip_runner" && (normalizedAdapterConfig.provider === "opencode" + || normalizedAdapterConfig.provider === "cursor_cloud" || (normalizedAdapterConfig.provider === "acpx" && ["pi", "kimi"].includes(String(normalizedAdapterConfig.acpxAgent)))) && (normalizedAdapterConfig.model === undefined || normalizedAdapterConfig.model === ""); // Inactive drafts can precede model selection. Test/hire always validate the complete profile. @@ -3274,7 +3279,25 @@ export function agentRoutes( } const testingSetup = savedAgent ? isAgentSetupDraft(savedAgent) : false; const poolBinding = requestedBinding ?? (savedAgent?.runtimeConfig?.aiConnection ? aiRuntimeConnectionBindingSchema.parse(savedAgent.runtimeConfig.aiConnection) : undefined); - const requestedEnvironmentId = await resolveAdapterTestEnvironmentId( + const prospectiveCursorDefault = type === "cursor_cloud" && !savedAgent + && (req.body.runner === undefined || req.body.runner === "auto") + && managedCursorApiDefaultEnabled((await instanceSettings.get()).experimental); + const testRunner = prospectiveCursorDefault ? "paperclip" : req.body.runner; + let providerApiTest; + try { + // An explicit Legacy switch must resolve its Computer and credentials + // before runner selection converts the saved native adapter. + providerApiTest = testRunner === "legacy" ? null : resolveProviderApiPlacement({ agent: { + adapterType: type === "cursor_cloud" && testRunner === "paperclip" ? "paperclip_runner" : type, + adapterConfig: type === "cursor_cloud" && testRunner === "paperclip" + ? { ...inputAdapterConfig, provider: "cursor_cloud" } : inputAdapterConfig, + } }); + } catch (error) { + if (error instanceof PaperclipRunnerProviderProfileError) throw unprocessable(error.message, { code: error.code }); + throw error; + } + if (providerApiTest) assertProviderApiPlacementPolicy(await instanceSettings.getGeneral()); + const requestedEnvironmentId = providerApiTest ? null : await resolveAdapterTestEnvironmentId( companyId, // Omission tests the saved selection. Explicit null tests a prospective // change back to the instance default. @@ -3320,7 +3343,7 @@ export function agentRoutes( ? { adapterType: type, adapterConfig: adapterConfigForTest } : await resolveNewAgentRunnerForCompany(db, companyId, { adapterType: type, adapterConfig: adapterConfigForTest, - runner: req.body.runner, + runner: testRunner, defaultEnvironmentId: requestedEnvironmentId, }); type = selection.adapterType; diff --git a/server/src/services/acknowledged-native-stop.ts b/server/src/services/acknowledged-native-stop.ts index e937f3df2a7..09124602f0d 100644 --- a/server/src/services/acknowledged-native-stop.ts +++ b/server/src/services/acknowledged-native-stop.ts @@ -1,7 +1,12 @@ -import { and, eq } from "drizzle-orm"; -import { environmentLeases, heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, desc, eq, inArray, isNotNull } from "drizzle-orm"; +import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db"; +import { parseObject } from "../adapters/utils.js"; +import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import { hasNativeLocalProcessStop } from "./native-local-process-stop.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; +import { validatePrpEvent } from "../vendor/paperclip-runner/index.js"; +import { nativeSha256 } from "./native-runtime/canonical.js"; +import { isConfirmedCursorCloudTerminalObservation } from "./native-runtime/cursor-cloud-runtime.js"; /** A server-recorded run-only Stop must not manufacture a recovery incident. */ export function hasAcknowledgedNativeRunStopReceipt(run: { @@ -37,6 +42,49 @@ export function isAcknowledgedNativeStop(run: Parameters): Promise { + if (run.runtimeMode !== "native" || run.driverKind !== "cursor_cloud_api" || !isAcknowledgedNativeStop(run) || + run.processPid !== null || run.processGroupId !== null || !run.nativeIssueId || !run.nativeSessionId || !run.runnerInstanceId) return false; + const intent = parseObject(run.resultJson?.nativeCancellation); + const requestedAt = typeof intent.recordedAt === "string" ? Date.parse(intent.recordedAt) : NaN; + if (!Number.isFinite(requestedAt) || !Array.isArray(intent.effects) || intent.effects.length !== 1 || + intent.effects[0] !== "release_run_resources" || + !String(intent.intentAuditId ?? "").trim() || !String(intent.acknowledgementAuditId ?? "").trim()) return false; + const profile = parseObject(run.runnerProfileJson); + const execution = parseObject(profile.nativeExecutionInput); + const binding = parseObject(execution.binding), session = parseObject(execution.session); + if (profile.executionPlacement === undefined || profile.backend !== "cursor_cloud_api" || + binding.companyId !== run.companyId || binding.runId !== run.id || binding.issueId !== run.nativeIssueId || + binding.agentId !== run.agentId || session.normalizedSessionId !== run.nativeSessionId || session.driverKind !== "cursor_cloud_api") return false; + try { + if (!resolveProviderApiPlacement({ run, agent: { adapterType: "process", adapterConfig: {} } })) return false; + } catch { return false; } + const [lease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and( + eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id), + )).limit(1); + if (lease || !await hasNativeLocalProcessStop(db, run.companyId, run.id)) return false; + const [terminal] = await db.select().from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id), + isNotNull(heartbeatRunEvents.sourceEventId), + inArray(heartbeatRunEvents.eventType, ["turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]), + )).orderBy(desc(heartbeatRunEvents.seq)).limit(1); + const validated = validatePrpEvent(parseObject(terminal?.payload).prpEvent); + if (!terminal || !validated.ok || terminal.sourcePayloadSha256 !== nativeSha256(validated.event)) return false; + const event = validated.event; + return isConfirmedCursorCloudTerminalObservation({ eventType: terminal.eventType, payload: event.payload }) && terminal.agentId === run.agentId && + event.schema === "paperclip.prp.event.v1" && [1, 2].includes(Number(event.schemaVersion)) && + terminal.protocolSchemaVersion === event.schemaVersion && event.eventType === terminal.eventType && + event.sourceKind === "runner" && event.runId === run.id && event.normalizedSessionId === run.nativeSessionId && + event.sourceInstanceId === run.runnerInstanceId && terminal.sourceInstanceId === event.sourceInstanceId && + terminal.sourceEventId === event.sourceEventId && terminal.sourceSeq === event.sourceSeq && + Number.isSafeInteger(event.sourceSeq) && Number(event.sourceSeq) > 0 && + typeof event.turnId === "string" && Boolean(event.turnId); +} + /** A Stop acknowledgement alone does not prove provider cleanup completed. */ export async function acknowledgedNativeStopExecutionHasStopped(db: Db, run: Pick) { diff --git a/server/src/services/activity.ts b/server/src/services/activity.ts index 5a34a5d430a..b3184891459 100644 --- a/server/src/services/activity.ts +++ b/server/src/services/activity.ts @@ -91,11 +91,26 @@ export function activityService(db: Db) { `.as("usageJson"); // A denied user needs the selected connection's name to repair their task. // Other readers need the connection's existing human sharing permission. + // A malformed present Stop receipt must stay unconfirmed, not disappear. const summarizedResultJson = (viewerUserId: string | null) => sql | null>` case when ${heartbeatRuns.resultJson} is null then null else jsonb_strip_nulls(jsonb_build_object( 'conversationReset', ${heartbeatRuns.resultJson} -> 'conversationReset', + 'executionCancellation', case when ${heartbeatRuns.resultJson} ? 'executionCancellation' + then jsonb_build_object('state', case + when jsonb_typeof(${heartbeatRuns.resultJson} -> 'executionCancellation') = 'object' + and ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' in ('requested', 'acknowledged') + then ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' else 'unknown' end) end, + 'nativeCancellation', case + when jsonb_typeof(${heartbeatRuns.resultJson} -> 'nativeCancellation') = 'object' + and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,schema}' = 'paperclip.native-cancellation.v1' + and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,runId}' = ${heartbeatRuns.id}::text + and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,companyId}' = ${heartbeatRuns.companyId}::text + and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}' in ('pending', 'acknowledged') + then jsonb_build_object('schema', 'paperclip.native-cancellation.v1', + 'runId', ${heartbeatRuns.id}, 'companyId', ${heartbeatRuns.companyId}, + 'dispatchState', ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}') end, 'configurationIncomplete', case when ${heartbeatRuns.resultJson} -> 'configurationIncomplete' ->> 'reason' = 'workspace_base_ref_unresolved' then jsonb_build_object('reason', 'workspace_base_ref_unresolved') diff --git a/server/src/services/adapter-settings-postgres.test.ts b/server/src/services/adapter-settings-postgres.test.ts index 299c4b14f35..6198c163e05 100644 --- a/server/src/services/adapter-settings-postgres.test.ts +++ b/server/src/services/adapter-settings-postgres.test.ts @@ -8,7 +8,7 @@ import os from "node:os"; import path from "node:path"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; -const legacy = vi.hoisted(() => ({ get: vi.fn(() => [] as string[]), set: vi.fn() })); +const legacy = vi.hoisted(() => ({ extraTypes: [] as string[], get: vi.fn(() => [] as string[]), set: vi.fn() })); vi.mock("./adapter-plugin-store.js", () => ({ getDisabledAdapterTypes: legacy.get, setAdapterDisabled: legacy.set, @@ -16,7 +16,7 @@ vi.mock("./adapter-plugin-store.js", () => ({ getAdapterPluginByType: () => undefined, })); vi.mock("../adapters/registry.js", () => ({ - listServerAdapters: () => [{ type: "codex_local" }, { type: "claude_local" }, { type: "paperclip_runner" }], + listServerAdapters: () => [{ type: "codex_local" }, { type: "claude_local" }, { type: "paperclip_runner" }, ...legacy.extraTypes.map(type => ({ type }))], findServerAdapter: (type: string) => ({ type }), findActiveServerAdapter: (type: string) => ({ type }), isOverridePaused: () => false, @@ -56,6 +56,7 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos beforeEach(async () => { directory = await fs.mkdtemp(path.join(os.tmpdir(), "adapter-settings-import-")); await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, "adapter-settings:v1")); + legacy.extraTypes = []; legacy.get.mockClear(); legacy.set.mockClear(); vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "synthetic-test-token"); @@ -188,6 +189,58 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos expect((await resolveNewAgentRunnerForCompany(db, "company", input)).adapterType).toBe("codex_local"); }); + it("refreshes API-only Cursor policy from PostgreSQL and authoritative operator declarations", async () => { + legacy.extraTypes = ["cursor_cloud"]; + await service().seedIfAbsent({ disabledTypes: [] }); + await db.insert(instanceSettings).values({ singletonKey: "default", general: {}, + experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }) + .onConflictDoUpdate({ target: instanceSettings.singletonKey, set: { general: {}, + experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } } }); + const draft = { adapterType: "cursor_cloud", runner: "auto" as const, setupOnly: true, adapterConfig: {}, disabledTypes: [] }; + const saved = { adapterType: "paperclip_runner", adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/acme/repo", lifecycleMode: "per_turn" } }; + const admitted = async (input = draft as typeof draft | typeof saved) => { + expect(await resolveNewAgentRunnerForCompany(db, "company", input)) + .toMatchObject({ adapterType: "paperclip_runner", defaultEnvironmentId: null, adapterConfig: { provider: "cursor_cloud" } }); + }; + const denied = async (input = draft as typeof draft | typeof saved) => { + await expect(resolveNewAgentRunnerForCompany(db, "company", input)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_runner_unavailable" } }); + }; + await admitted(); // The default managed tenant has no Computer or CLI probe. + for (const type of ["cursor_cloud", "paperclip_runner"]) { + await adapterSettingsService(peer, managed).setDisabled(type, true); + await denied(); + await denied(saved); + await adapterSettingsService(peer, managed).setDisabled(type, false); + await admitted(); + } + await adapterSettingsService(peer, managed).setDisabled("cursor_cloud", true); + const stored = await service().inspectStored(); + vi.stubEnv("PAPERCLIP_ADAPTERS", JSON.stringify([ + { adapterType: "cursor_cloud", enabled: true }, { adapterType: "paperclip_runner", enabled: true }, + ])); + await admitted(); // Operator allowlist supersedes, rather than unions, saved disables. + await admitted(saved); + expect(await service().inspectStored()).toEqual(stored); + for (const declaration of [ + [{ adapterType: "cursor_cloud", enabled: true }], + [{ adapterType: "paperclip_runner", enabled: true }], + [{ adapterType: "cursor_cloud", enabled: false }, { adapterType: "paperclip_runner", enabled: true }], + [{ adapterType: "cursor_cloud", enabled: true }, { adapterType: "paperclip_runner", enabled: false }], + ]) { + vi.stubEnv("PAPERCLIP_ADAPTERS", JSON.stringify(declaration)); + await denied(); + await denied(saved); + } + vi.stubEnv("PAPERCLIP_ADAPTERS", "invalid operator declaration"); + await expect(resolveNewAgentRunnerForCompany(db, "company", draft)).rejects.toThrow(); + vi.stubEnv("PAPERCLIP_ADAPTERS", undefined); + await denied(); // Removing the overlay restores the saved choice. + expect(await service().inspectStored()).toEqual(stored); + expect(legacy.get).not.toHaveBeenCalled(); + expect(legacy.set).not.toHaveBeenCalled(); + }); + it("rolls back the setting when its audit write fails", async () => { await expect(service().setDisabled("codex_local", true, async () => { throw new Error("audit failed"); })).rejects.toThrow("audit failed"); expect(await service().inspectStored()).toBeNull(); diff --git a/server/src/services/agent-harness-verification.ts b/server/src/services/agent-harness-verification.ts index 9a203f04e50..c8f7e21561a 100644 --- a/server/src/services/agent-harness-verification.ts +++ b/server/src/services/agent-harness-verification.ts @@ -10,6 +10,8 @@ import { aiConnectionService } from "./ai-connections.js"; import { withManagedAiProbe, stripAiAuthBindings } from "./ai-connection-runtime.js"; import { environmentService } from "./environments.js"; import { secretService } from "./secrets.js"; +import { instanceSettingsService } from "./instance-settings.js"; +import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import { museRunnerBroker } from "./muse-runner-broker.js"; import { dotRunnerBroker } from "./dot-runner-broker.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; @@ -51,6 +53,8 @@ export function agentHarnessVerificationService(db: Db, manager: PluginWorkerMan const binding = await museRunnerBroker(db).bindingForAgent(agent.companyId,agent.id); return binding?.status === "ready" && binding.paired && binding.receiverDetected && binding.backgroundReplyVerified && binding.id === agent.adapterConfig.museBindingId ? "complete" : "pending"; } + const providerApi = resolveProviderApiPlacement({ agent }); + if (providerApi) assertProviderApiPlacementPolicy(await instanceSettingsService(db).getGeneral()); const [company] = await db.select().from(companies).where(eq(companies.id, agent.companyId)); const responsibleUserId = agent.lifecycleOperation?.responsibleUserId ?? company.defaultResponsibleUserId; const context = { consumerType: "agent" as const, consumerId: agent.id, actorType: "system" as const, @@ -66,7 +70,7 @@ export function agentHarnessVerificationService(db: Db, manager: PluginWorkerMan } if (binding) config = { ...config, env: stripAiAuthBindings(config.env) }; config = (await secrets.resolveAdapterConfigForRuntime(agent.companyId, config, context, { adapterType: agent.adapterType })).config; - const environmentId = await tests.resolveAdapterTestEnvironmentId(agent.companyId, agent.defaultEnvironmentId, agent.adapterType === "paperclip_runner" && config.provider === "acpx" && config.acpxAgent === "copilot"); + const environmentId = providerApi ? null : await tests.resolveAdapterTestEnvironmentId(agent.companyId, agent.defaultEnvironmentId, agent.adapterType === "paperclip_runner" && config.provider === "acpx" && config.acpxAgent === "copilot"); if (environmentId) { await tests.assertAdapterTestEnvironmentForCompany(agent.companyId, environmentId, agent.adapterType); const environment = await environments.getById(environmentId); @@ -81,7 +85,9 @@ export function agentHarnessVerificationService(db: Db, manager: PluginWorkerMan adapterType: agent.adapterType, model: config.model, runnerProvider: config.provider, acpxAgent: config.acpxAgent, binding, }); - const target = await tests.resolveAdapterTestExecutionContext({ agentId: agent.id, companyId: agent.companyId, adapterType: agent.adapterType, environmentId }); + const target = providerApi + ? { executionTarget: null, environmentName: null, fallbackChecks: [], release: async (_status?: "released" | "failed") => {} } + : await tests.resolveAdapterTestExecutionContext({ agentId: agent.id, companyId: agent.companyId, adapterType: agent.adapterType, environmentId }); let status: "released" | "failed" = "failed"; try { if (target.fallbackChecks.length) throw Object.assign(new Error("The test environment is unavailable"), { code: "test_environment_unavailable" }); diff --git a/server/src/services/agent-hiring.ts b/server/src/services/agent-hiring.ts index 16253ac025a..48e7ec83489 100644 --- a/server/src/services/agent-hiring.ts +++ b/server/src/services/agent-hiring.ts @@ -17,6 +17,8 @@ import { agentCredentialService } from "./agent-credentials.js"; import { agentIdentityService } from "./agent-identity.js"; import { initializePrimaryAgent } from "./primary-agent.js"; import { resolveAgentRunnerTaskDriverForCompany, resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; +import { instanceSettingsService } from "./instance-settings.js"; +import { assertProviderApiPlacementPolicy, isProviderApiSetupDraft, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; export async function prepareAgentHire(db: Db, companyId: string, input: CreateAgentData, options?: CreateAgentOptions): Promise { await notifyAttentionRetentionWork(db, { companyId }); @@ -49,7 +51,11 @@ export async function prepareAgentHire(db: Db, companyId: string, input: CreateA ? await secretService(db).normalizeAdapterConfigForPersistence(companyId, data.adapterConfig, { adapterType }) : {}; const adapterConfig = normalizePaperclipRunnerAdapterConfig(adapterType, rawAdapterConfig); - if (adapterType === "paperclip_runner" && !["openai_dot", "grok_bot", "muse"].includes(String(adapterConfig.provider)) && + const profile = { adapterType, adapterConfig }; + const providerApiDraft = options?.setupOnly === true && isProviderApiSetupDraft(profile); + const providerApi = providerApiDraft ? null : resolveProviderApiPlacement({ agent: profile }); + if (providerApi) assertProviderApiPlacementPolicy(await instanceSettingsService(db).getGeneral()); + if (!providerApiDraft && !providerApi && adapterType === "paperclip_runner" && !["openai_dot", "grok_bot", "muse"].includes(String(adapterConfig.provider)) && (await resolveAgentRunnerTaskDriverForCompany(db, companyId, data.defaultEnvironmentId))?.warmTaskVersion === 1) { adapterConfig.lifecycleMode = "warm"; } diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index ea8fd4c4ad2..ef42fb865c5 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -24,13 +24,15 @@ import { trackIdleWork } from "./task-admission.js"; import { environmentService } from "./environments.js"; import { instanceSettingsService } from "./instance-settings.js"; import { pluginEnvironmentRequiresAgent } from "./plugin-environment-driver.js"; +import { isProviderApiSetupDraft } from "./native-runtime/provider-api-placement.js"; export { scheduleAgentLifecycle, processAgentLifecycle } from "../modules/agent-lifecycle/index.js"; export type { LifecycleDriver, LifecycleAgent } from "../modules/agent-lifecycle/index.js"; export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): lifecycle.LifecycleEffects { return { - async setupRequiresPreparation(db, companyId, environmentId) { + async setupRequiresPreparation(db, companyId, environmentId, profile) { + if (isProviderApiSetupDraft(profile)) return false; const id = environmentId ?? (await instanceSettingsService(db).get()).defaultEnvironmentId; const environment = id ? await environmentService(db).getById(id) : null; if (!environment || !await pluginEnvironmentRequiresAgent(db, environment)) return false; diff --git a/server/src/services/agent-runner-selection.test.ts b/server/src/services/agent-runner-selection.test.ts index 41b722e36e6..831cb255783 100644 --- a/server/src/services/agent-runner-selection.test.ts +++ b/server/src/services/agent-runner-selection.test.ts @@ -46,18 +46,96 @@ describe("server-owned Codex runner selection", () => { const config = { model: "caller-model", repoUrl: "https://github.com/acme/repo", repoStartingRef: "feature", runtimeEnvType: "cloud", env: { CURSOR_API_KEY: { type: "secret_ref", secretId: "saved-key", version: "latest" } }, instructionsFilePath: "/assigned/AGENTS.md", autoCreatePR: false }; expect(agentRunnerAvailability("cursor_cloud", target)).toEqual({ supportedRunners: ["paperclip", "legacy"], defaultRunner: "legacy" }); expect(resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, target })).toEqual({ adapterType: "cursor_cloud", adapterConfig: config }); + for (const runner of [undefined, "auto"] as const) { + for (const driver of ["unavailable", "sandbox", "ssh"]) { + expect(() => resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, runner, target: { driver } })) + .toThrow(expect.objectContaining({ status: 422, details: { code: "agent_runner_unavailable" } })); + } + } + expect(resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, runner: "legacy", target: { driver: "sandbox" } })) + .toEqual({ adapterType: "cursor_cloud", adapterConfig: config }); const converted = resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, runner: "paperclip", target }); expect(converted).toEqual({ adapterType: "paperclip_runner", adapterConfig: { model: config.model, repoUrl: config.repoUrl, repoRef: "feature", cloudEnvType: "cloud", env: config.env, instructionsFilePath: config.instructionsFilePath, autoCreatePR: false, provider: "cursor_cloud", lifecycleMode: "per_turn" } }); expect(resolveNewAgentRunner({ ...converted, runner: "legacy", target })).toEqual({ adapterType: "cursor_cloud", adapterConfig: config }); }); - it("rejects unsupported Cursor Cloud conversions and remote controllers without changing the original", () => { + it("rejects unsupported Cursor Cloud settings and keeps API placement independent of the selected Computer", () => { const config = { model: "caller-model", repoUrl: "https://github.com/acme/repo" }; for (const patch of [{ promptTemplate: "custom instructions" }, { runtimeEnvType: "pool" }, { model: "" }]) { expect(() => resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: { ...config, ...patch }, runner: "paperclip", target })).toThrow(); } expect(config).toEqual({ model: "caller-model", repoUrl: "https://github.com/acme/repo" }); - expect(agentRunnerAvailability("cursor_cloud", { ...target, driver: "ssh" })).toEqual({ supportedRunners: ["legacy"], defaultRunner: "legacy" }); - expect(() => resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, runner: "paperclip", target: { ...target, driver: "ssh" } })).toThrow("unavailable"); + expect(agentRunnerAvailability("cursor_cloud", { ...target, driver: "ssh" })).toEqual({ supportedRunners: ["paperclip"], defaultRunner: "paperclip" }); + expect(resolveNewAgentRunner({ adapterType: "cursor_cloud", adapterConfig: config, runner: "paperclip", target: { ...target, driver: "ssh" } })).toMatchObject({ adapterType: "paperclip_runner", adapterConfig: { provider: "cursor_cloud" } }); + }); + it("saves native Cursor Cloud without probing or binding a managed Computer", async () => { + state.settings = { defaultEnvironmentId: "managed", experimental: { enableManagedSandboxOnly: true } }; + const saved = await resolveNewAgentRunnerForCompany({} as never, "company", { + adapterType: "cursor_cloud", runner: "paperclip", defaultEnvironmentId: "managed", + adapterConfig: { model: "caller-model", repoUrl: "https://github.com/acme/repo" }, + }); + expect(saved).toMatchObject({ adapterType: "paperclip_runner", defaultEnvironmentId: null, adapterConfig: { provider: "cursor_cloud" } }); + expect(state.getEnvironment).not.toHaveBeenCalled(); + expect(state.ssh).not.toHaveBeenCalled(); + expect(state.sandboxQualification).not.toHaveBeenCalled(); + }); + it("refreshes managed adapter policy before API-only Cursor admission without a Computer", async () => { + vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "synthetic-cloud-test-token"); + state.settings = { experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + state.managedEnvironment.mockResolvedValue(null); + const input = { adapterType: "cursor_cloud", runner: "auto" as const, setupOnly: true, adapterConfig: {}, disabledTypes: [] }; + expect(await resolveNewAgentRunnerForCompany({} as never, "company", input)) + .toMatchObject({ adapterType: "paperclip_runner", defaultEnvironmentId: null, adapterConfig: { provider: "cursor_cloud" } }); + for (const disabled of ["cursor_cloud", "paperclip_runner"]) { + state.disabled = [disabled]; + await expect(resolveNewAgentRunnerForCompany({} as never, "company", input)) + .rejects.toMatchObject({ status: 422, details: { code: "agent_runner_unavailable" } }); + } + await expect(resolveNewAgentRunnerForCompany({} as never, "company", { + adapterType: "paperclip_runner", adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/acme/repo", lifecycleMode: "per_turn" }, + })).rejects.toMatchObject({ status: 422, details: { code: "agent_runner_unavailable" } }); + expect(state.managedEnvironment).not.toHaveBeenCalled(); + expect(state.getEnvironment).not.toHaveBeenCalled(); + expect(state.resolveEnvironment).not.toHaveBeenCalled(); + expect(state.ssh).not.toHaveBeenCalled(); + expect(state.sandboxQualification).not.toHaveBeenCalled(); + }); + it.each([undefined, "auto"] as const)("defaults a new managed Cursor setup draft to the native API controller without a Computer (%s)", async runner => { + state.settings = { experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + state.managedEnvironment.mockResolvedValue(null); + const saved = await resolveNewAgentRunnerForCompany({} as never, "company", { + adapterType: "cursor_cloud", runner, setupOnly: true, adapterConfig: {}, + }); + expect(saved).toMatchObject({ adapterType: "paperclip_runner", defaultEnvironmentId: null, + adapterConfig: { provider: "cursor_cloud", lifecycleMode: "per_turn" } }); + expect(state.managedEnvironment).not.toHaveBeenCalled(); + expect(state.getEnvironment).not.toHaveBeenCalled(); + }); + it.each([ + { setupOnly: false, enabled: true }, + { setupOnly: true, enabled: false }, + ])("does not use the new-draft Cursor API default outside its admission scope (%j)", async ({ setupOnly, enabled }) => { + state.settings = { experimental: { enableManagedSandboxOnly: true, enableNativeRunner: enabled } }; + state.managedEnvironment.mockResolvedValue(null); + await expect(resolveNewAgentRunnerForCompany({} as never, "company", { + adapterType: "cursor_cloud", runner: "auto", setupOnly, adapterConfig: {}, + })).rejects.toMatchObject({ status: 422, details: { code: "managed_sandbox_unavailable" } }); + }); + it("preserves an explicit Legacy choice on a held Cursor setup draft", async () => { + state.settings = { experimental: { enableManagedSandboxOnly: true, enableNativeRunner: true } }; + expect(await resolveNewAgentRunnerForCompany({} as never, "company", { + adapterType: "cursor_cloud", runner: "legacy", setupOnly: true, adapterConfig: {}, + })).toEqual({ adapterType: "cursor_cloud", adapterConfig: {} }); + expect(state.managedEnvironment).not.toHaveBeenCalled(); + }); + it("refuses native Cursor save and availability under Kubernetes-only policy without provisioning", async () => { + state.settings = { general: { executionMode: "kubernetes" } }; + await expect(resolveNewAgentRunnerForCompany({} as never, "company", { adapterType: "cursor_cloud", runner: "paperclip", + adapterConfig: { model: "caller-model", repoUrl: "https://github.com/acme/repo" } })) + .rejects.toMatchObject({ status: 422, details: { code: "provider_api_kubernetes_execution_unsupported" } }); + const selected = await resolveAgentRunnerTargetForCompany({} as never, "company"); + expect(agentRunnerAvailability("cursor_cloud", selected).supportedRunners).toEqual([]); + expect(state.getEnvironment).not.toHaveBeenCalled(); + expect(state.ensureKubernetesEnvironment).not.toHaveBeenCalled(); }); it("keeps legacy Cursor Cloud environment bindings and rejects their unsupported native conversion", () => { const config = { model: "caller-model", repoUrl: "https://github.com/acme/repo", @@ -389,7 +467,7 @@ describe("server-owned Codex runner selection", () => { it("advertises Legacy for a forced Kubernetes target before lazy environment creation", async () => { state.settings = { general: { executionMode: "kubernetes" } }; const selected = await resolveAgentRunnerTargetForCompany({} as never, "company"); - expect(selected).toEqual({ driver: "sandbox" }); + expect(selected).toEqual({ driver: "sandbox", executionMode: "kubernetes" }); expect(agentRunnerAvailability("codex_local", selected, state.disabled)).toEqual({ supportedRunners: ["legacy"], defaultRunner: "legacy" }); expect(state.kubernetesEnvironment).toHaveBeenCalledWith("company"); expect(state.ensureKubernetesEnvironment).not.toHaveBeenCalled(); diff --git a/server/src/services/agent-runner-selection.ts b/server/src/services/agent-runner-selection.ts index 68802917de8..a836da2e190 100644 --- a/server/src/services/agent-runner-selection.ts +++ b/server/src/services/agent-runner-selection.ts @@ -1,3 +1,4 @@ +import { assertProviderApiPlacementPolicy, managedCursorApiDefaultEnabled, resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import type { Db } from "@paperclipai/db"; import { agentHarnessType, paperclipRunnerProfileForHarness, isEnvironmentDriverSupportedForAdapter, paperclipRunnerSupportsPlatform, type AgentRunnerChoice, type AgentRunnerAvailability } from "@paperclipai/shared"; import { normalizeLegacyRunnerProvider, PAPERCLIP_RUNNER_DEFAULT_MODELS } from "@paperclipai/adapter-utils"; @@ -16,6 +17,7 @@ import { localAgentRunnerTarget } from "../lib/agent-runner-platform.js"; import { normalizeClaudeRunnerOptions, type AgentConfigurationWarning } from "@paperclipai/shared"; import { usesManagedCloudSandboxImage } from "./sandbox-snapshot-qualification.js"; import { environmentCustomImageTemplateMatchesBaseConfig } from "./environment-custom-image-runtime.js"; +import type { ExecutionPolicy } from "./execution-allowlist.js"; export type RunnerTarget = { driver: string; platform?: string; architecture?: string; libc?: "glibc" | "musl" | "unknown"; glibcVersion?: string; @@ -23,6 +25,7 @@ export type RunnerTarget = { runnerTasks?: boolean; managedCloudImage?: boolean; warmTaskVersion?: 1; + executionMode?: ExecutionPolicy["executionMode"]; }; export const AGENT_RUNNER_SSH_PLATFORM_COMMAND = "uname -s; uname -m; if paperclip_glibc_version=$(getconf GNU_LIBC_VERSION 2>/dev/null) && [ -n \"$paperclip_glibc_version\" ]; then printf 'glibc\\n%s\\n' \"$paperclip_glibc_version\"; else case \"$(ldd --version 2>&1)\" in *musl*) printf 'musl\\n' ;; *) printf 'unknown\\n' ;; esac; fi"; export interface AgentRunnerSelection { @@ -52,10 +55,15 @@ function runnerAdaptersAvailable(harness: string, driver: string, disabledTypes: export function agentRunnerAvailability(harness: string, target: RunnerTarget = localAgentRunnerTarget(), disabledTypes: string[] = localDisabledTypes()): AgentRunnerAvailability { if (harness === "cursor_cloud") { - const supported = target.driver === "local" && runnerAdaptersAvailable(harness, target.driver, disabledTypes) - && (!target.platform || paperclipRunnerSupportsPlatform(harness, target.platform, target.architecture ?? "", target.libc, target.glibcVersion)); - return { supportedRunners: supported ? ["paperclip", "legacy"] : ["legacy"], defaultRunner: "legacy" }; + if (target.executionMode === "kubernetes") return { supportedRunners: [], defaultRunner: "legacy" }; + const controller = localAgentRunnerTarget(); + const supported = runnerAdaptersAvailable(harness, "local", disabledTypes) + && paperclipRunnerSupportsPlatform(harness, controller.platform, controller.architecture, controller.libc, controller.glibcVersion); + const legacy = isEnvironmentDriverSupportedForAdapter(harness, target.driver); + return { supportedRunners: [...(supported ? ["paperclip" as const] : []), ...(legacy ? ["legacy" as const] : [])], + defaultRunner: legacy || !supported ? "legacy" : "paperclip" }; } + if (target.driver === "unavailable") return { supportedRunners: [], defaultRunner: "legacy" }; if (target.runnerTasks || target.managedCloudImage) { return runnerAdaptersAvailable(harness, target.driver, disabledTypes) ? { supportedRunners: target.runnerTasks ? ["paperclip"] : ["paperclip", "legacy"], defaultRunner: "paperclip" } @@ -187,10 +195,16 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; if (native && (input.runner === undefined || input.runner === "paperclip" || (harness !== "codex_local" && input.runner !== "legacy"))) return { adapterType, adapterConfig: normalizeLegacyRunnerProvider(config) }; const availability = agentRunnerAvailability(harness, input.target, disabledTypes); const choice = input.runner ?? "auto"; - const useNative = choice === "paperclip" || (choice === "auto" && availability.defaultRunner === "paperclip"); + const useNative = choice === "paperclip" || (choice === "auto" && harness !== "cursor_cloud" && availability.defaultRunner === "paperclip"); if (!useNative) { if (harness === "cursor_cloud") { - if (!native) return { adapterType, adapterConfig: config }; + if (!native) { + if (choice !== "legacy" && !availability.supportedRunners.includes("legacy")) throw unprocessable( + "Automatic Cursor selection is unavailable for this environment. Choose a supported runner in Advanced.", + { code: "agent_runner_unavailable" }, + ); + return { adapterType, adapterConfig: config }; + } if (config.lifecycleMode !== undefined && config.lifecycleMode !== "per_turn") incompatible(["lifecycleMode"]); for (const [current, legacy] of [["cloudEnvType", "runtimeEnvType"], ["cloudEnvName", "runtimeEnvName"], ["repoRef", "repoStartingRef"], ["prUrl", "repoPullRequestUrl"]]) { if (config[current] !== undefined) config[legacy] = config[current]; @@ -363,7 +377,33 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; } export async function resolveNewAgentRunnerForCompany(db: Db, companyId: string, input: AgentRunnerSelection & { defaultEnvironmentId?: string | null }) { - input = { ...input, disabledTypes: await adapterSettingsService(db).getDisabledTypes() }; + const disabledTypes = await adapterSettingsService(db).getDisabledTypes(); + input = { ...input, disabledTypes }; + const cursor = agentHarnessType(input.adapterType ?? "process", input.adapterConfig) === "cursor_cloud"; + const nativeCursor = cursor && input.runner !== "legacy" + && (input.adapterType === "paperclip_runner" || input.runner === "paperclip"); + const cursorSetupDefault = cursor && input.setupOnly === true + && (input.runner === undefined || input.runner === "auto"); + if (nativeCursor || cursorSetupDefault) { + const { instanceSettingsService } = await import("./instance-settings.js"); + const settings = await instanceSettingsService(db).get(); + // Only a held new setup may adopt the API-only managed default. Existing + // legacy agents still require an explicit conversion decision. + const useSetupDefault = cursorSetupDefault && managedCursorApiDefaultEnabled(settings.experimental); + if (nativeCursor || useSetupDefault) { + if (disabledTypes.includes("cursor_cloud") || disabledTypes.includes("paperclip_runner")) { + throw unprocessable("Cursor Cloud is disabled on this instance. Enable its adapter or choose another harness.", { code: "agent_runner_unavailable" }); + } + assertProviderApiPlacementPolicy(settings.general ?? {}); + const selected = resolveNewAgentRunner({ ...input, ...(useSetupDefault ? { runner: "paperclip" as const } : {}), target: localAgentRunnerTarget() }); + try { if (!input.setupOnly) resolveProviderApiPlacement({ agent: selected }); } + catch (error) { + if (error instanceof PaperclipRunnerProviderProfileError) throw unprocessable(error.message, { code: error.code }); + throw error; + } + return { ...selected, defaultEnvironmentId: null }; + } + } if (!paperclipRunnerProfileForHarness(agentHarnessType(input.adapterType ?? "process", input.adapterConfig)) || input.runner === "legacy" || (input.adapterType === "paperclip_runner" && (input.runner === undefined || input.runner === "paperclip"))) return resolveNewAgentRunner(input); const target = await resolveAgentRunnerTargetForCompany(db, companyId, input.defaultEnvironmentId, input.disabledTypes); @@ -372,7 +412,8 @@ export async function resolveNewAgentRunnerForCompany(db: Db, companyId: string, export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: string, defaultEnvironmentId?: string | null, disabledTypes?: string[]): Promise { await waitForExternalAdapters(); - const { environment, driver } = await resolveAgentRunnerEnvironmentForCompany(db, companyId, defaultEnvironmentId); + const { environment, driver, executionMode } = await resolveAgentRunnerEnvironmentForCompany(db, companyId, defaultEnvironmentId); + const withPolicy = (value: RunnerTarget): RunnerTarget => ({ ...value, ...(executionMode ? { executionMode } : {}) }); let target: RunnerTarget = driver === "local" ? localAgentRunnerTarget() : { driver }; if (driver === "plugin" && environment) { @@ -386,16 +427,16 @@ export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: stri environmentId: environment.id, provider: typeof environment.config.provider === "string" ? environment.config.provider : null, }); - if (!template?.templateRef) return { driver, managedCloudImage: true }; + if (!template?.templateRef) return withPolicy({ driver, managedCloudImage: true }); try { const parsed = parseEnvironmentDriverConfig(environment); if (parsed.driver === "sandbox" && !environmentCustomImageTemplateMatchesBaseConfig({ template, baseConfig: parsed.config, secretRefExcludePaths: await resolveSandboxProviderSecretRefPaths(db, template.provider), - })) return { driver, managedCloudImage: true }; + })) return withPolicy({ driver, managedCloudImage: true }); } catch { - return { driver }; + return withPolicy({ driver }); } } const { readSandboxPlatformQualification } = await import("./environment-probe.js"); @@ -413,7 +454,7 @@ export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: stri const [os = "", arch = "", libc, glibcVersion] = output.trim().split(/\r?\n/); target = { driver, platform: os === "Linux" ? "linux" : os === "Darwin" ? "darwin" : "unknown", architecture: arch === "x86_64" ? "x64" : ["aarch64", "arm64"].includes(arch) ? "arm64" : arch, ...(os === "Linux" && libc !== undefined ? { libc: libc === "glibc" || libc === "musl" ? libc : "unknown", ...(libc === "glibc" && glibcVersion !== undefined ? { glibcVersion } : {}) } : {}) }; } - return target; + return withPolicy(target); } export async function resolveAgentRunnerEnvironmentForCompany(db: Db, companyId: string, defaultEnvironmentId?: string | null) { @@ -432,16 +473,19 @@ async function resolveAgentEnvironmentForCompany(db: Db, companyId: string, defa const { instanceSettingsService } = await import("./instance-settings.js"); const { environmentService } = await import("./environments.js"); const settings = await instanceSettingsService(db).get(); + const executionMode = settings.general?.executionMode; const envs = environmentService(db); const environmentId = defaultEnvironmentId ?? settings.defaultEnvironmentId; let environment = environmentId ? await envs.getById(environmentId) : null; if (settings.experimental?.enableManagedSandboxOnly && (!environment || environment.driver === "local")) { environment = await envs.findManagedSandboxEnvironment(companyId); - if (!environment) throw unprocessable("The managed sandbox is unavailable. Restore Paperclip Computer and retry.", { code: "managed_sandbox_unavailable" }); + if (!environment) throw unprocessable("The managed sandbox is unavailable. Restore Paperclip Computer and retry.", { + code: "managed_sandbox_unavailable", ...(executionMode ? { executionMode } : {}), + }); } if (settings.general?.executionMode === "kubernetes") { environment = await envs.findKubernetesEnvironment(companyId); - if (!environment) return { environment: null, driver: "sandbox" }; + if (!environment) return { environment: null, driver: "sandbox", executionMode }; } if (environmentId && !environment) throw unprocessable("Environment not found."); if (environment) { @@ -449,7 +493,7 @@ async function resolveAgentEnvironmentForCompany(db: Db, companyId: string, defa if (boundCompanyIds.length > 0 && !boundCompanyIds.includes(companyId)) throw forbidden("The selected environment belongs to another company.", { code: "environment_company_mismatch" }); if (environment.status === "archived") throw unprocessable("Environment is archived."); } - return { environment, driver: environment?.driver ?? "local" }; + return { environment, driver: environment?.driver ?? "local", executionMode }; } // The plugin's manifest declaration decides whether its environment runs diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index 06b87a6665a..cbc23b0e7a7 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -79,7 +79,10 @@ export async function appendHeartbeatRunEvent( input: AppendHeartbeatRunEventInput, ): Promise { const projectsRetention = input.eventType === "lifecycle" && Boolean(input.message?.startsWith(RETRY_EXHAUSTION_MESSAGE_PREFIX)); - const projectsRecovery = ["native.process_start_requested", "native.process_identity_recorded", "native.local_process_stopped"].includes(input.eventType); + // Terminal proof may arrive after controller retirement and a quiet Stop + // check. Register the scoped wake before writing it, in the same transaction. + const projectsRecovery = ["native.process_start_requested", "native.process_identity_recorded", "native.local_process_stopped"].includes(input.eventType) + || Boolean(input.nativeSource && ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(input.eventType)); const projectsReassessment = readNativeEventEvidenceOutcome(input.payload?.prpEvent) !== null; const projectsProgress = isSafeNativeChatProgressEventType(input.eventType); const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 4ddcfc16a7e..68bf5fd630a 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js"; import { registerDeferredReceiptWork } from "./heartbeat/deferred-receipt-work.js"; import { reconcileTaskChatRetryReceipts } from "./task-retry.js"; import { registerIssueRecoveryWork } from "./recovery/work.js"; @@ -2266,7 +2267,8 @@ export function heartbeatService( const defaultIsolatedWorkspacesEnabled = isolatedWorkspacesEnabled && experimentalInstanceSettings.enableIsolatedWorkspacesByDefault; - const externalProvider = heartbeatExternalProvider({ run, agent }); + const providerApiPlacement = resolveProviderApiPlacement({ run, agent }); + const externalProvider = heartbeatExternalProvider({ run, agent }) ?? providerApiPlacement?.provider; const [taskWorkspaceIntentRow] = issueId && !externalProvider ? await db.select({ intent: issues.workspaceSelection }).from(issues) .where(and(eq(issues.id, issueId), eq(issues.companyId, agent.companyId))) : []; const freshWorkspace = await readFreshWorkspaceReadiness(db, run); @@ -2968,7 +2970,7 @@ export function heartbeatService( runtimeConfig = { ...runtimeConfig, env: gitExecutionEnv }; for (const key of MANAGED_GITHUB_TOKEN_KEYS) secretKeys.add(key); context.githubAuthenticationMode = useHostGitHub ? "host" : "managed"; - if (!useHostGitHub) { + if (!useHostGitHub && !providerApiPlacement) { const githubLaunchers = await timePreparation("github_launchers", () => prepareHeartbeatGitHubLaunchers({ native: agent.adapterType === "paperclip_runner", githubConfigured: githubSelection.configured, @@ -2990,7 +2992,7 @@ export function heartbeatService( runtimeConfig = { ...runtimeConfig, env: githubLaunchers.env }; secretKeys.add("PAPERCLIP_GITHUB_BROKER_TOKEN"); } - context.paperclipEnvironment = { + context.paperclipEnvironment = selectedEnvironment && activeEnvironmentLease ? { id: selectedEnvironment.id, name: selectedEnvironment.name, driver: selectedEnvironment.driver, @@ -3017,7 +3019,7 @@ export function heartbeatService( : undefined, } : {}), - }; + } : { placement: providerApiPlacement }; await db .update(heartbeatRuns) .set({ @@ -3244,8 +3246,8 @@ export function heartbeatService( executionWorkspace, persistedExecutionWorkspace, executionTarget, - environmentDriver: selectedEnvironment.driver, - leaseMetadata: activeEnvironmentLease.lease.metadata, + environmentDriver: selectedEnvironment?.driver ?? "provider_api", + leaseMetadata: activeEnvironmentLease?.lease.metadata ?? {}, })); const adapterEnv = Object.fromEntries( Object.entries(parseObject(runtimeConfig.env)).filter( @@ -3932,6 +3934,9 @@ export function heartbeatService( await settleNativeRunProcess(db, run.id).catch(error => { logger.warn({ err: error, runId: run.id }, "failed to release settled native process identity"); }); + await acknowledgeRemoteStop(run.id, run.companyId).catch(error => { + logger.warn({ err: error, runId: run.id }, "failed to acknowledge Stop after native process settlement"); + }); } await db.update(heartbeatRuns).set({ executionStage: sql`case when ${heartbeatRuns.status} = 'cancelled' then 'settled' else ${heartbeatRuns.executionStage} end`, diff --git a/server/src/services/heartbeat/environment-selection.ts b/server/src/services/heartbeat/environment-selection.ts index 41ba6e44d20..58c2713c9c2 100644 --- a/server/src/services/heartbeat/environment-selection.ts +++ b/server/src/services/heartbeat/environment-selection.ts @@ -1,3 +1,4 @@ +import { assertProviderApiPlacementPolicy, resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js"; import { heartbeatExternalProvider, nativeExecutionUsesTaskWorkspace } from "./external-provider.js"; import { heartbeatRuns, issues, type Db, type agents } from "@paperclipai/db"; import { and, eq } from "drizzle-orm"; @@ -106,16 +107,18 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron : null; // This retained name covers both external bridges; neither adopts task files. const isDotRun = heartbeatExternalProvider({ run, agent }) !== null; + const providerApiPlacement = resolveProviderApiPlacement({ run, agent }); + const controllerOnly = isDotRun || providerApiPlacement !== null; const persistedNativeExecutionWorkspaceId = - persistedNativeExecutionInput?.binding.executionWorkspaceId ?? null; + providerApiPlacement ? null : persistedNativeExecutionInput?.binding.executionWorkspaceId ?? null; const fresh = input.task.freshWorkspace?.mode === "fresh_isolated_workspace"; - const requestedExecutionWorkspaceId = isDotRun ? null : + const requestedExecutionWorkspaceId = controllerOnly ? null : persistedNativeExecutionWorkspaceId ?? readNonEmptyString(issueRef?.executionWorkspaceId); let existingExecutionWorkspace = requestedExecutionWorkspaceId ? await executionWorkspacesSvc.getStoredById(agent.companyId, requestedExecutionWorkspaceId) : null; - if (existingExecutionWorkspace && issueId && !isDotRun && !fresh + if (existingExecutionWorkspace && issueId && !controllerOnly && !fresh && existingExecutionWorkspace.id === readNonEmptyString(issueRef?.executionWorkspaceId) && (existingExecutionWorkspace.status === "archived" || existingExecutionWorkspace.status === "cleanup_failed") && (readGitCacheExpiration(existingExecutionWorkspace as Parameters[0])?.workspaceRemoved @@ -154,7 +157,7 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron const workspaceReuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({ issueExecutionWorkspaceId: requestedExecutionWorkspaceId, - issueExecutionWorkspacePreference: isDotRun ? null : nativeRecoveryExecutionWorkspaceId + issueExecutionWorkspacePreference: controllerOnly ? null : nativeRecoveryExecutionWorkspaceId ? "reuse_existing" : (issueRef?.executionWorkspacePreference ?? null), existingExecutionWorkspaceStatus: @@ -168,11 +171,11 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron : null; const workspaceAuthorizationActor = { type: "agent" as const, agentId: agent.id, companyId: agent.companyId, source: "agent_jwt" as const, runId: run.id, onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId }; - if (!isDotRun && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) { + if (!controllerOnly && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) { const workspaceRequiringAccess = reusableExistingExecutionWorkspace ?? boundSourceWorkspace; if (workspaceRequiringAccess && issueId) { await assertTaskWorkspaceExecutionAccess(db, workspaceAuthorizationActor, agent.companyId, { issueId, workspaceId: workspaceRequiringAccess.id }); - } else if (!isDotRun && executionProjectId && (hasSelectedWorkspaceSource || workspaceProjectContext?.hasWorkspace)) { + } else if (!controllerOnly && executionProjectId && (hasSelectedWorkspaceSource || workspaceProjectContext?.hasWorkspace)) { await assertTaskWorkspaceSourceProjectAccess(db, workspaceAuthorizationActor, agent.companyId, executionProjectId); } } @@ -191,10 +194,12 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron const managedSandboxOnly = (await instanceSettings.getExperimental()).enableManagedSandboxOnly === true; - const managedSandboxEnvironment = managedSandboxOnly + const managedSandboxEnvironment = managedSandboxOnly && !providerApiPlacement ? await environmentsSvc.findManagedSandboxEnvironment(agent.companyId) : null; - const environmentResolution = resolveExecutionWorkspaceEnvironmentId({ + const environmentResolution = providerApiPlacement + ? { environmentId: null, source: "default" as const } + : resolveExecutionWorkspaceEnvironmentId({ lowTrustIssueEnvironmentId: trustPreset.kind === "low_trust_review" ? parseIssueExecutionWorkspaceSettings(issueContext?.executionWorkspaceSettings, {includeEnvironmentId: true})?.environmentId : null, agentDefaultEnvironmentId: agent.defaultEnvironmentId, @@ -218,8 +223,9 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron }; const executionForcedToKubernetes = isExecutionForcedToKubernetes(executionPolicy); + if (providerApiPlacement) assertProviderApiPlacementPolicy(executionPolicy); let selectedEnvironmentId = environmentResolution.environmentId; - if (executionForcedToKubernetes) { + if (executionForcedToKubernetes && !providerApiPlacement) { let kubernetesEnvironment = await environmentsSvc.findKubernetesEnvironment(agent.companyId); if (!kubernetesEnvironment) { @@ -292,7 +298,7 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron : selectedEnvironmentId ? await environmentsSvc.getById(selectedEnvironmentId) : null; - const nativeChatWorkspaceScope = isDotRun ? null : await findNativeChatWorkspaceScope(db, { + const nativeChatWorkspaceScope = controllerOnly ? null : await findNativeChatWorkspaceScope(db, { adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver ?? null, companyId: agent.companyId, @@ -391,7 +397,7 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron // project or issue settings request serialization. Sandbox protection still uses // the existing holder staleness and workspace_busy retry ladder. if ( - !isDotRun && issueRef?.projectWorkspaceId && + !controllerOnly && issueRef?.projectWorkspaceId && effectiveExecutionWorkspaceMode === "shared_workspace" ) { const workspaceHolder = await findSharedWorkspaceHolder({ @@ -472,18 +478,18 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron ); } } - if (!isDotRun && taskWorkspaceIntent && !boundSourceWorkspace && !nativeRecoveryExecutionWorkspaceId) { + if (!controllerOnly && taskWorkspaceIntent && !boundSourceWorkspace && !nativeRecoveryExecutionWorkspaceId) { await executionWorkspacesSvc.validateStoredSelectionForRun({ companyId: agent.companyId, actor: { type: "agent", agentId: agent.id, companyId: agent.companyId, runId: run.id, source: "agent_jwt", onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId }, selection: taskWorkspaceIntent.selection, issueId, preserveImplicitDefaultSettings: ["project", "task_default"].includes(taskWorkspaceIntent.source) }); } - const explicitTaskDirectory = !isDotRun && taskWorkspaceIntent?.selection.kind === "task_directory"; + const explicitTaskDirectory = !controllerOnly && taskWorkspaceIntent?.selection.kind === "task_directory"; const explicitDirectoryStrategies = [ ...(reusableExistingExecutionWorkspace?.providerType === "adapter_managed" ? [config.workspaceStrategy] : []), issueAssigneeOverrides?.adapterConfig?.workspaceStrategy, projectExecutionWorkspacePolicy?.workspaceStrategy, issueExecutionWorkspaceSettings?.workspaceStrategy]; - const boundTaskDirectory = !isDotRun && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null && + const boundTaskDirectory = !controllerOnly && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null && requestedShouldReuseExisting && requestedExecutionWorkspaceMode === "isolated_workspace" && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId && taskWorkspaceIntent?.selection.kind !== "configured_source" && @@ -503,7 +509,7 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron if (retainedDirectory && !boundTaskDirectory) throw new WorkspaceValidationFailure("The saved Retry task directory is unavailable.", { workspaceValidation: { reason: "isolated_task_directory_binding_mismatch", issueId }, }); - const useIsolatedTaskDirectory = !isDotRun && issueRef !== null && (reuseTaskDirectory || (fresh && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId) || shouldUseIsolatedTaskDirectory({ + const useIsolatedTaskDirectory = !controllerOnly && issueRef !== null && (reuseTaskDirectory || (fresh && !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId) || shouldUseIsolatedTaskDirectory({ trustPreset: trustPreset.kind, environmentDriver: selectedEnvironmentForConfig?.driver ?? null, mode: requestedExecutionWorkspaceMode, @@ -520,6 +526,7 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron persistedRunnerProfile, persistedNativeExecutionInput, isDotRun, + providerApiPlacement, existingExecutionWorkspace, nativeRecoveryExecutionWorkspaceId, workspaceReuseRequest, diff --git a/server/src/services/heartbeat/external-provider.ts b/server/src/services/heartbeat/external-provider.ts index 6dfb45a4516..5cb656a23a0 100644 --- a/server/src/services/heartbeat/external-provider.ts +++ b/server/src/services/heartbeat/external-provider.ts @@ -24,5 +24,5 @@ type TaskWorkspaceExecution = Exclude)?.deadlineAt)) || Date.now(), })); } else { + // Retirement may commit before Stop acknowledgement. This is only a + // work hint; the owner rechecks the complete provider/process proof. + const pendingApiStop = and(eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.driverKind, "cursor_cloud_api"), + sql`${heartbeatRuns.runnerProfileJson}->>'backend' = 'cursor_cloud_api'`, + sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'requested'`); const candidates = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }) .from(heartbeatRuns).leftJoin(nativeRunFinalizations, and(eq(nativeRunFinalizations.runId, heartbeatRuns.id), @@ -76,9 +81,9 @@ export function registerLiveRecoveryWork(db: Db, coordinator: ReturnType'nativeQuestionCancellation' is null`) : and(eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["failed", "cancelled", "timed_out", "interrupted"]), - isNotNull(heartbeatRuns.finishedAt), or(isNotNull(heartbeatRuns.processPid), - isNotNull(heartbeatRuns.processGroupId), isNotNull(heartbeatRuns.processStartedAt)), - settledNativeProcessIdentityCondition(db)), + isNotNull(heartbeatRuns.finishedAt), or(pendingApiStop, + and(or(isNotNull(heartbeatRuns.processPid), isNotNull(heartbeatRuns.processGroupId), + isNotNull(heartbeatRuns.processStartedAt)), settledNativeProcessIdentityCondition(db)))), )).orderBy(asc(heartbeatRuns.id)).limit(100); rows = candidates.map(row => ({ id: row.id, companyId: row.companyId, dueAt: kind === "native" ? Math.max(Date.now(), row.nextAttemptAt?.getTime() ?? 0, row.leaseExpiresAt?.getTime() ?? 0) : Date.now() })); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 48d1a532c27..40351ddfb1a 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -2775,7 +2775,8 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD scope.onDeferredDeadline(Date.now() + 30_000); return; } - await releaseSettledNativeRunProcess(db, scope.runId, scope); + const outcome = await releaseSettledNativeRunProcess(db, scope.runId, scope); + if (outcome !== "pending") await acknowledgeRemoteStop(scope.runId, scope.companyId); }, orphanLease: scope => sweepOrphanedActiveLeases({ ...scope, backoffMs: 5 * 60_000 }), shutdown: scope => sweepLegacyShutdownWorkspaceSettlements(db, new Date(), scope), diff --git a/server/src/services/heartbeat/run-control.test.ts b/server/src/services/heartbeat/run-control.test.ts index bbcd348b1ca..fef333bc61a 100644 --- a/server/src/services/heartbeat/run-control.test.ts +++ b/server/src/services/heartbeat/run-control.test.ts @@ -1,6 +1,6 @@ import { randomUUID } from "node:crypto"; import type { Db } from "@paperclipai/db"; -import { activityLog, agents, agentWakeupRequests, caseEvents, caseHandoffs, cases, companies, environmentLeases, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db"; +import { activityLog, agents, agentWakeupRequests, caseEvents, caseHandoffs, cases, companies, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db"; import { eq, inArray, sql } from "drizzle-orm"; import { afterEach, describe, expect, it, vi } from "vitest"; import { runningProcesses } from "../../adapters/index.js"; @@ -24,9 +24,18 @@ import { caseHandoffService } from "../case-handoffs.js"; import { withQueuedCommentIdsInWakePayload } from "../issue-queued-comment-queue.js"; import { instanceSettingsService } from "../instance-settings.js"; import { issueService } from "../issues.js"; +import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js"; +import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js"; +import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; +import { acknowledgedNativeApiStopExecutionHasStopped } from "../acknowledged-native-stop.js"; +import { nativeSha256 } from "../native-runtime/canonical.js"; import { callbacks, runControlPostgresSupport, setupRunControlDatabase } from "./__tests__/run-control-fixtures.js"; vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() })); +vi.mock("../acknowledged-native-stop.js", async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, acknowledgedNativeApiStopExecutionHasStopped: vi.fn(actual.acknowledgedNativeApiStopExecutionHasStopped) }; +}); vi.mock("../adapter-execution-control.js", async (importOriginal) => { const actual = await importOriginal(); return { @@ -628,6 +637,7 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat releasedAt: proof === "missing" ? null : new Date(), cleanupStatus: proof === "missing" ? null : "success", metadata: proof === "missing" ? {} : { remoteExecutionTermination: receipt } }); const control = createHeartbeatRunControl(db, callbacks(db)); + vi.mocked(publishLiveEvent).mockClear(); await control.acknowledgeRemoteStop(run.id, company.id); await control.acknowledgeRemoteStop(run.id, company.id); const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); @@ -635,6 +645,149 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat if (proof === "confirmed") expect(stopped.resultJson?.executionCancellation).toMatchObject({ proof: "provider_termination_receipt" }); expect(stopped.resultJson?.workspaceRestoreFailure).toBe("restore_failed"); expect(stopped.resultJson?.workspaceRestoreRecovery).toMatchObject({ leaseIds: [leaseId] }); + expect(vi.mocked(publishLiveEvent).mock.calls).toEqual(proof === "confirmed" ? [[{ + companyId: company.id, + type: "heartbeat.run.status", + payload: expect.objectContaining({ runId, agentId: agent.id, status: "cancelled" }), + }]] : []); + }); + + it.each(["confirmed", "missing_process_stop", "later_start", "missing_terminal", "unproven_terminal", + "foreign_terminal", "foreign_binding", "missing_placement", "unexpected_lease", "running_pid", + "wrong_intent", "pending_ack", "unresolved_mode", "missing_hash", "wrong_hash", "malformed_terminal", + "race_event_start", "race_profile_binding", "race_benign_event", + "natural_completed", "natural_cancelled", "natural_error", "natural_expired", + "mismatched_terminal_status", "local_failure", "unproven_cancelled", + "pre_stop_completed", "pre_stop_cancelled", "pre_stop_error", "pre_stop_expired", "pre_stop_interrupted", + "pre_stop_later_turn", "pre_stop_unproven", + ] as const)("publishes an API-only native Stop only after its exact provider and controller stopped: %s", async proof => { + const { company, agent } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "API-only Stop", assigneeAgentId: agent.id }).returning(); + const runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID(); + const recordedAt = new Date(Date.now() - 1_000).toISOString(); + const nativeExecutionInput = buildNativeExecutionInput({ companyId: company.id, agentId: agent.id, + runId: proof === "foreign_binding" ? randomUUID() : runId, provider: "cursor_cloud", model: "explicit-model", + cursorCloud: { schema: "paperclip.cursor-cloud.config.v1", repos: [{ url: "https://github.com/example/qualification" }] }, + issue: { id: issue.id, identifier: "RC-1", title: issue.title, description: null, workMode: "standard" }, + taskPrompt: "Stop the foreground command", normalizedSessionId: sessionId, + workspace: { id: randomUUID(), cwd: "/private/controller", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Stop the foreground command", criteria: [{ id: "objective", requirement: "Stop the foreground command" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); + const profile = { backend: "cursor_cloud_api", nativeExecutionInput, + ...(proof === "missing_placement" ? {} : { executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } }) }; + const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId: company.id, agentId: agent.id, + nativeIssueId: issue.id, runtimeMode: proof === "unresolved_mode" ? "legacy" : "native", driverKind: "cursor_cloud_api", + nativeSessionId: sessionId, runnerInstanceId: runnerId, runnerProfileJson: profile, + status: "cancelled", finishedAt: new Date(), processPid: proof === "running_pid" ? process.pid : null, + resultJson: { executionCancellation: { state: "requested", requestedAt: recordedAt }, + cancelledByActorType: "user", cancelledByUserId: "operator", nativeCancellation: { + schema: "paperclip.native-cancellation.v1", runId: proof === "wrong_intent" ? randomUUID() : runId, + companyId: company.id, issueId: issue.id, scope: "run", reasonCode: "cancellation_run_only", + dispatchState: proof === "pending_ack" ? "pending" : "acknowledged", dispatched: true, + effects: ["release_run_resources"], recordedAt, acknowledgedAt: new Date().toISOString(), + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), + } }, + }).returning(); + if (proof === "unexpected_lease") await db.insert(environmentLeases).values({ companyId: company.id, issueId: issue.id, + heartbeatRunId: runId, provider: "local", status: "released", leasePolicy: "retain_on_failure", + releasedAt: new Date(), cleanupStatus: "success" }); + if (proof !== "missing_terminal") { + const preStop = proof.startsWith("pre_stop_"); + const terminalProof = proof === "pre_stop_later_turn" || proof === "pre_stop_unproven" ? "natural_completed" + : preStop ? `natural_${proof.slice("pre_stop_".length)}` : proof; + const terminalAt = preStop ? new Date(Date.parse(recordedAt) - 1_000) : new Date(); + const terminalType = terminalProof === "natural_cancelled" || terminalProof === "unproven_cancelled" ? "turn.cancelled" + : ["natural_error", "natural_expired", "local_failure"].includes(terminalProof) ? "turn.failed" + : terminalProof === "natural_completed" || terminalProof === "mismatched_terminal_status" ? "turn.completed" : "turn.interrupted"; + const terminalError = terminalProof === "natural_error" ? { code: "cursor_cloud_error", message: "Cursor Cloud run failed" } + : terminalProof === "natural_expired" ? { code: "cursor_cloud_expired", message: "Cursor Cloud run expired" } + : terminalProof === "local_failure" ? { code: "process_lost", message: "Synthetic local process loss" } : null; + const event = { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: terminalType, + sourceKind: "runner", sourceInstanceId: runnerId, sourceEventId: `${runnerId}:${runId}:1`, sourceSeq: 1, + runId, turnId: "api-stop-turn", normalizedSessionId: proof === "foreign_terminal" ? randomUUID() : sessionId, + emittedAt: terminalAt.toISOString(), priority: 1, + payload: { status: terminalProof === "mismatched_terminal_status" ? "interrupted" : terminalType.slice("turn.".length), error: terminalError, + ...(terminalProof === "natural_cancelled" ? { providerTerminalState: "completed", reason: "cancelled_after_provider_completed" } : {}), + ...(terminalProof === "unproven_cancelled" ? { providerTerminalState: "unknown", reason: "operator_cancel_requested" } : {}), + ...(proof === "unproven_terminal" || proof === "pre_stop_unproven" ? { providerTerminalObserved: false } : {}) } }; + await appendHeartbeatRunEvent(db, { companyId: company.id, agentId: agent.id, runId, + eventType: event.eventType, payload: { prpEvent: event }, nativeSource: { + sourceInstanceId: runnerId, sourceEventId: event.sourceEventId, sourceSeq: 1, + protocolSchemaVersion: 1, canonicalPayload: event, + } }); + if (preStop) await db.update(heartbeatRunEvents).set({ createdAt: terminalAt }).where(eq(heartbeatRunEvents.runId, runId)); + if (proof === "pre_stop_later_turn") { + const later = { ...event, eventType: "turn.started", sourceEventId: `${runnerId}:${runId}:2`, sourceSeq: 2, + turnId: "later-api-turn", emittedAt: new Date().toISOString(), payload: { status: "running" } }; + await appendHeartbeatRunEvent(db, { companyId: company.id, agentId: agent.id, runId, + eventType: later.eventType, payload: { prpEvent: later }, nativeSource: { + sourceInstanceId: runnerId, sourceEventId: later.sourceEventId, sourceSeq: 2, + protocolSchemaVersion: 1, canonicalPayload: later, + } }); + } + if (["missing_hash", "wrong_hash", "malformed_terminal"].includes(proof)) { + const malformed = { ...event, priority: undefined }; + await db.update(heartbeatRunEvents).set({ + sourcePayloadSha256: proof === "missing_hash" ? null : proof === "wrong_hash" ? "0".repeat(64) : nativeSha256(malformed), + ...(proof === "malformed_terminal" ? { payload: { prpEvent: malformed } } : {}), + }).where(eq(heartbeatRunEvents.runId, runId)); + } + } + const control = createHeartbeatRunControl(db, callbacks(db)); + vi.mocked(publishLiveEvent).mockClear(); + if (proof === "confirmed") { + await control.acknowledgeRemoteStop(run.id, company.id); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0] + .resultJson?.executionCancellation).toMatchObject({ state: "requested" }); + expect(publishLiveEvent).not.toHaveBeenCalled(); + } + if (proof !== "missing_process_stop") await appendHeartbeatRunEvent(db, { companyId: company.id, + agentId: agent.id, runId, eventType: "native.local_process_stopped", payload: { processPid: 424242, processGroupId: 424242 } }); + if (proof === "later_start") await appendHeartbeatRunEvent(db, { companyId: company.id, + agentId: agent.id, runId, eventType: "native.process_start_requested", payload: { targetKind: "local" } }); + if (proof.startsWith("race_")) { + const actual = await vi.importActual("../acknowledged-native-stop.js"); + vi.mocked(acknowledgedNativeApiStopExecutionHasStopped).mockImplementationOnce(async (...args) => { + const proven = await actual.acknowledgedNativeApiStopExecutionHasStopped(...args); + expect(proven).toBe(true); + if (proof === "race_event_start" || proof === "race_benign_event") { + const [before] = await db.select({ cursor: heartbeatRuns.nextEventSeq }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await appendHeartbeatRunEvent(db, { companyId: company.id, agentId: agent.id, runId, + eventType: proof === "race_event_start" ? "native.process_start_requested" : "lifecycle", + payload: proof === "race_event_start" ? { targetKind: "local" } : { reason: "synthetic finalization log" } }); + const [after] = await db.select({ cursor: heartbeatRuns.nextEventSeq }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(after.cursor).toBe(before.cursor + 1); + } else { + await db.update(heartbeatRuns).set({ runnerProfileJson: { + ...profile, + nativeExecutionInput: { ...nativeExecutionInput, binding: { ...nativeExecutionInput.binding, issueId: randomUUID() } }, + } }).where(eq(heartbeatRuns.id, runId)); + } + return proven; + }); + } + await control.acknowledgeRemoteStop(run.id, company.id); + if (proof === "race_benign_event") expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0] + .resultJson?.executionCancellation).toMatchObject({ state: "acknowledged" }); + await control.acknowledgeRemoteStop(run.id, company.id); + const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + const acknowledged = ["confirmed", "race_benign_event", "natural_completed", "natural_cancelled", "natural_error", "natural_expired", + "pre_stop_completed", "pre_stop_cancelled", "pre_stop_error", "pre_stop_expired", "pre_stop_interrupted"].includes(proof); + expect(stopped.resultJson?.executionCancellation).toMatchObject({ state: acknowledged ? "acknowledged" : "requested" }); + if (acknowledged) expect(stopped.resultJson?.executionCancellation).toMatchObject({ proof: "provider_api_stop_receipt" }); + expect(vi.mocked(publishLiveEvent).mock.calls).toEqual(acknowledged ? [[{ + companyId: company.id, + type: "heartbeat.run.status", + payload: expect.objectContaining({ runId, agentId: agent.id, status: "cancelled" }), + }]] : []); + if (acknowledged) { + const payload = vi.mocked(publishLiveEvent).mock.calls[0]![0].payload; + expect(payload).not.toHaveProperty("resultJson"); + expect(payload).not.toHaveProperty("executionCancellation"); + expect(payload).not.toHaveProperty("runnerProfileJson"); + } }); it("keeps pre-Stop input pending until a later user message authorizes one continuation", async () => { diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 835ac1fa078..7555e16fe92 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -12,6 +12,8 @@ import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED, readNativeLocalProc import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js"; import { pendingCaseHandoffForRun, cancelCaseHandoffsForIssue, isCaseHandoffSourceReleased } from "./case-handoff-lifecycle.js"; import { publishActivity, type ActivityPublication } from "../activity-log.js"; +import { buildHeartbeatRunStatusLiveEventPayload } from "../heartbeat-run-status-payload.js"; +import { publishLiveEvent } from "../live-events.js"; import { DEFERRED_WAKE_CONTEXT_KEY } from "./queue.js"; import { CANCELLABLE_HEARTBEAT_RUN_STATUSES } from "./recovery.js"; import { @@ -37,6 +39,7 @@ import { settleSlackConversation } from "../slack-conversation-lifecycle.js"; import { readQueuedInteractionResponse } from "../queued-interaction-response.js"; import { isAcknowledgedNativeStop, + acknowledgedNativeApiStopExecutionHasStopped, acknowledgedNativeStopExecutionHasStopped, } from "../acknowledged-native-stop.js"; import { completeTerminatedRemoteNativeSessionCleanup } from "../../vendor/paperclip-runner/index.js"; @@ -506,27 +509,61 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont completeTerminatedRemoteNativeSessionCleanup({ companyId, runId, remoteCleanupScope }); } } - const stopped = state?.status === "cancelled" ? await getRun(runId) : null; - // Native interrupt acknowledgement is not physical containment. Publish the - // shared Stop receipt only after every exact remote lease confirms teardown. - // Keep workspace preservation separate: a stopped provider can still owe files. - if (stopped?.status === "cancelled" && - parseObject(stopped.resultJson?.executionCancellation).state === "requested" && - (stopped.runtimeMode === "native" ? isAcknowledgedNativeStop(stopped) - : stopped.runtimeMode === "legacy" && await runUsedConversationAdapter(db, stopped)) && - await remoteExecutionHasStopped(db, companyId, runId)) { - await db.update(heartbeatRuns).set({ - resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ - executionCancellation: { ...parseObject(stopped.resultJson?.executionCancellation), - state: "acknowledged", acknowledgedAt: new Date().toISOString(), - proof: "provider_termination_receipt" }, - ...(stopped.runtimeMode === "legacy" ? { conversationContinuation: CONVERSATION_CONTINUATION_POLICY } : {}), - })}::jsonb`, - updatedAt: new Date(), - }).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId), - eq(heartbeatRuns.status, "cancelled"), - sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'requested'`, - stopped.runtimeMode === "native" ? sql`${heartbeatRuns.resultJson}->'nativeCancellation' = ${JSON.stringify(stopped.resultJson?.nativeCancellation)}::jsonb` : undefined)); + // Finalization can append a harmless event after proof was read. Retry + // only a lost API compare-and-set, with fresh complete proof each time. + for (let attempt = 0; attempt < 3; attempt++) { + const stopped = state?.status === "cancelled" ? await getRun(runId) : null; + // Native interrupt acknowledgement is not physical containment. Require + // exact remote lease teardown, or the API provider's terminal receipt and + // the local controller's retirement, before publishing shared Stop proof. + // Keep workspace preservation separate: a stopped provider can still owe files. + if (stopped?.status === "cancelled" && stopped.companyId === companyId && + parseObject(stopped.resultJson?.executionCancellation).state === "requested" && + (stopped.runtimeMode === "native" ? isAcknowledgedNativeStop(stopped) + : stopped.runtimeMode === "legacy" && await runUsedConversationAdapter(db, stopped))) { + const apiStopped = stopped.runtimeMode === "native" && await acknowledgedNativeApiStopExecutionHasStopped(db, stopped); + if (!apiStopped && !await remoteExecutionHasStopped(db, companyId, runId)) return; + const acknowledged = await db.update(heartbeatRuns).set({ + resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ + executionCancellation: { ...parseObject(stopped.resultJson?.executionCancellation), + state: "acknowledged", acknowledgedAt: new Date().toISOString(), + proof: apiStopped ? "provider_api_stop_receipt" : "provider_termination_receipt" }, + ...(stopped.runtimeMode === "legacy" ? { conversationContinuation: CONVERSATION_CONTINUATION_POLICY } : {}), + })}::jsonb`, + updatedAt: new Date(), + }).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId), + eq(heartbeatRuns.status, "cancelled"), + sql`${heartbeatRuns.resultJson}->'executionCancellation'->>'state' = 'requested'`, + apiStopped ? and(eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.driverKind, "cursor_cloud_api"), + eq(heartbeatRuns.nextEventSeq, stopped.nextEventSeq), + eq(heartbeatRuns.agentId, stopped.agentId), eq(heartbeatRuns.nativeIssueId, stopped.nativeIssueId!), + isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), + eq(heartbeatRuns.nativeSessionId, stopped.nativeSessionId!), eq(heartbeatRuns.runnerInstanceId, stopped.runnerInstanceId!), + sql`${heartbeatRuns.runnerProfileJson}->'executionPlacement' = ${JSON.stringify({ kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" })}::jsonb`, + sql`${heartbeatRuns.runnerProfileJson}->>'backend' = 'cursor_cloud_api'`, + sql`jsonb_build_object( + 'companyId', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'binding'->'companyId', + 'runId', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'binding'->'runId', + 'agentId', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'binding'->'agentId', + 'issueId', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'binding'->'issueId' + ) = ${JSON.stringify({ companyId, runId, agentId: stopped.agentId, issueId: stopped.nativeIssueId })}::jsonb`, + sql`jsonb_build_object( + 'normalizedSessionId', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'session'->'normalizedSessionId', + 'driverKind', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'session'->'driverKind' + ) = ${JSON.stringify({ normalizedSessionId: stopped.nativeSessionId, driverKind: "cursor_cloud_api" })}::jsonb`, + sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'->'provider'->>'kind' = 'cursor_cloud'`, + sql`not exists (select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} + and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id})`) : undefined, + stopped.runtimeMode === "native" ? sql`${heartbeatRuns.resultJson}->'nativeCancellation' = ${JSON.stringify(stopped.resultJson?.nativeCancellation)}::jsonb` : undefined)).returning({ id: heartbeatRuns.id }); + if (acknowledged.length > 0) { + publishLiveEvent({ + companyId: stopped.companyId, + type: "heartbeat.run.status", + payload: buildHeartbeatRunStatusLiveEventPayload(stopped), + }); + } + if (!apiStopped || acknowledged.length > 0) return; + } else return; } } diff --git a/server/src/services/heartbeat/runtime-execution.ts b/server/src/services/heartbeat/runtime-execution.ts index bcbe9cd71ba..6a0898f46f0 100644 --- a/server/src/services/heartbeat/runtime-execution.ts +++ b/server/src/services/heartbeat/runtime-execution.ts @@ -135,7 +135,7 @@ export interface HeartbeatRuntimeExecutionInput { executionTarget: Selection["workspace"]["executionTarget"]; workspaceOperationRecorder: ReturnType["createRecorder"]>; nativeWorkspaceSync: Selected["nativeWorkspaceSync"]; - workspaceRestoreSource: Parameters[3]; + workspaceRestoreSource: Parameters[3] | null; remoteExecution: unknown; }; config: Pick & { @@ -937,6 +937,7 @@ export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExe input.effects.onWorkspaceRestoreFailure(requiredWorkspaceRestoreEvidence); // Retention is the fallback even if recording this receipt fails. input.effects.onProviderResourceDisposition("stop_and_retain"); + if (workspaceRestoreSource === null) throw new Error("legacy_workspace_restore_source_unavailable"); await recordLegacyWorkspaceRestoreFailure(db, run, requiredWorkspaceRestoreEvidence, workspaceRestoreSource); } for (const stream of ["stdout", "stderr"] as const) { diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index d3d3a8ba4c3..032a80e73c6 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -2,10 +2,11 @@ import { randomUUID } from "node:crypto"; import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { activityLog, agents, companies, completionContracts, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js"; import { buildNativeRuntimeContext } from "../native-runtime/runtime-context.js"; import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js"; @@ -28,7 +29,8 @@ vi.mock("../native-runtime/native-workspace-sync.js", async (original) => ({ prepareNativeWorkspaceSync: vi.fn(async () => null), })); -const support = await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUNTIME_SELECTION_TEST_DATABASE_URL"); +const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () => { let database: Awaited>; let db: Db; @@ -36,7 +38,9 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = beforeAll(async () => { instructionRoot = await mkdtemp(join(tmpdir(), "paperclip-selection-instructions-")); await writeFile(join(instructionRoot, "AGENTS.md"), "Complete the assigned task.\n"); - database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-selection"); + if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl); + database = externalDatabaseUrl ? { connectionString: externalDatabaseUrl, cleanup: async () => {} } + : await startEmbeddedPostgresTestDatabase("paperclip-runtime-selection"); db = createDb(database.connectionString); await db.execute(sql`set client_min_messages = warning`); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); @@ -78,6 +82,65 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id)); const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false }; + it.each([false, true])("persists the Cursor provider API placement and completion authority without a Computer lease (ephemeral: %s)", async ephemeral => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral)); + const { input, run, issue } = await fixture(); + input.agent = { ...input.agent, adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification" } }; + await db.update(agents).set({ adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id)); + input.workspace.activeEnvironmentLease = null; + input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig, + agent: input.agent, issue, target: null, workspaceId: null }); + const result = await selectHeartbeatRuntime(db, input); + expect(result.selected).toBe(true); + if (!result.selected) throw new Error("Expected native Cursor selection"); + expect(result.nativeExecution?.provider).toMatchObject({ kind: "cursor_cloud", model: "caller-model" }); + expect(await read(run.id)).toMatchObject({ runtimeMode: "native", driverKind: "cursor_cloud_api", runnerProfileJson: { + backend: "cursor_cloud_api", executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" }, + nativeExecutionInput: result.nativeExecution, + } }); + expect(await db.select().from(completionContracts)).toHaveLength(1); + expect(await coordinators(run.id)).toHaveLength(1); + expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled(); + expect(input.stopControllerLease).toHaveBeenCalledOnce(); + }); + + it.each(["cursor_cloud_api", null] as const)("restores a historical Cursor workspace binding without allocating a Computer (driver: %s)", async driverKind => { + const { input, run, issue } = await fixture(); + input.agent = { ...input.agent, adapterConfig: { provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification" } }; + await db.update(agents).set({ adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id)); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId: input.agent.companyId, sourceIssueId: issue.id, + name: "Previously admitted Cursor workspace", mode: "shared_workspace", strategyType: "project_primary", cwd: "/previous-cursor-workspace" }).returning(); + input.workspace.persistedExecutionWorkspace = workspace; + input.workspace.activeEnvironmentLease = null; + input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig, + agent: input.agent, issue, target: null, workspaceId: workspace.id }); + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native Cursor selection"); + expect(selected.nativeExecution.binding.executionWorkspaceId).toBe(workspace.id); + const saved = await read(run.id); + const historicalProfile: Record = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "Cursor provider checkpoint" } }; + delete historicalProfile.executionPlacement; + await db.update(heartbeatRuns).set({ runnerProfileJson: historicalProfile, driverKind }).where(eq(heartbeatRuns.id, run.id)); + input.run = await read(run.id); + input.nativeRuntimeResolution = resolveHeartbeatNativeRuntimeMode({ persisted: input.run, enabled: true, runtimeConfig: {}, adapterConfig: input.agent.adapterConfig, + agent: input.agent, issue, target: null, workspaceId: null }); + input.session.persistedRunnerProfile = historicalProfile; + input.session.persistedNativeExecutionInput = selected.nativeExecution; + input.workspace.persistedExecutionWorkspace = null; + input.workspace.executionWorkspace.cwd = "/private-cursor-controller"; + vi.mocked(buildNativeRuntimeContext).mockClear(); + const resumed = await selectHeartbeatRuntime(db, input); + expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId }); + expect(await read(run.id)).toMatchObject({ driverKind: "cursor_cloud_api" }); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ ...historicalProfile, + executionPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } }); + expect(await db.select().from(executionWorkspaces)).toEqual([workspace]); + expect(await db.select().from(completionContracts)).toHaveLength(1); + expect(await coordinators(run.id)).toHaveLength(1); + expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled(); + expect(buildNativeRuntimeContext).not.toHaveBeenCalled(); + }); + it("persists and admits a standing-chat response contract without requiring the unavailable action", async () => { const { input, issue, agent, run } = await fixture(); const [conversation] = await db.update(issues).set({ title: "Chat with CEO", description: null, diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 8ea310b1eb7..b73ca32b3aa 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1,3 +1,4 @@ +import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js"; import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js"; import { museRunnerBroker } from "../muse-runner-broker.js"; import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js"; @@ -123,7 +124,7 @@ export interface HeartbeatRuntimeSelectionInput { persistedExecutionWorkspace: { id: string; metadata?: Record | null } | null; executionWorkspace: Omit[0]["workspace"], "id">; executionTarget: Parameters[0]["target"]; - activeEnvironmentLease: Pick[0], "lease">; + activeEnvironmentLease: Pick[0], "lease"> | null; isDotRun: boolean; projectRepositoryPaths: string[]; onCheckpoint?: Parameters[0]["onCheckpoint"]; @@ -279,7 +280,11 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele throw new Error("native_runtime_ineligible: issue is required"); } const nativeExecutionWorkspaceId = - persistedExecutionWorkspace?.id ?? run.id; + persistedNativeExecutionInput?.provider.kind === "cursor_cloud" + // Older Cursor runs keep their admitted binding identity. Their task + // files and Computer are not part of the recovered API controller. + ? persistedNativeExecutionInput.binding.executionWorkspaceId + : persistedExecutionWorkspace?.id ?? run.id; const nativeReviewContext = readNativeReviewAssignmentContext(context); const nativeReview = nativeReviewContext ? await measureRunPhase("runtime_selection.review", () => getNativeReviewAssignment(db, { companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, @@ -514,10 +519,10 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele ((executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" && !!executionTarget.runnerTask) || supportsManagedNativeWarmSession({ ...nativeRuntimeResolution.profile, acpxAgent: parseObject(agent.adapterConfig).acpxAgent }, managedAiRuntime?.attribution)); const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ?? - (["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession + (["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession ? { mode: "per_turn" as const, idleTimeoutMs: null } : environmentLifecyclePolicy ?? agentLifecyclePolicy); - if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend)) { + if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend)) { assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode); } if ( @@ -774,7 +779,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }), ) ?? `# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`, - !["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0 + !["openai_dot_mcp", "grok_bot_mcp", "muse_external", "cursor_cloud_api"].includes(nativeRuntimeResolution.profile.backend) && projectRepositoryPaths.length > 0 ? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}` : null, ].filter(Boolean).join("\n\n"), @@ -970,6 +975,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele runnerProfileJson: { ...nativeRuntimeResolution.profile, ...lockedProfile, + ...(nativeExecution?.provider.kind === "cursor_cloud" ? { executionPlacement: resolveProviderApiPlacement({ agent, run }) } : {}), nativePreparationFencingVersion: !lockedRun.runnerInstanceId && !lockedRun.nativeSessionId ? NATIVE_PREPARATION_FENCING_VERSION : lockedProfile.nativePreparationFencingVersion, @@ -1058,7 +1064,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }); if (!nativeSelected) return { selected: false }; stopControllerLease(); - nativeWorkspaceSync = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata) ? null + if (nativeExecution?.provider.kind !== "cursor_cloud" && !activeEnvironmentLease) throw new Error("native_environment_lease_unavailable"); + nativeWorkspaceSync = nativeExecution?.provider.kind === "cursor_cloud" || readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata) ? null : await measureRunPhase("runtime_selection.workspace_sync", () => prepareNativeWorkspaceSync({ db, runId: run.id, @@ -1066,7 +1073,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele workspaceId: nativeExecutionWorkspaceId, workspaceLocalDir: executionWorkspace.cwd, target: executionTarget, - lease: activeEnvironmentLease.lease, + lease: activeEnvironmentLease!.lease, restartRecovery: runOptions.nativeRestartRecovery, sameRunRecovery: Boolean(runOptions.nativeLeaseOwner), resourceDisposition: providerResourceDispositionForRun, diff --git a/server/src/services/heartbeat/workspace-preparation.test.ts b/server/src/services/heartbeat/workspace-preparation.test.ts index 8a1d010f8a7..3a218e50b8e 100644 --- a/server/src/services/heartbeat/workspace-preparation.test.ts +++ b/server/src/services/heartbeat/workspace-preparation.test.ts @@ -126,6 +126,56 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace }; return { input, source, issue, run, envOrchestrator }; } + it.each([false, true])("places Cursor Cloud in its provider API without a Computer under managed-only policy (ephemeral: %s)", async ephemeral => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral)); + const { input, source, issue, run } = await fixture(); + input.agent = { ...input.agent, adapterType: "paperclip_runner", adapterConfig: { + provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", + } }; + await db.update(agents).set({ adapterType: input.agent.adapterType, adapterConfig: input.agent.adapterConfig }).where(eq(agents.id, input.agent.id)); + const settingsBefore = await instanceSettingsService(db).get(); + try { + await instanceSettingsService(db).updateExperimental({ enableManagedSandboxOnly: true, enableNativeRunner: true }); + const selectionInput = { + run: input.run, agent: input.agent, + task: { issueRef: issue, issueContext: issue, issueId: issue.id, context: input.task.context, + executionProjectId: issue.projectId, workspaceProjectContext: { hasWorkspace: true }, boundSourceWorkspace: null, + hasSelectedWorkspaceSource: true, taskWorkspaceIntent: null, responsibleUserId: null }, + policy: { ...input.policy }, config: { config: input.agent.adapterConfig, sourceWorkspaceConfig: input.agent.adapterConfig, issueAssigneeOverrides: null }, + services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db), + instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null, findExecutionWorkspaceHolder: async () => null }, + }; + const selection = await selectHeartbeatEnvironment(db, selectionInput); + expect(selection).toMatchObject({ providerApiPlacement: { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" }, + selectedEnvironmentId: null, selectedEnvironmentForConfig: null, executionPolicy: { managedSandboxOnly: true } }); + input.environment = { ...input.environment, selectedEnvironmentId: selection.selectedEnvironmentId, + selectedEnvironmentForConfig: selection.selectedEnvironmentForConfig, environmentResolution: selection.environmentResolution, + resolvedInstanceSettings: selection.resolvedInstanceSettings }; + input.policy.executionPolicy = selection.executionPolicy; + const acquire = vi.spyOn(input.services.envOrchestrator, "acquireForRun"); + const realize = vi.spyOn(input.services.envOrchestrator, "realizeForRun"); + const resolveWorkspace = vi.spyOn(input.services, "resolveWorkspaceForRun"); + const result = await prepareHeartbeatWorkspace(db, input); + expect(result).toMatchObject({ selectedEnvironment: null, activeEnvironmentLease: null, executionTarget: null, + persistedExecutionWorkspace: null, projectRepositoryPaths: [], hostExecutionWorkspaceConfig: {}, skipHostProvision: true }); + expect(result.executionWorkspace.cwd).toContain(`/cursor-cloud-controllers/${input.agent.companyId}/${input.agent.id}/${issue.id}`); + expect(result.executionWorkspace).toMatchObject({ projectId: null, repoUrl: null, repoRef: null, created: false }); + expect(await readFile(join(source, "README.md"), "utf8")).toBe("source content\n"); + expect(acquire).not.toHaveBeenCalled(); expect(realize).not.toHaveBeenCalled(); expect(resolveWorkspace).not.toHaveBeenCalled(); + expect(await db.select().from(environmentLeases).where(eq(environmentLeases.heartbeatRunId, run.id))).toEqual([]); + expect(await db.select().from(executionWorkspaces)).toEqual([]); + expect((await instanceSettingsService(db).getExperimental()).enableManagedSandboxOnly).toBe(true); + await instanceSettingsService(db).updateGeneral({ executionMode: "kubernetes" }); + await expect(selectHeartbeatEnvironment(db, selectionInput)).rejects.toMatchObject({ status: 422, + details: { code: "provider_api_kubernetes_execution_unsupported" } }); + } finally { + await instanceSettingsService(db).updateGeneral({ executionMode: settingsBefore.general.executionMode ?? "any" }); + await instanceSettingsService(db).updateExperimental({ + enableManagedSandboxOnly: settingsBefore.experimental.enableManagedSandboxOnly, + enableNativeRunner: settingsBefore.experimental.enableNativeRunner, + }); + } + }); it.each(["unconfigured", "operator-disabled"] as const)("auto-enrollment keeps %s native sandboxes on their existing host path", async mode => { const { input, source, envOrchestrator } = await fixture(); input.agent.adapterType = "paperclip_runner"; @@ -1713,7 +1763,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace if (transport !== "daytona") { await expect(prepareNativeWorkspaceSync({ db, runId: input.run.id, companyId: issue.companyId, workspaceId: first.persistedExecutionWorkspace!.id, workspaceLocalDir: first.executionWorkspace.cwd, - target, lease: result.activeEnvironmentLease.lease })).resolves.toBeNull(); + target, lease: result.activeEnvironmentLease!.lease })).resolves.toBeNull(); } expect(admittedRoot).not.toBe(first.executionWorkspace.cwd); } else { @@ -1817,7 +1867,7 @@ describe.skipIf(!support.supported && !externalDatabaseUrl)("heartbeat workspace const result = await prepareHeartbeatWorkspace(db, input); expect((await readIssue(issue.id)).executionWorkspaceId).toBe(result.persistedExecutionWorkspace!.id); expect(result.persistedExecutionWorkspace!.name).toBe("Realized workspace"); - result.activeEnvironmentLease.lease.metadata!.remoteCwd = "/rebound"; - expect(result.workspaceRestoreSource.metadata?.remoteCwd).toBe("/original"); + result.activeEnvironmentLease!.lease.metadata!.remoteCwd = "/rebound"; + expect(result.workspaceRestoreSource!.metadata?.remoteCwd).toBe("/original"); }); }); diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 838e8c75fb2..15958e0c6b9 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -1,3 +1,4 @@ +import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js"; import { assertEphemeralWorkspaceExecution, readRemoteWorkspaceAuthority, RemoteWorkspaceWriterActiveError, reserveRemoteWorkspace, shouldUseRemoteWorkspaces } from "../remote-workspaces.js"; import { WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS, WorkspaceBusyDeferral } from "./retries.js"; import { WORKSPACE_BUSY_RETRY_REASON } from "../../modules/run-dispatch/index.js"; @@ -127,10 +128,10 @@ export interface HeartbeatWorkspacePreparationInput { executionPolicy: Parameters[0]; }; environment: { - selectedEnvironmentId: string; + selectedEnvironmentId: string | null; localEnvironment: Environment; selectedEnvironmentForConfig: Environment | null; - environmentResolution: ExecutionWorkspaceEnvironmentResolution; + environmentResolution: ExecutionWorkspaceEnvironmentResolution | { environmentId: null; source: "default" }; resolvedInstanceSettings: Awaited["get"]>>; }; config: { @@ -169,13 +170,14 @@ function readNonEmptyString(value: unknown): string | null { export async function prepareHeartbeatWorkspace(db: Db, input: HeartbeatWorkspacePreparationInput) { const sourceIssueId = input.reuse.reusableExistingExecutionWorkspace?.sourceIssueId ?? input.reuse.existingExecutionWorkspace?.sourceIssueId ?? input.task.issueId; - if (!sourceIssueId || input.reuse.isDotRun) return prepareHeartbeatWorkspaceLocked(db, input); + if (!sourceIssueId || input.reuse.isDotRun || resolveProviderApiPlacement(input)) return prepareHeartbeatWorkspaceLocked(db, input); return withIsolatedRepositorySourceLock(db, input.agent.companyId, sourceIssueId, () => prepareHeartbeatWorkspaceLocked(db, input)); } async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspacePreparationInput) { const { run, agent, controllerLease, nativeRunnerPreparationSpans } = input; + const providerApiPlacement = resolveProviderApiPlacement(input); const fresh = input.task.freshWorkspace?.mode === "fresh_isolated_workspace"; const timeWorkspace = createHeartbeatPreparationTiming(run.id); const { @@ -217,7 +219,8 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace } = input.reuse; // An ordinary clean task directory already has a new per-attempt identity. // It is not a repository source and must not pass through Git copy admission. - const freshNeedsSeed = fresh && !isDotRun && !useIsolatedTaskDirectory && + const controllerOnly = isDotRun || providerApiPlacement !== null; + const freshNeedsSeed = fresh && !controllerOnly && !useIsolatedTaskDirectory && !(nativeChatWorkspaceScope && !nativeChatWorkspaceScope.projectId); const { envOrchestrator, @@ -237,14 +240,14 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace } } const enrolledRemoteAuthority = readRemoteWorkspaceAuthority((reusableExistingExecutionWorkspace ?? existingExecutionWorkspace)?.metadata) !== null; - let remoteAuthority = !isDotRun && shouldUseRemoteWorkspaces({ + let remoteAuthority = !controllerOnly && shouldUseRemoteWorkspaces({ adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver ?? null, experimentalEnabled: (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" || resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled) && Boolean(issueId && issueRef) && !projectExecutionWorkspacePolicy?.storageProviderPluginId, existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace, }); - assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, + if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority }); const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({ @@ -260,7 +263,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace "Remote workspace storage requires a task.", { reason: "remote_workspace_task_required" }); const remoteSourceState: { value: Extract>, { authority: "remote" }> | null } = { value: null }; const pluginConfig = parseObject(selectedEnvironmentForConfig?.config); - if (fresh && !isDotRun && pluginTaskWorkspace) { + if (fresh && !controllerOnly && pluginTaskWorkspace) { const plugin = typeof pluginConfig.pluginKey === "string" ? await pluginRegistryService(db).getByKey(pluginConfig.pluginKey) : null; const driver = plugin?.manifestJson.environmentDrivers?.find(value => value.driverKey === pluginConfig.driverKey); if (driver?.freshWorkspaceVersion !== 1) throw new WorkspaceValidationFailure( @@ -298,10 +301,11 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace } : null, resolveSelectedEnvironmentDriver: async () => { + if (providerApiPlacement) return "provider_api"; const preflightEnvironment = await envOrchestrator.resolveEnvironment( { companyId: agent.companyId, - selectedEnvironmentId, + selectedEnvironmentId: selectedEnvironmentId ?? localEnvironment.id, localEnvironmentId: localEnvironment.id, }, ); @@ -329,7 +333,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace remoteAuthority = false; // Source resolution can retain a configured host folder. Check the // resolved authority before materializing it or acquiring a sandbox. - assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, + if (!providerApiPlacement) assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, adapterType: agent.adapterType, pluginTaskWorkspace, remoteAuthority }); } else { @@ -354,10 +358,10 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace }; } } - if (isDotRun) { + if (controllerOnly) { // This is private controller storage, never a provider filesystem. // External providers receive workspace.access=none and cwd=null. - const cwd = path.resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", (persistedNativeExecutionInput?.provider.kind ?? mergedConfig.provider) === "muse" ? "muse-controllers" : "dot-controllers", agent.companyId, run.id); + const cwd = path.resolve(resolvePaperclipInstanceRoot(), "runtime", "paperclip-runner", providerApiPlacement ? "cursor-cloud-controllers" : (persistedNativeExecutionInput?.provider.kind ?? mergedConfig.provider) === "muse" ? "muse-controllers" : "dot-controllers", agent.companyId, ...(providerApiPlacement ? [agent.id, issueId ?? run.id] : [run.id])); await fs.mkdir(cwd, { recursive: true, mode: 0o700 }); return { cwd, source: "agent_home" as const, projectId: null, workspaceId: null, repoUrl: null, repoRef: null, workspaceHints: [], warnings: [], baseCwdFallback: false, materializationFailures: [], additionalWorkspaces: [], referencedProjectFailures: [] }; @@ -497,7 +501,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace let providerFresh: PluginFreshWorkspace | undefined; let previousProviderWorkspaceId: string | null = null; let previousProviderBindingRevision = 0; - if (input.task.freshWorkspace?.mode === "fresh_isolated_workspace" && !isDotRun && issueId) { + if (input.task.freshWorkspace?.mode === "fresh_isolated_workspace" && !controllerOnly && issueId) { const [task] = await db.select({ workspaceId: issues.executionWorkspaceId, revision: issues.workspaceBindingRevision }).from(issues) .where(and(eq(issues.id, issueId), eq(issues.companyId, agent.companyId))); previousProviderWorkspaceId = task?.workspaceId ?? null; @@ -513,7 +517,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace previousWorkspaceId: previousProviderWorkspaceId }); } } - const pluginPrepared = isDotRun || remoteAuthority ? null : await preparePluginWorkspace(db, input.services.workspaceWorkers, { + const pluginPrepared = controllerOnly || remoteAuthority ? null : await preparePluginWorkspace(db, input.services.workspaceWorkers, { companyId: agent.companyId, projectId: resolvedWorkspace.projectId, projectWorkspaceId: resolvedWorkspace.workspaceId, issueId, runId: run.id, agentId: agent.id, responsibleUserId, existingWorkspaceId: providerFresh ? providerFresh.previousWorkspaceId : reusableExistingExecutionWorkspace?.id ?? issueRef?.executionWorkspaceId ?? null, @@ -544,7 +548,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace boundWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace, })); const skipHostProvision = copyOnlyProvisioning || hostProvisionDriver === "sandbox" || hostProvisionDriver === "ssh" || hostProvisionDriver === "plugin"; - const hostExecutionWorkspaceConfig = isDotRun ? {} : + const hostExecutionWorkspaceConfig = controllerOnly ? {} : stripHostWorkspaceProvision({ config: fresh && !remoteAuthority ? { ...mergedConfig, workspaceStrategy: { type: "project_primary" } } : mergedConfig, skipHostProvision, @@ -726,7 +730,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace warnings: [], created: false, branchCreatedByRuntime: false, baseRefSha: pluginPrepared.metadata?.workspaceProviderBaseSha as string | null, } as RealizedExecutionWorkspace, reusedExecutionWorkspace: Boolean(reusableExistingExecutionWorkspace), policy: workspaceReuseProvisioningPolicy } - : isDotRun || pluginTaskWorkspace || remoteAuthority ? { executionWorkspace: { ...executionWorkspaceBase, strategy: pluginTaskWorkspace || remoteAuthority ? latestWorkspaceStrategyType : "project_primary" as const, cwd: freshTaskReservation?.cwd ?? resolvedWorkspace.cwd, branchName: remoteSourceState.value?.source.kind === "git" ? remoteSourceState.value!.source.branchName ?? null : null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false } as RealizedExecutionWorkspace, reusedExecutionWorkspace: (pluginTaskWorkspace || remoteAuthority) && workspaceReuseProvisioningPolicy.shouldRestoreExistingWorkspace && Boolean(reusableExistingExecutionWorkspace), policy: workspaceReuseProvisioningPolicy } : await timeWorkspace("workspace_provision", () => provisionExecutionWorkspaceForFreshnessDecision( + : controllerOnly || pluginTaskWorkspace || remoteAuthority ? { executionWorkspace: { ...executionWorkspaceBase, strategy: pluginTaskWorkspace || remoteAuthority ? latestWorkspaceStrategyType : "project_primary" as const, cwd: freshTaskReservation?.cwd ?? resolvedWorkspace.cwd, branchName: remoteSourceState.value?.source.kind === "git" ? remoteSourceState.value!.source.branchName ?? null : null, worktreePath: null, warnings: [], created: false, branchCreatedByRuntime: false } as RealizedExecutionWorkspace, reusedExecutionWorkspace: (pluginTaskWorkspace || remoteAuthority) && workspaceReuseProvisioningPolicy.shouldRestoreExistingWorkspace && Boolean(reusableExistingExecutionWorkspace), policy: workspaceReuseProvisioningPolicy } : await timeWorkspace("workspace_provision", () => provisionExecutionWorkspaceForFreshnessDecision( { requestedShouldReuseExisting, existingExecutionWorkspaceId: @@ -854,7 +858,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace const resolvedProjectWorkspaceId = resolvedWorkspaceReusePolicy.shouldRestoreExistingWorkspace && reusableExistingExecutionWorkspace?.strategyType === "git_worktree" ? reusableExistingExecutionWorkspace.projectWorkspaceId - : isDotRun ? null : issueRef?.projectWorkspaceId ?? resolvedWorkspace.workspaceId ?? null; + : controllerOnly ? null : issueRef?.projectWorkspaceId ?? resolvedWorkspace.workspaceId ?? null; let persistedExecutionWorkspace: ExecutionWorkspace | null = null; let issueExecutionWorkspaceIdForRun = issueRef?.executionWorkspaceId ?? null; @@ -875,7 +879,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace const bindIssueToPersistedExecutionWorkspace = async ( workspace: ExecutionWorkspace | null, ) => { - if (isDotRun || !issueId || !workspace || nativeRecoveryExecutionWorkspaceId) { + if (controllerOnly || !issueId || !workspace || nativeRecoveryExecutionWorkspaceId) { return; } const nextIssueWorkspaceMode = @@ -996,6 +1000,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace try { if (remoteAuthority) { if (!remoteSourceState.value || !issueId) throw new Error("remote_workspace_source_missing"); + if (!selectedEnvironmentId) throw new Error("remote_workspace_environment_missing"); await assertRemotePreparationOwned(); const reserved = await reserveRemoteWorkspace({ db, companyId: agent.companyId, issueId, runId: run.id, agentId: agent.id, @@ -1053,7 +1058,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace ), }, )) - : !isDotRun && !persistedNativeExecutionInput && (resolvedProjectId || issueRef) + : !controllerOnly && !persistedNativeExecutionInput && (resolvedProjectId || issueRef) ? await timeWorkspace("workspace_record_create", () => executionWorkspacesSvc.create({ companyId: agent.companyId, projectId: resolvedProjectId, @@ -1178,7 +1183,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace const projectRepositoryPaths: string[] = []; if (remoteAuthority) projectRepositoryPaths.push(...(remoteSourceState.value?.source.repositories ?? []) .filter(repository => !repository.readOnly).map(repository => repository.relativePath)); - if (!pluginPrepared && !pluginTaskWorkspace && !remoteAuthority && !isDotRun && persistedExecutionWorkspace && issueId && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) { + if (!pluginPrepared && !pluginTaskWorkspace && !remoteAuthority && !controllerOnly && persistedExecutionWorkspace && issueId && !persistedNativeExecutionInput && !nativeRecoveryExecutionWorkspaceId) { const taskRepositories = await timeWorkspace("workspace_repository_admission", () => executionWorkspacesSvc.prepareTaskRepositoriesForAdmission({ companyId: agent.companyId, issueId, workspaceId: persistedExecutionWorkspace!.id, cwd: executionWorkspace.cwd, @@ -1231,6 +1236,21 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace }) .where(eq(heartbeatRuns.id, run.id)); } + if (providerApiPlacement) { + await controllerLease.assertOwned(); + return { + resolvedWorkspace, hostExecutionWorkspaceConfig, skipHostProvision: true, + latestWorkspaceConfigMetadata, workspaceConfigFreshness, workspaceOperationRecorder, + executionWorkspace, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, + persistedExecutionWorkspace: null, projectRepositoryPaths: [], + environmentAcquireStartedAtMs: Date.now(), selectedEnvironment: null, + activeEnvironmentLease: null, + realizationResult: { remoteExecution: null, executionTarget: null, persistedExecutionWorkspace: null, + lease: null, workspaceRealization: null }, + environmentRealizeEndedAtMs: Date.now(), workspaceRealization: null, + executionTarget: null, workspaceRestoreSource: null, + }; + } const environmentAcquireStartedAtMs = Date.now(); let acquiredEnvironment: Awaited< ReturnType @@ -1254,7 +1274,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace } acquiredEnvironment = await timeWorkspace("workspace_environment_acquire", () => envOrchestrator.acquireForRun({ companyId: agent.companyId, - selectedEnvironmentId, + selectedEnvironmentId: selectedEnvironmentId ?? localEnvironment.id, localEnvironmentId: localEnvironment.id, adapterType: agent.adapterType, adapterConfig: parseObject(agent.adapterConfig), @@ -1423,7 +1443,7 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace if (wake) wake.executionContinuation = continuation; } } - if (!pluginPrepared && !pluginTaskWorkspace && !remoteAuthority && !isDotRun && issueId && (persistedNativeExecutionInput || nativeRecoveryExecutionWorkspaceId) + if (!pluginPrepared && !pluginTaskWorkspace && !remoteAuthority && !controllerOnly && issueId && (persistedNativeExecutionInput || nativeRecoveryExecutionWorkspaceId) && !(executionTarget?.kind === "remote" && (executionTarget.transport === "ssh" || (executionTarget.transport === "sandbox" && executionTarget.runnerTask)))) { const workspaceId = persistedNativeExecutionInput?.binding.executionWorkspaceId ?? nativeRecoveryExecutionWorkspaceId!; diff --git a/server/src/services/native-runtime/cursor-cloud-runtime.ts b/server/src/services/native-runtime/cursor-cloud-runtime.ts index ce4db554d3e..902f9354bc8 100644 --- a/server/src/services/native-runtime/cursor-cloud-runtime.ts +++ b/server/src/services/native-runtime/cursor-cloud-runtime.ts @@ -4,9 +4,24 @@ import { join } from "node:path"; import { and, eq } from "drizzle-orm"; import { heartbeatRuns, type Db } from "@paperclipai/db"; import { loadConfig } from "../../config.js"; +import { parseObject } from "../../adapters/utils.js"; import { archiveCursorCloudSessionCheckpoint, CURSOR_CLOUD_DEFAULT_TIMEOUT_MS, type NativeExecutionInput, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js"; import { createRemoteMcpRelayToken, registerRemoteMcpRelay, remoteMcpRelayUrl } from "./remote-mcp-relay.js"; +/** Normalize Cursor's terminal evidence before shared Stop authority consumes it. */ +export function isConfirmedCursorCloudTerminalObservation(event: { eventType: string; payload: unknown }): boolean { + const payload = parseObject(event.payload), errorCode = parseObject(payload.error).code; + // Containment can join an already confirmed provider end. A local failure or + // an explicitly unobserved provider end cannot prove remote termination. + const confirmedTerminal = (event.eventType === "turn.interrupted" && payload.status === "interrupted") + || (event.eventType === "turn.completed" && payload.status === "completed") + || (event.eventType === "turn.cancelled" && payload.status === "cancelled" + && payload.providerTerminalState === "completed" && payload.reason === "cancelled_after_provider_completed") + || (event.eventType === "turn.failed" && payload.status === "failed" && typeof errorCode === "string" + && ["cursor_cloud_error", "cursor_cloud_expired"].includes(errorCode)); + return confirmedTerminal && (payload.providerTerminalObserved === undefined || payload.providerTerminalObserved === true); +} + export function resolveCursorCloudPublicOrigin(): string | undefined { return loadConfig().authPublicBaseUrl; } diff --git a/server/src/services/native-runtime/provider-api-placement.test.ts b/server/src/services/native-runtime/provider-api-placement.test.ts new file mode 100644 index 00000000000..dc1389b8c3c --- /dev/null +++ b/server/src/services/native-runtime/provider-api-placement.test.ts @@ -0,0 +1,56 @@ +import { randomUUID } from "node:crypto"; +import { describe, expect, it } from "vitest"; +import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { resolveProviderApiPlacement } from "./provider-api-placement.js"; +import { evaluateExecutionAllowlist } from "../execution-allowlist.js"; + +const placement = { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" }; +const agent = { adapterType: "paperclip_runner", adapterConfig: { + provider: "cursor_cloud", model: "caller-model", repoUrl: "https://github.com/example/qualification", +} }; +function execution(provider: "cursor_cloud" | "codex") { + return buildNativeExecutionInput({ companyId: randomUUID(), agentId: randomUUID(), runId: randomUUID(), provider, + model: "caller-model", ...(provider === "cursor_cloud" ? { cursorCloud: { schema: "paperclip.cursor-cloud.config.v1" as const, repos: [{ url: "https://github.com/example/qualification" }] } } : {}), + issue: { id: randomUUID(), identifier: "API-1", title: "Bound task", description: null, workMode: "standard" }, + taskPrompt: "Complete the task", normalizedSessionId: randomUUID(), + workspace: { id: randomUUID(), cwd: "/private/controller", repoUrl: null, repoRef: null, branchName: null }, + completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Complete the task", criteria: [{ id: "objective", requirement: "Complete the task" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); +} + +describe("provider API controller placement", () => { + it("derives API placement only from a validated native Cursor contract while local agent code stays denied", () => { + expect(resolveProviderApiPlacement({ agent })).toEqual(placement); + expect(resolveProviderApiPlacement({ agent: { ...agent, adapterType: "cursor_cloud" } })).toBeNull(); + expect(resolveProviderApiPlacement({ agent: { ...agent, adapterConfig: { provider: "codex", executionPlacement: placement } } })).toBeNull(); + expect(evaluateExecutionAllowlist({ managedSandboxOnly: true }, { driver: "local", provider: null }).allowed).toBe(false); + }); + it.each(["command", "extraArgs", "cwd", "workspaceStrategy", "workspaceRuntime", "executionPlacement", "providerApiPlacement", "controllerPlacement", "apiOnly"])("refuses configurable host authority through %s", key => { + expect(() => resolveProviderApiPlacement({ agent: { ...agent, adapterConfig: { ...agent.adapterConfig, [key]: key === "extraArgs" ? ["--execute"] : "injected" } } })).toThrow(); + }); + it("requires exact model and refuses unrelated environment or a preconfigured home", () => { + for (const patch of [{ model: "" }, { env: { NODE_OPTIONS: "--require=untrusted" } }, { homeMode: "preconfigured" }]) { + expect(() => resolveProviderApiPlacement({ agent: { ...agent, adapterConfig: { ...agent.adapterConfig, ...patch } } })).toThrow(); + } + }); + it("restores the admitted provider rather than reinterpreting a later agent configuration", () => { + const nativeExecutionInput = execution("cursor_cloud"); + const run = { runtimeMode: "native", runnerProfileJson: { backend: "cursor_cloud_api", executionPlacement: placement, nativeExecutionInput } }; + expect(resolveProviderApiPlacement({ agent: { ...agent, adapterConfig: { provider: "codex" } }, run })).toEqual(placement); + expect(() => resolveProviderApiPlacement({ agent, run: { ...run, runnerProfileJson: { ...run.runnerProfileJson, backend: "codex_app_server" } } })).toThrow("binding_mismatch"); + expect(() => resolveProviderApiPlacement({ agent, run: { ...run, runnerProfileJson: { ...run.runnerProfileJson, nativeExecutionInput: execution("codex") } } })).toThrow("binding_mismatch"); + expect(resolveProviderApiPlacement({ agent, run: { runtimeMode: "native", runnerProfileJson: { nativeExecutionInput: execution("codex") } } })).toBeNull(); + }); + it("binds historical Cursor input to its backend and recorded driver before granting API placement", () => { + const nativeExecutionInput = execution("cursor_cloud"); + const run = { runtimeMode: "native", driverKind: "cursor_cloud_api", runnerProfileJson: { backend: "cursor_cloud_api", nativeExecutionInput } }; + expect(resolveProviderApiPlacement({ agent, run })).toEqual(placement); + expect(resolveProviderApiPlacement({ agent, run: { ...run, driverKind: null } })).toEqual(placement); + expect(resolveProviderApiPlacement({ agent, run: { ...run, driverKind: null, runnerProfileJson: { nativeExecutionInput } } })).toEqual(placement); + expect(() => resolveProviderApiPlacement({ agent, run: { ...run, runnerProfileJson: { ...run.runnerProfileJson, backend: "codex_app_server" } } })).toThrow("binding_mismatch"); + expect(() => resolveProviderApiPlacement({ agent, run: { ...run, driverKind: "codex_app_server" } })).toThrow("binding_mismatch"); + }); +}); diff --git a/server/src/services/native-runtime/provider-api-placement.ts b/server/src/services/native-runtime/provider-api-placement.ts new file mode 100644 index 00000000000..3029508a8e9 --- /dev/null +++ b/server/src/services/native-runtime/provider-api-placement.ts @@ -0,0 +1,60 @@ +import { parseObject } from "../../adapters/utils.js"; +import { unprocessable } from "../../errors.js"; +import { isExecutionForcedToKubernetes, type ExecutionPolicy } from "../execution-allowlist.js"; +import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { resolvePaperclipRunnerNativeProviderInput } from "./provider-profile.js"; + +export interface ProviderApiPlacement { + kind: "provider_api"; + provider: "cursor_cloud"; + controller: "paperclip"; +} + +export function managedCursorApiDefaultEnabled(experimental: { + enableManagedSandboxOnly?: boolean; enableNativeRunner?: boolean; +} | null | undefined): boolean { + return experimental?.enableManagedSandboxOnly === true && experimental.enableNativeRunner === true; +} + +export function assertProviderApiPlacementPolicy(policy: ExecutionPolicy): void { + if (isExecutionForcedToKubernetes(policy)) { + throw unprocessable("Cursor Cloud API execution is unavailable under the Kubernetes-only policy.", { + code: "provider_api_kubernetes_execution_unsupported", + }); + } +} + +/** Held setup may be incomplete. This only skips unused Computer preparation; + * active hire and verification must still derive a fully validated placement. */ +export function isProviderApiSetupDraft(agent: { adapterType?: string; adapterConfig?: unknown }): boolean { + return agent.adapterType === "paperclip_runner" && parseObject(agent.adapterConfig).provider === "cursor_cloud"; +} + +/** Placement is server-owned. A provider's API client is not an agent Computer. */ +export function resolveProviderApiPlacement(input: { + agent: { adapterType: string; adapterConfig: unknown }; + run?: { runtimeMode: string | null; runnerProfileJson: unknown; driverKind?: string | null }; +}): ProviderApiPlacement | null { + const profile = parseObject(input.run?.runnerProfileJson); + const persisted = input.run?.runtimeMode === "native" && profile.nativeExecutionInput !== undefined + ? parseNativeExecutionInput(profile.nativeExecutionInput) : null; + if (persisted?.provider.kind === "cursor_cloud" + && ((profile.backend != null && profile.backend !== "cursor_cloud_api") + || (input.run?.driverKind != null && input.run.driverKind !== "cursor_cloud_api"))) { + throw new Error("provider_api_placement_binding_mismatch"); + } + const saved = profile.executionPlacement; + const savedRecord = parseObject(saved); + if (saved !== undefined && (Object.keys(savedRecord).sort().join(",") !== "controller,kind,provider" + || savedRecord.kind !== "provider_api" || savedRecord.provider !== "cursor_cloud" || savedRecord.controller !== "paperclip" + || persisted?.provider.kind !== "cursor_cloud" || profile.backend !== "cursor_cloud_api")) { + throw new Error("provider_api_placement_binding_mismatch"); + } + if (persisted) return persisted.provider.kind === "cursor_cloud" + ? { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" } : null; + const config = parseObject(input.agent.adapterConfig); + if (input.agent.adapterType !== "paperclip_runner" || config.provider !== "cursor_cloud") return null; + // Validate the provider's complete launch contract before granting placement. + resolvePaperclipRunnerNativeProviderInput({ backend: "cursor_cloud_api", adapterConfig: config }); + return { kind: "provider_api", provider: "cursor_cloud", controller: "paperclip" }; +} diff --git a/server/src/services/native-runtime/provider-profile.ts b/server/src/services/native-runtime/provider-profile.ts index 301c9b590c5..d1d7cb0bc93 100644 --- a/server/src/services/native-runtime/provider-profile.ts +++ b/server/src/services/native-runtime/provider-profile.ts @@ -182,7 +182,7 @@ export function resolvePaperclipRunnerCursorCloudConfig(config: Record config[key] !== undefined && config[key] !== "" && !(Array.isArray(config[key]) && config[key].length === 0)); if (config.lifecycleMode !== undefined && config.lifecycleMode !== "per_turn") unsupported.push("lifecycleMode"); if (config.homeMode === "preconfigured") unsupported.push("homeMode"); diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index bf76e7c8c3f..f5c240c0df4 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -758,6 +758,9 @@ describe("runner E2E catalog", () => { 'summary:"PAPERCLIP_E2E_PLAN_DONE_nonce"', ); expect(task!.buildPrompt("nonce")).toContain("first call get_task_context"); + expect(task!.buildPrompt("nonce")).toContain("criteria:CURRENT_TURN_CRITERIA"); + expect(task!.buildPrompt("nonce")).toContain("entry for every current criterion ID"); + expect(task!.buildPrompt("nonce")).not.toContain('criterionId:"objective"'); expect(task!.buildPrompt("nonce")).toContain( "identifies the exact revised Plan revision used as the confirmation target as accepted", ); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 4100e707f9d..81e7e4fdf85 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -685,7 +685,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [ "Only after the revised plan is accepted, implement it by posting one final visible task-thread response containing exactly " + `PAPERCLIP_E2E_PLAN_DONE_${nonce}` + " and mark the task Done.", - `For a native runner, remain in the requested planning collaboration mode. Call write_document for key \`plan\`, then call request_human_input exactly once with interactionKind \`confirmation\`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy \`wake_assignee\`. For both the initial Plan and the revised Plan, those two tool calls form one indivisible response sequence: immediately after write_document succeeds, request_human_input must be your next action using that call's returned latestRevisionId. Do not emit assistant text, end the response or heartbeat, or stop after write_document alone before the matching confirmation request succeeds. Do not call paperclip_finish while waiting for either Plan confirmation. When an acceptance wake arrives, first call get_task_context. Treat the wake as valid only when that control-plane result is for the current task and identifies the exact revised Plan revision used as the confirmation target as accepted; otherwise do not finish and continue waiting for the matching revision-bound confirmation. After that verification succeeds, your immediate next action must be the paperclip_finish tool call. Do not call list_documents or any other tool, and do not emit any assistant text, acknowledgement, progress note, or preamble between verification and paperclip_finish. Use the current turn completion revision and criterion IDs (CURRENT_TURN_REVISION below is a placeholder, not a literal). Call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"PAPERCLIP_E2E_PLAN_DONE_${nonce}",completionClaim:{contractRevision:CURRENT_TURN_REVISION,objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}. Wait for that tool call to succeed, then emit only PAPERCLIP_E2E_PLAN_DONE_${nonce} as the complete final response. Do not write a user-facing final response before paperclip_finish succeeds, and do not call another tool.`, + `For a native runner, remain in the requested planning collaboration mode. Call write_document for key \`plan\`, then call request_human_input exactly once with interactionKind \`confirmation\`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy \`wake_assignee\`. For both the initial Plan and the revised Plan, those two tool calls form one indivisible response sequence: immediately after write_document succeeds, request_human_input must be your next action using that call's returned latestRevisionId. Do not emit assistant text, end the response or heartbeat, or stop after write_document alone before the matching confirmation request succeeds. Do not call paperclip_finish while waiting for either Plan confirmation. When an acceptance wake arrives, first call get_task_context. Treat the wake as valid only when that control-plane result is for the current task and identifies the exact revised Plan revision used as the confirmation target as accepted; otherwise do not finish and continue waiting for the matching revision-bound confirmation. After that verification succeeds, your immediate next action must be the paperclip_finish tool call. Do not call list_documents or any other tool, and do not emit any assistant text, acknowledgement, progress note, or preamble between verification and paperclip_finish. Use the current turn completion revision and criterion IDs. CURRENT_TURN_REVISION and CURRENT_TURN_CRITERIA below are placeholders, not literals. CURRENT_TURN_CRITERIA must contain one {criterionId:,status:"satisfied",evidenceRefs:[]} entry for every current criterion ID. Call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"PAPERCLIP_E2E_PLAN_DONE_${nonce}",completionClaim:{contractRevision:CURRENT_TURN_REVISION,objectiveSatisfied:true,criteria:CURRENT_TURN_CRITERIA,remainingWork:[]},evidence:[],verification:[]}. Wait for that tool call to succeed, then emit only PAPERCLIP_E2E_PLAN_DONE_${nonce} as the complete final response. Do not write a user-facing final response before paperclip_finish succeeds, and do not call another tool.`, `For a legacy runner, use the public Paperclip API. The first PUT of the \`plan\` issue document creates it. For every later PUT, first GET the current document and set \`baseRevisionId\` to its \`latestRevisionId\`; a 409 means you must GET again and retry with the new latest revision. Create a \`request_confirmation\` targeting the successful PUT response's \`latestRevisionId\` with \`continuationPolicy: wake_assignee\`, and move the issue to \`in_review\` while waiting. After the revised Plan is accepted, write PAPERCLIP_E2E_PLAN_DONE_${nonce} exactly once through one atomic issue PATCH with status \`done\` and that exact comment; do not POST a separate comment or perform a second write.`, "Do not create files, child tasks, or unrelated work, and do not expose credentials.", ].join("\n"), diff --git a/tests/runner-e2e/cursor-cloud-replay-fault.test.ts b/tests/runner-e2e/cursor-cloud-replay-fault.test.ts index a2e34c1771a..4b29286b59f 100644 --- a/tests/runner-e2e/cursor-cloud-replay-fault.test.ts +++ b/tests/runner-e2e/cursor-cloud-replay-fault.test.ts @@ -212,7 +212,7 @@ describe("Cursor Cloud replay bundle treatment", () => { it("the actual driver reconnects the real client with its saved cursor and deduplicates the replayed frame without creating another prompt", async () => { const directory = await root(), events: PrpEvent[] = [], proofs: CursorCloudReplayFaultProof[] = []; const nativeRun = "run-33333333-3333-4333-8333-333333333333", time = "2026-10-28T00:00:00Z"; - let nativeAgent = "", creates = 0, streams = 0; + let nativeAgent = "", creates = 0, streams = 0, cancelled = false; const requests: Array<{ path: string; headers: Headers }> = []; const peer: typeof fetch = async (resource, init) => { const path = new URL(String(resource)).pathname, headers = new Headers(init?.headers); requests.push({ path, headers }); @@ -226,7 +226,8 @@ it("the actual driver reconnects the real client with its saved cursor and dedup const withheld = 'event: assistant\nid: withheld\ndata: {"text":"must be replayed"}\n\n'; return bytes([Buffer.from(first + withheld + (streams === 1 ? "" : `event: assistant\nid: new-cursor\ndata: {"text":"Recovered."}\n\nevent: result\nid: terminal\ndata: {"runId":"${nativeRun}","status":"FINISHED"}\n\n`))]); } - if (path.endsWith(nativeRun)) return Response.json({ id: nativeRun, agentId: nativeAgent, status: streams < 2 ? "RUNNING" : "FINISHED", createdAt: time, updatedAt: time }); + if (path.endsWith("/cancel") && init?.method === "POST") { cancelled = true; return Response.json({ id: nativeRun }); } + if (path.endsWith(nativeRun)) return Response.json({ id: nativeRun, agentId: nativeAgent, status: cancelled ? "CANCELLED" : streams < 2 ? "RUNNING" : "FINISHED", createdAt: time, updatedAt: time }); throw new Error("Unexpected controlled request"); }; const client = new CursorCloudClient({ apiKey: "private-auth-canary", fetch: createCursorCloudReplayFaultFetch({ fetch: peer, digest, record: value => proofs.push(value) }) }); diff --git a/tests/runner-e2e/cursor-cloud-replay-fault.ts b/tests/runner-e2e/cursor-cloud-replay-fault.ts index 29257ab3bab..ada7b5b4bc4 100644 --- a/tests/runner-e2e/cursor-cloud-replay-fault.ts +++ b/tests/runner-e2e/cursor-cloud-replay-fault.ts @@ -101,7 +101,7 @@ export function createCursorCloudReplayFaultFetch(input: { if (cutNextRead) { proof.cutCount += 1; save(); - controller.error(new Error("Injected Cursor Cloud SSE disconnect")); + controller.error(new TypeError("Injected Cursor Cloud SSE disconnect")); void reader.cancel().catch(() => {}); return; } diff --git a/ui/src/adapters/codex-local/config-fields.tsx b/ui/src/adapters/codex-local/config-fields.tsx index 148caa904d9..9e161eb72d6 100644 --- a/ui/src/adapters/codex-local/config-fields.tsx +++ b/ui/src/adapters/codex-local/config-fields.tsx @@ -302,7 +302,7 @@ export function CodexLocalConfigFields({ )} {runnerManaged && runnerProvider === "grok_bot" && } {runnerManaged && isCursorCloud && <> -

Experimental. Runs in Cursor Cloud and requires a local Paperclip computer. Cursor manages command permissions; effective model identity is unavailable. Agent environment settings support only CURSOR_API_KEY. Configure hosted secrets in your Cursor Cloud environment.

+

Experimental. Cursor hosts this agent’s execution environment. A Paperclip computer is not required. Cursor manages command permissions; effective model identity is unavailable. Agent environment settings support only CURSOR_API_KEY. Configure hosted secrets in your Cursor Cloud environment.

{[ { key: "repoUrl", label: "GitHub repository", fallback: "", legacy: undefined, placeholder: "https://github.com/your-org/repo", hint: "Select a repository or a named Cursor cloud environment." }, { key: "repoRef", label: "Starting ref", fallback: config.repoStartingRef ?? "", legacy: "repoStartingRef", placeholder: "Repository default" }, diff --git a/ui/src/components/AgentConfigForm.render.test.tsx b/ui/src/components/AgentConfigForm.render.test.tsx index 7cba9d2a5d6..7877501077d 100644 --- a/ui/src/components/AgentConfigForm.render.test.tsx +++ b/ui/src/components/AgentConfigForm.render.test.tsx @@ -1324,6 +1324,49 @@ describe("AgentConfigForm environment selector", () => { expect(result.container.querySelector('[aria-label="Environment override"]')).toBeNull(); }); + it.each([null, "saved-computer"])("tests native Cursor Cloud without a Computer when the saved target is %s", async defaultEnvironmentId => { + mockInstanceSettingsApi.getExperimental.mockResolvedValue({ enableEnvironments: true, enableManagedSandboxOnly: true }); + mockInstanceSettingsApi.get.mockResolvedValue({ defaultEnvironmentId: "instance-computer" }); + mockAdaptersApi.list.mockResolvedValue([{ type: "cursor_cloud", supportedRunners: ["paperclip", "legacy"], defaultRunner: "legacy" }]); + const result = await renderForm([], { + adapterType: "paperclip_runner", defaultEnvironmentId, + adapterConfig: { provider: "cursor_cloud", model: "claude-sonnet-4-5", repoUrl: "https://github.com/example/qualification" }, + }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + + expect(result.container.querySelector('[aria-label="Environment override"]')).toBeNull(); + await runTest(result.container); + expect(mockAgentsApi.testEnvironment).toHaveBeenCalledExactlyOnceWith("company-1", "paperclip_runner", expect.objectContaining({ environmentId: null, agentId: "agent-1" })); + expect(mockAgentsApi.qualifyRunner).not.toHaveBeenCalled(); + expect(mockEnvironmentsApi.probe).not.toHaveBeenCalled(); + }); + + it("clears a Computer inherited by a new native Cursor Cloud draft", async () => { + mockInstanceSettingsApi.getExperimental.mockResolvedValue({ enableEnvironments: true, enableManagedSandboxOnly: true }); + mockAdaptersApi.list.mockResolvedValue([{ type: "cursor_cloud", supportedRunners: ["paperclip", "legacy"], defaultRunner: "legacy" }]); + const result = await renderCreateForm([], { + adapterType: "paperclip_runner", runner: "paperclip", defaultEnvironmentId: "previous-computer", + adapterSchemaValues: { provider: "cursor_cloud", model: "claude-sonnet-4-5" }, + }); + roots.push(result.root); + expect(result.onChange).toHaveBeenCalledWith({ defaultEnvironmentId: "" }); + expect(result.container.querySelector('[aria-label="Environment override"]')).toBeNull(); + }); + + it("tests the native default explicitly when creating Cursor Cloud with automatic runner selection", async () => { + mockInstanceSettingsApi.getExperimental.mockResolvedValue({ enableEnvironments: true, enableManagedSandboxOnly: true }); + mockAdaptersApi.list.mockResolvedValue([{ type: "cursor_cloud", supportedRunners: ["paperclip"], defaultRunner: "paperclip" }]); + const result = await renderCreateForm([], { + adapterType: "cursor_cloud", runner: "auto", model: "claude-sonnet-4-5", + adapterSchemaValues: { provider: "cursor_cloud", repoUrl: "https://github.com/example/qualification" }, + }, { showAdapterTestEnvironmentButton: true }); + roots.push(result.root); + await runTest(result.container); + expect(mockAgentsApi.testEnvironment).toHaveBeenCalledExactlyOnceWith("company-1", "cursor_cloud", expect.objectContaining({ + environmentId: null, runner: "paperclip", + })); + }); + it("requests the native Kimi model catalog while preserving an explicit saved model", async () => { mockAgentsApi.adapterModels.mockResolvedValue([]); const result = await renderForm([], { diff --git a/ui/src/components/AgentConfigForm.tsx b/ui/src/components/AgentConfigForm.tsx index d5f65b5d45e..1bb9a896ba4 100644 --- a/ui/src/components/AgentConfigForm.tsx +++ b/ui/src/components/AgentConfigForm.tsx @@ -659,6 +659,11 @@ export function AgentConfigForm(props: AgentConfigFormProps) { ? (runner === "paperclip" || (runner === "auto" && runnerAvailability?.defaultRunner === "paperclip") ? "paperclip_runner" : harnessType) : adapterType; const isCursorCloudRunner = harnessType === "cursor_cloud" && fieldsAdapterType === "paperclip_runner"; + useEffect(() => { + if (props.mode !== "create" || !isCursorCloudRunner || !props.values.defaultEnvironmentId) return; + props.onChange({ defaultEnvironmentId: "" }); + }, [props.mode, isCursorCloudRunner, isCreate ? props.values.defaultEnvironmentId : undefined, + isCreate ? props.onChange : undefined]); useEffect(() => { if (props.mode !== "create" || !isRunnerHarness) return; const resolvedRunner = runner === "auto" ? runnerAvailability?.defaultRunner : runner; @@ -992,10 +997,10 @@ export function AgentConfigForm(props: AgentConfigFormProps) { // Only applies in create mode and only once the K8s environment is loaded; if // none is available the UI surfaces a notice instead of silently selecting it. useEffect(() => { - if (!isCreate || !set || !forcedKubernetes || !kubernetesEnvironment) return; + if (!isCreate || !set || isCursorCloudRunner || !forcedKubernetes || !kubernetesEnvironment) return; if (currentDefaultEnvironmentId === kubernetesEnvironment.id) return; set({ defaultEnvironmentId: kubernetesEnvironment.id }); - }, [isCreate, set, forcedKubernetes, kubernetesEnvironment, currentDefaultEnvironmentId]); + }, [isCreate, set, isCursorCloudRunner, forcedKubernetes, kubernetesEnvironment, currentDefaultEnvironmentId]); const runnableEnvironments = useMemo( () => environments.filter((environment) => { @@ -1172,7 +1177,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { let settings = instanceSettings; let environmentList = environments; let managedSandboxOnly = experimentalSettings?.enableManagedSandboxOnly === true; - if (!rawCurrentDefaultEnvironmentId) { + if (!isCursorCloudRunner && !rawCurrentDefaultEnvironmentId) { // The agent has no own environment, so the Test resolves the instance // default, the local default, or the managed sandbox. Resolve the // settings, the environment list, and the managed-sandbox-only policy @@ -1210,7 +1215,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { // policy is on, probe the managed sandbox the real run uses instead. The // resolver throws when no managed sandbox is available, which the mutation // surfaces as a fail-closed error rather than a local host probe. - const environmentId = resolveAdapterTestEnvironmentId({ + const environmentId = isCursorCloudRunner ? null : resolveAdapterTestEnvironmentId({ agentDefaultEnvironmentId: rawCurrentDefaultEnvironmentId || null, instanceDefaultEnvironmentId: settings?.defaultEnvironmentId ?? null, localDefaultEnvironmentId: resolveLocalDefaultEnvironmentId(environmentList), @@ -1227,6 +1232,9 @@ export function AgentConfigForm(props: AgentConfigFormProps) { const aiConnection = isCreate ? undefined : aiRuntimeConnectionBindingSchema.safeParse( (overlay.runtime.runtimeConfig as Record | undefined)?.aiConnection ?? props.agent.runtimeConfig.aiConnection, ).data; + const testRunnerSelection = isCursorCloudRunner + ? { runner: "paperclip" as const } + : isRunnerHarness && (isCreate ? runner !== "auto" : overlay.runner !== undefined) ? { runner } : {}; if (!isCreate && props.agent.adapterType === "paperclip_runner" && adapterType === "paperclip_runner" && runnerAvailability?.defaultRunner !== "paperclip" && !isExternalRunner && aiConnection?.mode !== "router" && !isDirty && environmentId && canQualifyRunnerEnvironment(environmentList.find(environment => environment.id === environmentId))) { @@ -1246,9 +1254,9 @@ export function AgentConfigForm(props: AgentConfigFormProps) { : adapterConfig.provider === "acpx" && adapterConfig.acpxAgent === "claude" ? "claude_local" : adapterType : adapterType; - return testAgentSetup({ companyId: selectedCompanyId, adapterType, providerAdapter, adapterConfig, agentId, aiConnection, environmentId, ...(isRunnerHarness && (isCreate ? runner !== "auto" : overlay.runner !== undefined) ? { runner } : {}) }); + return testAgentSetup({ companyId: selectedCompanyId, adapterType, providerAdapter, adapterConfig, agentId, aiConnection, environmentId, ...testRunnerSelection }); } - return agentsApi.testEnvironment(selectedCompanyId, adapterType, { adapterConfig, agentId, aiConnection, environmentId, ...(isRunnerHarness && (isCreate ? runner !== "auto" : overlay.runner !== undefined) ? { runner } : {}) }); + return agentsApi.testEnvironment(selectedCompanyId, adapterType, { adapterConfig, agentId, aiConnection, environmentId, ...testRunnerSelection }); }, }); const [testActionPending, setTestActionPending] = useState(false); @@ -1718,7 +1726,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { )} {/* ---- Execution ---- */} - {forcedKubernetes ? ( + {isCursorCloudRunner ? null : forcedKubernetes ? ( // Instance execution policy forces the managed Kubernetes sandbox // (executionMode=kubernetes): never offer local / non-Kubernetes targets. // Render the environment read-only instead of the selectable picker. diff --git a/ui/src/components/NewAgentDialog.test.tsx b/ui/src/components/NewAgentDialog.test.tsx index e3c5a89ca09..df5d4c3cb5f 100644 --- a/ui/src/components/NewAgentDialog.test.tsx +++ b/ui/src/components/NewAgentDialog.test.tsx @@ -218,6 +218,54 @@ it("offers one Codex harness without a runner/provider tile", async () => { expect(document.querySelectorAll('input[value="codex_local"]')).toHaveLength(1); expect(document.querySelector("select")).toBeNull(); }); +it("opens native Cursor Cloud from the ordinary hosted adapter picker", async () => { + await act(async () => { + cache.setQueryData(queryKeys.health, { status: "ok", cloud: { managed: true } }); + cache.setQueryData(queryKeys.adapters.availability("company-1", null), [...state.adapters, + { type: "cursor_cloud", loaded: true, supportedRunners: ["paperclip"], defaultRunner: "paperclip" }]); + }); + await name(); + const cursor = document.querySelector('input[value="cursor_cloud"]'); + expect(cursor).not.toBeNull(); + await act(async () => cursor!.click()); + await click("Configure agent"); + expect(agents.update).toHaveBeenCalledWith(agents.create.mock.calls[0][1].setupId, expect.objectContaining({ + adapterType: "paperclip_runner", runner: "paperclip", adapterConfig: { provider: "cursor_cloud" }, + }), "company-1"); + const query = new URL(state.navigate.mock.calls[0][0], "http://local").searchParams; + expect(query.get("adapterType")).toBe("paperclip_runner"); + expect(query.get("runnerProvider")).toBe("cursor_cloud"); + expect(query.get("runner")).toBe("paperclip"); +}); + +it("hides explicitly unsupported harnesses while keeping the qualified Cursor API card", async () => { + await act(async () => { + cache.setQueryData(queryKeys.health, { status: "ok", cloud: { managed: true } }); + cache.setQueryData(queryKeys.adapters.availability("company-1", null), [ + { type: "codex_local", loaded: true, supportedRunners: [] }, + { type: "opencode_local", loaded: true, supportedRunners: [] }, + { type: "paperclip_runner", loaded: true, supportedRunners: [] }, + { type: "cursor_cloud", loaded: true, supportedRunners: ["paperclip"], defaultRunner: "paperclip" }, + ]); + }); + await name(); + expect(document.querySelector('input[value="codex_local"]')).toBeNull(); + expect(document.querySelector('input[value="opencode_local"]')).toBeNull(); + expect(document.querySelector('input[value="copilot_runtime"]')).toBeNull(); + expect(document.querySelector('input[value="antigravity_runtime"]')).toBeNull(); + expect(document.querySelector('input[value="cursor_cloud"]')).not.toBeNull(); +}); + +it.each(["gate off", "native unavailable"])("keeps hosted Cursor Cloud hidden when %s", async reason => { + await act(async () => { + cache.setQueryData(queryKeys.health, { status: "ok", cloud: { managed: true } }); + if (reason === "gate off") cache.setQueryData(queryKeys.instance.experimentalSettings, { enableNativeRunner: false }); + cache.setQueryData(queryKeys.adapters.availability("company-1", null), [...state.adapters, + { type: "cursor_cloud", loaded: true, supportedRunners: reason === "native unavailable" ? ["legacy"] : ["paperclip"], defaultRunner: "legacy" }]); + }); + await name(); + expect(document.querySelector('input[value="cursor_cloud"]')).toBeNull(); +}); it.each([true, false, undefined])("opens saved-token Copilot setup without an experimental override (%s)", async enableNativeRunner => { await act(async () => cache.setQueryData(queryKeys.instance.experimentalSettings, { enableNativeRunner })); await name(); diff --git a/ui/src/components/TaskChatThread.test.tsx b/ui/src/components/TaskChatThread.test.tsx index 4cffcc90ec3..b628c4e224f 100644 --- a/ui/src/components/TaskChatThread.test.tsx +++ b/ui/src/components/TaskChatThread.test.tsx @@ -23,6 +23,8 @@ import { pendingConnectionIntentInteraction } from "@/fixtures/issueThreadIntera import { TaskChatScrollNavigation, useTaskChatScrollNavigation } from "./task-chat/scroll-navigation"; import { CoordinatorPresentation } from "./coordinator/CoordinatorPresentation"; import type { HeartbeatRunEvent } from "@paperclipai/shared"; +import type { HeartbeatRun } from "@paperclipai/shared"; +import { waitForStoppedRuns } from "@/lib/wait-for-stopped-runs"; const transcriptState = vi.hoisted(() => ({ transcriptByRun: new Map(), @@ -5221,6 +5223,36 @@ describe("TaskChatThread live transcript", () => { }); describe("TaskChatThread composer execution controls", () => { + it("reconciles a timed-out Stop from exact company-bound historical proof after its live action disappears", async () => { + vi.useFakeTimers(); + try { + const run = { id: "stopped-run", status: "running", runtimeMode: "native" as const, invocationSource: "issue", triggerDetail: null, + startedAt: "2026-11-03T15:00:00Z", finishedAt: null, createdAt: "2026-11-03T15:00:00Z", agentId: "agent-1", agentName: "Alex", adapterType: "paperclip_runner" }; + const requested: Partial & { runtimeMode: "native" } = { + id: run.id, companyId: "company", status: "cancelled", runtimeMode: "native", + resultJson: { executionCancellation: { state: "requested" } }, + }; + const onStop = () => waitForStoppedRuns([run.id], { timeoutMs: 1000, getRun: async () => requested as HeartbeatRun }); + const base = { comments: [], onAdd: async () => {}, issueStatus: "in_progress", companyId: "company" }; + render(); + await act(async () => { container.querySelector('[data-testid="task-chat-composer-stop"]')!.click(); }); + await act(async () => { await vi.advanceTimersByTimeAsync(1000); }); + expect(container.querySelector('[role="alert"]')?.textContent).toContain("stop was requested"); + const historical = { runId: run.id, agentId: run.agentId, adapterType: run.adapterType, createdAt: run.createdAt, + startedAt: run.startedAt, finishedAt: run.startedAt, status: "cancelled", runtimeMode: "native" as const, + resultJson: { executionCancellation: { state: "acknowledged" }, nativeCancellation: { + schema: "paperclip.native-cancellation.v1", runId: run.id, companyId: "company", dispatchState: "acknowledged" } } }; + render(); + await act(async () => {}); expect(container.querySelector('[role="alert"]')).not.toBeNull(); + render(); + await act(async () => {}); expect(container.querySelector('[role="alert"]')).not.toBeNull(); + render(); + await act(async () => {}); expect(container.querySelector('[role="alert"]')).not.toBeNull(); + render(); + await act(async () => {}); expect(container.querySelector('[role="alert"]')).toBeNull(); + } finally { vi.useRealTimers(); } + }); it.each(["process", "paperclip_runner"])("passes the task's stop action through for %s execution", async (adapterType) => { const onStop = vi.fn(async () => {}); const run = { diff --git a/ui/src/components/TaskChatThread.tsx b/ui/src/components/TaskChatThread.tsx index 5a271aa9c01..5c8b4682177 100644 --- a/ui/src/components/TaskChatThread.tsx +++ b/ui/src/components/TaskChatThread.tsx @@ -143,6 +143,7 @@ import { issuesApi } from "@/api/issues"; import { queryKeys } from "@/lib/queryKeys"; import { useQueryClient } from "@tanstack/react-query"; import { TaskChatPresentationProvider } from "@/components/task-chat/presentation-mode"; +import { isRunStopConfirmed } from "@/lib/wait-for-stopped-runs"; const EMPTY_BROWSERS: TaskBrowser[] = []; const COMPACT_RUN_LOG_READ_LIMIT_BYTES = 32_000; @@ -587,6 +588,11 @@ export function TaskChatThread(props: TaskChatThreadProps) { resumeAssigneePending = false, onOpenSkill, } = props; + const confirmedStoppedRunIds = useMemo(() => new Set((linkedRuns ?? []).filter((run) => + companyId && (run.runtimeMode === "legacy" || run.runtimeMode === "native") && isRunStopConfirmed({ + id: run.runId, companyId, status: run.status, runtimeMode: run.runtimeMode, resultJson: run.resultJson ?? null, + }), + ).map((run) => run.runId)), [companyId, linkedRuns]); const retryFailedRunHandler = isTerminalIssueStatus(issueStatus) || interactions?.some((interaction) => interaction.status === "pending") || @@ -3485,6 +3491,7 @@ export function TaskChatThread(props: TaskChatThreadProps) { !("clientStatus" in comment && comment.clientStatus) ).map((comment) => comment.clientRequestId!))} onStop={liveRun ? onCancelRun : undefined} + confirmedStoppedRunIds={confirmedStoppedRunIds} stopPending={stopPending} stopScope={stopScope} workMode={issueWorkMode} diff --git a/ui/src/components/new-agent/AgentBasicsDialog.tsx b/ui/src/components/new-agent/AgentBasicsDialog.tsx index fabfce6ad1c..8c5554496f8 100644 --- a/ui/src/components/new-agent/AgentBasicsDialog.tsx +++ b/ui/src/components/new-agent/AgentBasicsDialog.tsx @@ -89,7 +89,7 @@ export function AgentBasicsDialog({ queryFn: () => adaptersApi.list({ companyId: companyId! }), enabled: open && Boolean(companyId), }); - const nativeRunner = adapters?.find(adapter => adapter.type === "paperclip_runner" && adapter.loaded && !adapter.disabled); + const nativeRunner = adapters?.find(adapter => adapter.type === "paperclip_runner" && adapter.loaded && !adapter.disabled && adapter.supportedRunners?.length !== 0); const copilotAvailable = Boolean(nativeRunner) && PAPERCLIP_RUNNER_ACPX_PROFILES.some(profile => profile.value === "copilot" && profile.qualified); const kimiAvailable = Boolean(nativeRunner) && experimental.data?.enableNativeRunner === true && PAPERCLIP_RUNNER_ACPX_PROFILES.some(profile => profile.value === "kimi" && profile.qualified); @@ -100,18 +100,20 @@ export function AgentBasicsDialog({ (adapter) => adapter.loaded && !adapter.disabled && + adapter.supportedRunners?.length !== 0 && adapter.type !== "paperclip_runner" && isNewAgentAdapterAllowed(adapter.type, { cloud, nativeRunnerEnabled: experimental.data?.enableNativeRunner === true, }) && + (!cloud || adapter.type !== "cursor_cloud" || adapter.supportedRunners?.includes("paperclip")) && !["process", "http"].includes(adapter.type) && !getAdapterDisplay(adapter.type).comingSoon, ), ...(copilotAvailable ? [{ ...nativeRunner!, type: "copilot_runtime" }] : []), ...(antigravityAvailable ? [{ ...nativeRunner!, type: "antigravity_runtime" }] : []), ...(kimiAvailable ? [{ ...nativeRunner!, type: "kimi_runtime" }] : [])]; const experimentalRunnerAvailable = experimental.data?.enableNativeRunner === true - && adapters?.some(adapter => adapter.type === "paperclip_runner" && adapter.loaded && !adapter.disabled); + && adapters?.some(adapter => adapter.type === "paperclip_runner" && adapter.loaded && !adapter.disabled && adapter.supportedRunners?.length !== 0); const validAdapter = choices.some((adapter) => adapter.type === adapterType) || (experimentalRunnerAvailable && adapterType === "paperclip_runner" && (runnerProvider !== "antigravity" || antigravityQualified)); async function submit() { @@ -142,8 +144,10 @@ export function AgentBasicsDialog({ if (active.current) setName(agent.name); } saved.current = agent; - const nextAdapter = ["openai_dot", "copilot_runtime", "antigravity_runtime", "kimi_runtime"].includes(adapterType) ? "paperclip_runner" : adapterType; - const nextProvider = adapterType === "kimi_runtime" ? "kimi" : adapterType === "antigravity_runtime" ? "antigravity" : adapterType === "copilot_runtime" ? "copilot" : adapterType === "openai_dot" ? "openai_dot" : adapterType === "paperclip_runner" ? runnerProvider : "codex"; + const hostedCursor = cloud && adapterType === "cursor_cloud"; + const nextAdapter = hostedCursor || ["openai_dot", "copilot_runtime", "antigravity_runtime", "kimi_runtime"].includes(adapterType) ? "paperclip_runner" : adapterType; + const nextProvider = hostedCursor ? "cursor_cloud" : adapterType === "kimi_runtime" ? "kimi" : adapterType === "antigravity_runtime" ? "antigravity" : adapterType === "copilot_runtime" ? "copilot" : adapterType === "openai_dot" ? "openai_dot" : adapterType === "paperclip_runner" ? runnerProvider : "codex"; + const nextRunner = hostedCursor ? "paperclip" as const : "auto" as const; // Pending providers can show setup while the saved name-only draft stays held. const pendingAntigravity = nextProvider === "antigravity" && !antigravityQualified; if (step === "adapter" || (!creatingRecord && agent.name !== name.trim())) { @@ -155,7 +159,7 @@ export function AgentBasicsDialog({ setupAdapterChoice: { adapterType: nextAdapter, runnerProvider: nextProvider, runner: "auto" } }, } : {}), ...(step === "adapter" && !pendingAntigravity ? { - adapterType: nextAdapter, runner: "auto" as const, + adapterType: nextAdapter, runner: nextRunner, adapterConfig: nextAdapter === "paperclip_runner" ? { provider: ["claude", "grok", "copilot", "antigravity", "kimi"].includes(nextProvider) ? "acpx" : nextProvider, ...(["claude", "grok", "copilot", "antigravity", "kimi"].includes(nextProvider) ? { acpxAgent: nextProvider } : {}), @@ -171,7 +175,7 @@ export function AgentBasicsDialog({ if (!active.current) return; if (step === "name") setStep("adapter"); else onContinue({ name: agent.name, adapterType: nextAdapter, runnerProvider: nextProvider, - runner: "auto", setupAgentId: agent.id }); + runner: nextRunner, setupAgentId: agent.id }); } catch (cause) { if (active.current) setSaveError(cause instanceof Error ? cause.message : "Could not save agent setup."); } finally { diff --git a/ui/src/components/new-agent/NewAgentSetup.tsx b/ui/src/components/new-agent/NewAgentSetup.tsx index e9c319587fb..0defa9a61b9 100644 --- a/ui/src/components/new-agent/NewAgentSetup.tsx +++ b/ui/src/components/new-agent/NewAgentSetup.tsx @@ -315,7 +315,7 @@ function Setup({ let environmentId: string | null = null; let environmentError: string | null = null; try { - environmentId = forced.forced + environmentId = nativeCursorCloud ? null : forced.forced ? (forced.kubernetesEnvironment?.id ?? null) : resolveAdapterTestEnvironmentId({ agentDefaultEnvironmentId: environmentOverride || null, @@ -414,6 +414,7 @@ function Setup({ const cloud = Boolean(useCloudInstance()); const available = (!nativeCursorCloud || experimental.data?.enableNativeRunner === true) && + (!cloud || adapterType !== "cursor_cloud" || nativeCursorCloud) && isNewAgentAdapterAllowed(adapterType, { cloud, nativeRunnerEnabled: experimental.data?.enableNativeRunner === true, @@ -447,12 +448,13 @@ function Setup({ !general.error && !agents.error && !caps.error && - (!(managedOnly || forced.forced) || environmentId), + (nativeCursorCloud || !(managedOnly || forced.forced) || environmentId), ); const needsCurrentSetup = ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(adapterType); const preparingComputer = created?.pauseReason === "setup" && ["preparing", "pausing"].includes(created.lifecycleState ?? ""); - const environmentSelectionSaved = (!environmentOverride && created?.defaultEnvironmentId == null) - || (created?.defaultEnvironmentId ?? null) === (environmentOverride || (forced.forced || managedOnly ? environmentId : null)); + const environmentSelectionSaved = nativeCursorCloud + || (!environmentOverride && created?.defaultEnvironmentId == null) + || (created?.defaultEnvironmentId ?? null) === (environmentOverride || (forced.forced || managedOnly ? environmentId : null)); const ready = (pendingAntigravity || Boolean(created) && environmentSelectionSaved) && !environmentSaving && !preparingComputer && connectionSettingsReady && (!needsCurrentSetup || ( envs.isSuccess && !envs.isFetching && settings.isSuccess && !settings.isFetching @@ -614,7 +616,7 @@ function Setup({ provider: ["claude", "grok", "copilot", "pi", "antigravity", "kimi"].includes(runnerProvider) ? "acpx" : runnerProvider, ...(["claude", "grok", "copilot", "pi", "antigravity", "kimi"].includes(runnerProvider) ? { acpxAgent: runnerProvider } : {}), ...(antigravityCandidate ? { model: antigravityCandidate.model } : {}), - } : {}, defaultEnvironmentId: environmentOverride || (forced.forced || managedOnly ? environmentId : null), + } : {}, defaultEnvironmentId: nativeCursorCloud ? null : environmentOverride || (forced.forced || managedOnly ? environmentId : null), runtimeConfig: buildNewAgentRuntimeConfig({ heartbeatEnabled: false }), }).then(async agent => { await cache.cancelQueries({ queryKey: queryKeys.agents.detail(agent.id) }); @@ -677,7 +679,7 @@ function Setup({ submitted = true; const agent = await agentsApi.update(created.id, { adapterType: requestAdapterType, runner, adapterConfig: config, - defaultEnvironmentId: selectedEnvironment || (forced.forced || managedOnly ? environmentId : null), + defaultEnvironmentId: nativeCursorCloud ? null : selectedEnvironment || (forced.forced || managedOnly ? environmentId : null), runtimeConfig: { ...created.runtimeConfig, ...(binding ? { aiConnection: binding } : {}) }, ...(antigravityCandidate ? { metadata: { ...created.metadata, setupAdapterPending: false, setupAdapterChoice: undefined } } : {}), ...(nextConnection?.applyStoredClaudeLogin ? { applyStoredClaudeLogin: true } : {}), @@ -805,7 +807,7 @@ function Setup({ adapterType: requestAdapterType, ...(["codex_local", "claude_local", "opencode_local", "pi_local", "cursor_cloud"].includes(adapterType) && runner !== "auto" ? { runner } : {}), adapterConfig: config, - defaultEnvironmentId: + defaultEnvironmentId: nativeCursorCloud ? null : environmentOverride || (forced.forced || managedOnly ? environmentId : null), runtimeConfig: { ...buildNewAgentRuntimeConfig({ heartbeatEnabled: false }), ...(aiBinding ? { aiConnection: aiBinding } : {}) }, @@ -995,7 +997,7 @@ function Setup({ This adapter is unavailable. Choose an enabled adapter.

)} - {(managedOnly || forced.forced) && + {!nativeCursorCloud && (managedOnly || forced.forced) && !envs.isPending && !environmentId && (

@@ -1444,7 +1446,7 @@ function Setup({ )} {adapterType === "cursor_cloud" && ( <> - {nativeCursorCloud &&

Experimental. Requires a local Paperclip computer. Cursor manages cloud command permissions; the effective model is not acknowledged.

} + {nativeCursorCloud &&

Experimental. Cursor hosts this agent’s execution environment. A Paperclip computer is not required. Cursor manages cloud command permissions; the effective model is not acknowledged.

}
{ setRunner(value); resetTest(); }} /> } - {adapterType !== "hermes_gateway" && (adapterType !== "cursor_cloud" || nativeCursorCloud) && ( + {adapterType !== "hermes_gateway" && adapterType !== "cursor_cloud" && (

Environment