From 35511366dcbe359626de9a698c71b7a583432a2b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 10 Oct 2026 07:06:31 -0500 Subject: [PATCH 1/3] perf(server): schedule browser cleanup only while sessions exist (#15757) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Hosted instances must become quiet when they have no work. > - Browser Use queried its session table every three seconds, even when empty. > - Existing session rows already store the next poll time and cleanup lease. > - This pull request uses commit signals and those saved deadlines to schedule cleanup. > - Empty instances stop browser database scans. Live sessions keep their safety checks. ## Linked Issues or Issue Description **What existing behavior does this improve?** Browser Use background cleanup and recovery. This follows #15625 and #15750. **Subsystem affected** server/ — REST API and orchestration services. **Current behavior** The app scans for due sessions every three seconds, including instances that have never used Browser Use. **Proposed behavior** Scan once at startup. Wake after session creation, provider-create settlement, stop requests, or credential-removal cleanup. Schedule the next check from `nextPollAt` and `leaseUntil`. Stop scheduling when all sessions are terminal. **Reason and benefit** Remove empty database traffic so idle stacks and their database connections can become quiet. **Breaking changes** No API or schema change. Running and idle sessions retain provider polling, cost accounting, access checks, expiry checks, and cleanup retries. ## What Changed - Register browser cleanup with the existing work coordinator. Remove the app's three-second interval. - Signal existing session transactions before writes. No new tables or provider infrastructure. - Restore deadlines from live sessions and their leases. Wake after a credential-removal sweep releases its lease. - Wait for concurrent cleanup instead of returning while it still runs. Drain and standby preserve deferred work. - Add scheduler tests and database-backed commit and restart coverage. Update the connector guide and sleep inventory. ## Verification - Passed: focused browser worker and shared coordinator tests, 17 tests. - Passed: browser lifecycle, route-scope, viewport, and transport tests, 50 tests. The lifecycle suite uses real local Postgres and a simulated provider. - Passed: `pnpm -r typecheck` and `pnpm build`. - Passed: all hosted test groups for this commit. Started full local `pnpm test:run`, then stopped its duplicate serial run after the complete hosted sharded suite passed; no full local-suite pass is claimed. - Passed: direct server TypeScript check. Greptile reviewed `99c3217a2dea505be94081e729941e542e6fbc03` at 5/5 with no findings. All 54 applicable hosted checks passed; the two Storybook checks were skipped because this PR has no UI changes. ## Risks - Process-local signals cover services that share the instrumented database owner. Independent processes and direct SQL writes require startup recovery; this does not add cross-process notification. - Live sessions still poll. This removes empty scans; it does not make a running hosted browser sleep. - Cleanup uses the existing provider protocol and lease fences. Tests simulate the provider; no paid live browser was created for this scheduling change. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/connections/BROWSER-USE.md | 7 +- ...-instance-sleep-and-scheduler-inventory.md | 12 +- .../__tests__/browser-use-connection.test.ts | 43 ++++ server/src/__tests__/browser-use-work.test.ts | 108 +++++++++ server/src/app.ts | 9 +- server/src/services/browser-use-work.ts | 23 ++ server/src/services/browser-use.ts | 226 ++++++++++-------- .../services/delivery-work-notifications.ts | 1 + 8 files changed, 316 insertions(+), 113 deletions(-) create mode 100644 server/src/__tests__/browser-use-work.test.ts create mode 100644 server/src/services/browser-use-work.ts diff --git a/doc/connections/BROWSER-USE.md b/doc/connections/BROWSER-USE.md index 72104d93981..1a1a3c7769e 100644 --- a/doc/connections/BROWSER-USE.md +++ b/doc/connections/BROWSER-USE.md @@ -72,7 +72,12 @@ runs and browser instances. Each hosted run records its invocation, parent run, event cursor and cost checkpoint. A leased reconciler polls minimal status, drains bounded event pages, discovers browser instances by conversation, and checks each instance's authoritative conversation association. Restarting the -server resumes reconciliation. A lost paid-create response becomes an explicit +server resumes reconciliation. The app schedules the next check from saved +session deadlines and cleanup leases. Session writes wake it after commit. +With no live sessions it performs no recurring browser database scan. Existing +sessions still poll provider state (about three seconds while running, fifteen +while idle), enforce expiry and access changes, and retry failed cleanup. +A lost paid-create response becomes an explicit unknown outcome and is never submitted again automatically; the operator must inspect Browser Use before starting replacement work. diff --git a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md index 0646390908a..24db42b22ac 100644 --- a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md +++ b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md @@ -438,10 +438,14 @@ Longer polling intervals alone do not complete an item. catch-up remain the recovery boundaries. Source: [email-channels.ts](../../server/src/services/email-channels.ts). -- [ ] **JOB-03 — Browser-use cleanup.** Current: every **3s** with due-session - queries even when empty. Change: register actual session reconciliation/expiry - deadlines. **Complete when:** no sessions means no queries, while active leases, - expiry, and failed cleanup remain recoverable. Sources: +- [x] **JOB-03 — Browser-use cleanup.** Replaced the empty **3s** scan with + transaction-aware session signals and deadlines from existing `nextPollAt` and + `leaseUntil` columns. Startup restores unfinished work; no live sessions means + no recurring browser SQL or timer. Active and idle sessions retain provider + polling, expiry checks, lease renewal, and cleanup retries. Credential-removal + sweeps signal lease release so background cleanup resumes promptly. Standby + suppresses SQL; idle drain defers dispatch with a memory-only wake. No new + tables. Independent DB roots/direct SQL require restart recovery. Sources: [app.ts](../../server/src/app.ts), [browser-use.ts](../../server/src/services/browser-use.ts). diff --git a/server/src/__tests__/browser-use-connection.test.ts b/server/src/__tests__/browser-use-connection.test.ts index b206fabe93c..f9282b9f082 100644 --- a/server/src/__tests__/browser-use-connection.test.ts +++ b/server/src/__tests__/browser-use-connection.test.ts @@ -34,6 +34,8 @@ import { import { toolAccessService } from "../services/tool-access.js"; import { createToolGatewayService } from "../services/tool-gateway.js"; import { browserUseViewports } from "../services/browser-use-viewport.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerBrowserUseCleanup } from "../services/browser-use-work.js"; import { browserUseService } from "../services/browser-use.js"; import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "../services/connector-runtime.js"; import { registerAssignedMcpGateway } from "../services/native-runtime/assigned-mcp-tools.js"; @@ -313,6 +315,47 @@ const actor = { actorType: "user" as const, actorId: "browser-reviewer" }; }, }; } + it("wakes the app worker from request-service commits and restores durable poll and lease deadlines", async () => { + const f = await fixture(); + const background = browserUseService(db, f.request); + const sweep = vi.spyOn(background, "sweep"); + const onError = vi.fn(); + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError }); + try { + await registerBrowserUseCleanup(coordinator, background, () => true).ready; + expect(coordinator.nextWakeAt()).toBeNull(); + expect(await background.nextSweepAt()).toBeNull(); + const started = await f.service.execute(f.binding, f.grant, randomUUID(), "browser_start", { task: "Read example.com" }) as { sessionId: string }; + await vi.waitFor(() => expect(sweep.mock.calls.length).toBeGreaterThan(1)); + await coordinator.stop(); + expect(onError).not.toHaveBeenCalled(); + + const poll = new Date(Date.now() + 60_000); + const lease = new Date(Date.now() + 180_000); + await db.update(browserUseSessions).set({ nextPollAt: poll, leaseUntil: lease }) + .where(eq(browserUseSessions.id, started.sessionId)); + expect(await background.nextSweepAt()).toBe(lease.getTime()); + const restarted = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError }); + try { + await registerBrowserUseCleanup(restarted, browserUseService(db, f.request), () => true).ready; + expect(restarted.nextWakeAt()).toBe(lease.getTime()); + } finally { await restarted.stop(); } + await db.update(browserUseSessions).set({ leaseUntil: new Date(0) }) + .where(eq(browserUseSessions.id, started.sessionId)); + expect(await background.nextSweepAt()).toBe(poll.getTime()); + await db.update(browserUseSessions).set({ leaseUntil: null }) + .where(eq(browserUseSessions.id, started.sessionId)); + expect(await background.nextSweepAt()).toBe(poll.getTime()); + const [session] = await db.select().from(browserUseSessions).where(eq(browserUseSessions.id, started.sessionId)); + await f.service.control(session, "end"); + expect(await background.nextSweepAt()).toBeLessThan(poll.getTime()); + await db.update(browserUseSessions).set({ status: "closed" }).where(eq(browserUseSessions.id, started.sessionId)); + expect(await background.nextSweepAt()).toBeNull(); + } finally { + await coordinator.stop(); + await db.update(browserUseSessions).set({ status: "closed" }).where(eq(browserUseSessions.companyId, f.company.id)); + } + }); it("keeps Cloud instructions out of universal skills and unassigned runtime overlays", async () => { const skills = await listPaperclipSkillEntries(fileURLToPath(new URL("../", import.meta.url)), [fileURLToPath(new URL("../../../skills", import.meta.url))]); expect(skills.some(skill => skill.runtimeName === "paperclip")).toBe(true); diff --git a/server/src/__tests__/browser-use-work.test.ts b/server/src/__tests__/browser-use-work.test.ts new file mode 100644 index 00000000000..ae8a2dfbec2 --- /dev/null +++ b/server/src/__tests__/browser-use-work.test.ts @@ -0,0 +1,108 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../services/delivery-work-notifications.js"; +import { registerBrowserUseCleanup } from "../services/browser-use-work.js"; + +const coordinators: ReturnType[] = []; +afterEach(async () => { + for (const coordinator of coordinators.splice(0)) await coordinator.stop(); + vi.useRealTimers(); +}); +function setup() { + vi.useFakeTimers(); + const execute = vi.fn(async () => [{ xid: "42", status: "committed" }]); + const owner = installDatabaseWorkSignals({ execute, async transaction(callback: (tx: any) => Promise) { + return callback({ execute }); + } }); + const enabled = vi.fn(() => true), primary = vi.fn(() => true), onError = vi.fn(); + const coordinator = createDeliveryWorkCoordinator({ + owner, canRun: () => enabled() && primary(), canReconcile: primary, onError, + }); + coordinators.push(coordinator); + const service = { sweep: vi.fn(async () => {}), nextSweepAt: vi.fn(async (): Promise => null) }; + const start = () => registerBrowserUseCleanup(coordinator, service, enabled); + const commit = () => owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.browser)); + return { service, coordinator, start, commit, enabled, primary, onError }; +} +describe("browser cleanup scheduling", () => { + it("stays quiet after an empty startup, wakes on commit, and stops after the final session closes", async () => { + const s = setup(); + await s.start().ready; + await vi.advanceTimersByTimeAsync(3_600_000); + expect(s.service.sweep).toHaveBeenCalledTimes(1); + expect(s.service.nextSweepAt).toHaveBeenCalledTimes(1); + expect(vi.getTimerCount()).toBe(0); + const due = Date.now() + 15_000; + s.service.nextSweepAt.mockResolvedValue(due); + await s.commit(); + await vi.advanceTimersByTimeAsync(1); + expect(s.service.sweep).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBe(due); + await vi.advanceTimersByTimeAsync(14_998); + expect(s.service.sweep).toHaveBeenCalledTimes(2); + s.service.nextSweepAt.mockResolvedValue(null); + await vi.advanceTimersByTimeAsync(1); + expect(s.service.sweep).toHaveBeenCalledTimes(3); + expect(vi.getTimerCount()).toBe(0); + }); + it("restores a future lease/retry deadline at startup and lets a control commit preempt it", async () => { + const s = setup(); + const due = Date.now() + 180_000; + s.service.nextSweepAt.mockResolvedValue(due); + await s.start().ready; + await vi.advanceTimersByTimeAsync(120_000); + expect(s.service.sweep).toHaveBeenCalledTimes(1); + await s.commit(); + await vi.advanceTimersByTimeAsync(1); + expect(s.service.sweep).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBe(due); + }); + it("retries a failed scan and bounds already-due batches", async () => { + const s = setup(); + s.service.sweep.mockRejectedValueOnce(new Error("database unavailable")); + await s.start().ready; + expect(s.onError).toHaveBeenCalledTimes(1); + s.service.nextSweepAt.mockResolvedValue(Date.now() - 1000); + await vi.advanceTimersByTimeAsync(3000); + expect(s.service.sweep).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBe(Date.now() + 1000); + s.service.nextSweepAt.mockResolvedValue(null); + await vi.advanceTimersByTimeAsync(1000); + expect(vi.getTimerCount()).toBe(0); + }); + it("does no session SQL during standby or drain and recovers a first commit after drain", async () => { + const s = setup(); + s.primary.mockReturnValue(false); + await s.start().ready; + await vi.advanceTimersByTimeAsync(6000); + expect(s.service.sweep).not.toHaveBeenCalled(); + expect(s.service.nextSweepAt).not.toHaveBeenCalled(); + s.primary.mockReturnValue(true); + s.enabled.mockReturnValue(false); + await s.commit(); + await vi.advanceTimersByTimeAsync(6000); + expect(s.service.sweep).not.toHaveBeenCalled(); + expect(s.service.nextSweepAt).not.toHaveBeenCalled(); + s.enabled.mockReturnValue(true); + await vi.advanceTimersByTimeAsync(3000); + expect(s.service.sweep).toHaveBeenCalledTimes(1); + expect(vi.getTimerCount()).toBe(0); + }); + it("waits for in-flight cleanup on shutdown without rearming its deadline", async () => { + const s = setup(); + let finish!: () => void; + s.service.sweep.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + const worker = s.start(); + await vi.advanceTimersByTimeAsync(1); + let stopped = false; + const stop = s.coordinator.stop().then(() => { stopped = true; }); + await Promise.resolve(); + expect(stopped).toBe(false); + finish(); + await stop; + await worker.ready; + expect(s.service.nextSweepAt).not.toHaveBeenCalled(); + expect(vi.getTimerCount()).toBe(0); + }); +}); diff --git a/server/src/app.ts b/server/src/app.ts index bce7324803c..c622f62e27c 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -9,6 +9,7 @@ import { customerSuccessRoutes } from "./routes/customer-success.js"; import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js"; import type { CloudWarmStandby } from "./services/cloud-warm-standby.js"; import { browserUseRoutes } from "./routes/browser-use.js"; +import { registerBrowserUseCleanup } from "./services/browser-use-work.js"; import { browserUseService } from "./services/browser-use.js"; import { slackToolRoutes } from "./routes/slack-tools.js"; import { createPublicMcpOAuth, publicMcpConfig } from "./services/public-mcp/oauth.js"; @@ -1290,12 +1291,7 @@ export async function createApp( ); }); }; - const browserUseTimer = setInterval(() => { - if (isWarmStandby() || isIdleTaskDrainActive()) return; - void browserUse.sweep().catch(() => logger.warn("Browser Use reconciliation failed; retrying.")); - }, 3000); - browserUseTimer.unref?.(); - if (!isWarmStandby() && !isIdleTaskDrainActive()) void browserUse.sweep().catch(() => logger.warn("Browser Use startup reconciliation failed; retrying.")); + registerBrowserUseCleanup(deliveryWork, browserUse, () => !isIdleTaskDrainActive()); let importTransferSweepTimer: ReturnType | null = setInterval( sweepImportTransferSpools, @@ -1408,7 +1404,6 @@ export async function createApp( chatPublicationTimer = null; } await chatReconciliation.drain(); - clearInterval(browserUseTimer); if (importTransferSweepTimer) { clearInterval(importTransferSweepTimer); importTransferSweepTimer = null; diff --git a/server/src/services/browser-use-work.ts b/server/src/services/browser-use-work.ts new file mode 100644 index 00000000000..ae8c7a36244 --- /dev/null +++ b/server/src/services/browser-use-work.ts @@ -0,0 +1,23 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; + +/** The app owns this worker; request-scoped services only signal durable writes. */ +export function registerBrowserUseCleanup( + coordinator: ReturnType, + service: { sweep(): Promise; nextSweepAt(): Promise }, + canRun: () => boolean, +) { + let next: number | null = null; + return coordinator.register(DELIVERY_QUEUES.browser, { + retryMs: 3000, + run: () => service.sweep(), + hasPending: async () => { + // During idle drain retain a memory-only wake, including startup or a + // newly committed first session. Warm standby is gated by the coordinator. + next = canRun() ? await service.nextSweepAt() : Date.now() + 3000; + return false; + }, + // Bound overdue batches without spinning on a busy database. + nextRunAt: () => next === null ? null : Math.max(Date.now() + 1000, next), + }); +} diff --git a/server/src/services/browser-use.ts b/server/src/services/browser-use.ts index 849b1d8bb08..683fe107ca8 100644 --- a/server/src/services/browser-use.ts +++ b/server/src/services/browser-use.ts @@ -43,6 +43,7 @@ import { createCostEventInTransaction } from "./costs.js"; import { createFinanceEventInTransaction } from "./finance.js"; import { withAccountingTransaction } from "./accounting-transaction.js"; import { logActivity } from "./activity-log.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { browserUseViewports } from "./browser-use-viewport.js"; import { forbidden, notFound, conflict } from "../errors.js"; @@ -64,6 +65,7 @@ const argsSchema = z }) .strict(); const terminalSessions = ["closed", "failed"]; +const liveSessions = ["starting", "running", "idle", "stopping"]; const requestMarker = (invocationId: string) => `\n\n[Paperclip request: ${invocationId}]`; export function browserUseService( @@ -422,6 +424,7 @@ export function browserUseService( ); const sid = args.sessionId; const s = await db.transaction(async (tx) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); let row: Session; if (sid) { await owned(binding, grant, sid); @@ -429,6 +432,7 @@ export function browserUseService( .update(sessions) .set({ status: "starting", + nextPollAt: new Date(), idleDeadline: null, updatedAt: new Date(), }) @@ -486,6 +490,7 @@ export function browserUseService( ...(maxCostUsd === undefined ? {} : { maxCostUsd }), }); await db.transaction(async (tx) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); await tx .update(runs) .set({ providerRunId: result.id, status: result.status }) @@ -507,6 +512,7 @@ export function browserUseService( const message = rejected ? error.message : "Browser run creation could not be confirmed. Paperclip is locating and stopping possible provider work. Do not start another run yet."; await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); const [current] = await tx.select().from(sessions).where(eq(sessions.id, s.id)).for("update"); // A restart must not leave a drained rejection attached to a starting // session: that combination has neither recovery work nor an idle timer. @@ -566,25 +572,28 @@ export function browserUseService( throw conflict("Only an idle browser can be kept open."); } else { if (terminalSessions.includes(s.status)) return; - await db - .update(sessions) - .set({ - stopRequested: action, - ...(action === "end" ? { status: "stopping" } : {}), - nextPollAt: new Date(), - updatedAt: new Date(), - }) - .where( - and( - eq(sessions.id, s.id), - action === "cancel" - ? or( - isNull(sessions.stopRequested), - ne(sessions.stopRequested, "end"), - ) - : undefined, - ), - ); + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); + await tx + .update(sessions) + .set({ + stopRequested: action, + ...(action === "end" ? { status: "stopping" } : {}), + nextPollAt: new Date(), + updatedAt: new Date(), + }) + .where( + and( + eq(sessions.id, s.id), + action === "cancel" + ? or( + isNull(sessions.stopRequested), + ne(sessions.stopRequested, "end"), + ) + : undefined, + ), + ); + }); } await signal( s, @@ -1218,97 +1227,109 @@ export function browserUseService( ); } } - let sweeping = false; - async function sweep(connectionId?: string) { - if (sweeping) return; - sweeping = true; - try { - const due = await db - .select() - .from(sessions) + // Wait for a concurrent sweep rather than reporting completion while its + // lease is still held. A filtered cleanup must still inspect its own scope. + let sweeping: Promise | null = null; + async function sweep(connectionId?: string): Promise { + if (sweeping) { + await sweeping; + return sweep(connectionId); + } + sweeping = reconcileDue(connectionId); + try { await sweeping; } finally { sweeping = null; } + } + async function reconcileDue(connectionId?: string) { + const due = await db + .select() + .from(sessions) + .where( + and( + connectionId ? eq(sessions.connectionId, connectionId) : undefined, + inArray(sessions.status, liveSessions), + lte(sessions.nextPollAt, new Date()), + or( + isNull(sessions.leaseUntil), + lte(sessions.leaseUntil, new Date()), + ), + ), + ) + .limit(30); + for (const s of due) { + const token = randomUUID(); + const lease = await db + .update(sessions) + .set({ + leaseToken: token, + leaseUntil: new Date(Date.now() + 180_000), + }) .where( and( - connectionId ? eq(sessions.connectionId, connectionId) : undefined, - inArray(sessions.status, [ - "starting", - "running", - "idle", - "stopping", - ]), - lte(sessions.nextPollAt, new Date()), + eq(sessions.id, s.id), or( isNull(sessions.leaseUntil), lte(sessions.leaseUntil, new Date()), ), ), ) - .limit(30); - for (const s of due) { - const token = randomUUID(); - const lease = await db + .returning(); + if (!lease.length) continue; + // Renew while paginating provider events or stopping multiple browsers. + const renewal = setInterval(() => { + void db + .update(sessions) + .set({ leaseUntil: new Date(Date.now() + 180_000) }) + .where(and(eq(sessions.id, s.id), eq(sessions.leaseToken, token))) + .catch(() => {}); + }, 30_000); + renewal.unref?.(); + let delay = s.status === "idle" ? 15000 : 3000; + try { + await reconcile(s); + } catch (e) { + delay = Math.max( + 10000, + e instanceof BrowserUseError ? e.retryAfterMs : 0, + ); + await db .update(sessions) .set({ - leaseToken: token, - leaseUntil: new Date(Date.now() + 180_000), + error: + s.stopRequested === "end" + ? "Browser shutdown is not confirmed. Paperclip will retry." + : "Browser synchronization failed. Paperclip will retry.", }) - .where( - and( - eq(sessions.id, s.id), - or( - isNull(sessions.leaseUntil), - lte(sessions.leaseUntil, new Date()), - ), - ), - ) - .returning(); - if (!lease.length) continue; - // Renew while paginating provider events or stopping multiple browsers. - const renewal = setInterval(() => { - void db - .update(sessions) - .set({ leaseUntil: new Date(Date.now() + 180_000) }) - .where(and(eq(sessions.id, s.id), eq(sessions.leaseToken, token))) - .catch(() => {}); - }, 30_000); - renewal.unref?.(); - let delay = s.status === "idle" ? 15000 : 3000; - try { - await reconcile(s); - } catch (e) { - delay = Math.max( - 10000, - e instanceof BrowserUseError ? e.retryAfterMs : 0, - ); - await db - .update(sessions) - .set({ - error: - s.stopRequested === "end" - ? "Browser shutdown is not confirmed. Paperclip will retry." - : "Browser synchronization failed. Paperclip will retry.", - }) - .where(eq(sessions.id, s.id)); - } finally { - clearInterval(renewal); - await db - .update(sessions) - .set({ - leaseToken: null, - leaseUntil: null, - nextPollAt: new Date( - Date.now() + delay + Math.floor(Math.random() * 500), - ), - }) - .where(and(eq(sessions.id, s.id), eq(sessions.leaseToken, token))); + .where(eq(sessions.id, s.id)); + } finally { + clearInterval(renewal); + const release = (tx: Pick) => tx.update(sessions).set({ + leaseToken: null, + leaseUntil: null, + nextPollAt: new Date(Date.now() + delay + Math.floor(Math.random() * 500)), + }).where(and(eq(sessions.id, s.id), eq(sessions.leaseToken, token))); + if (connectionId) { + // Credential-removal sweeps can race the app worker. Wake it when + // their lease ends so it does not wait for the old lease expiry. + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); + await release(tx); + }); + } else { + await release(db); } } - } finally { - sweeping = false; } } + async function nextSweepAt(): Promise { + const [row] = await db.select({ + // An overdue row owned by another sweep cannot run before its lease ends. + at: sql`min(greatest(${sessions.nextPollAt}, ${sessions.leaseUntil}))`, + }).from(sessions).where(inArray(sessions.status, liveSessions)); + return row?.at == null ? null : new Date(row.at).getTime(); + } return { execute, sweep, + nextSweepAt, list, viewer, resize, @@ -1322,16 +1343,19 @@ export function browserUseService( const scope = and( eq(sessions.companyId, companyId), eq(sessions.connectionId, connectionId), - inArray(sessions.status, ["starting", "running", "idle", "stopping"]), + inArray(sessions.status, liveSessions), ); - await db - .update(sessions) - .set({ - stopRequested: "end", - nextPollAt: new Date(), - updatedAt: new Date(), - }) - .where(scope); + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.browser); + await tx + .update(sessions) + .set({ + stopRequested: "end", + nextPollAt: new Date(), + updatedAt: new Date(), + }) + .where(scope); + }); await sweep(connectionId); const [pending] = await db .select({ id: sessions.id }) diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index d9696da5cf2..42bed21ae4d 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + browser: "browser-use", email: "email-channels", feedback: "feedback-exports", chatCompletion: "chat-completions", From d6e6d1c78a5f3777a3ccfd350b5f769b6fa4c9f5 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 10 Oct 2026 07:15:41 -0500 Subject: [PATCH 2/3] Stream OpenCode immediately, scope reasoning to turns, and deduplicate instructions (#15708) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip helps people manage AI agents and observe their work. > - The native runner carries provider output into the task transcript. > - OpenCode sends incremental text in `message.part.delta` events. > - The driver ignored those events and waited for a full part snapshot. Accepted chats also waited after lifecycle verification was complete. > - This pull request forwards each valid chunk as it arrives and removes unnecessary isolated startup work. > - Valid chunks reach Paperclip sooner without changing completion or permission rules. The matched browser experiment below does not establish a reliable end-to-end latency improvement. ## Linked Issues or Issue Description **What happened?** The native OpenCode driver withheld text until OpenCode sent a full snapshot. A local delayed-provider probe with the pinned OpenCode 1.18.34 runtime showed a 1,496 ms gap between the first native text delta and the first Paperclip text delta (1,510 ms from provider text). **Expected behavior** Forward valid assistant text and reasoning deltas immediately. Preserve repeated identical tokens, message roles, turn ownership, and final response selection. **Steps to reproduce** Send an empty text part, then text deltas, and withhold the final full snapshot. The new fixture requires the consumer to acknowledge its first text delta before releasing the snapshot. The previous driver cannot pass this handshake. **Paperclip version or commit** Baseline: `011e5669bad61c435a4745fb10812bef70376841`. The patch is rebased onto current master. **Deployment mode** Local native OpenCode 1.18.34 on macOS arm64. Live diagnostics use `openrouter/deepseek/deepseek-v4-flash-0731`. These are narrow diagnostics, not full Product E2E qualification. ## What Changed - Keep the macOS verified OpenCode launch pathname until child exit, with guarded reuse and cleanup on failed launches. This fixes a reproduced asynchronous executable-lifetime failure. - Dispatch accepted queued chats immediately after lifecycle readiness commits, through normal scheduler admission. Preserve ready state if dispatch fails. - Validate native OpenCode model access with the real hello probe without a redundant catalog scan. Keep legacy adapter discovery unchanged and isolate unspecified native probe directories. - Map incremental text and reasoning to the existing canonical event path, using observed part and message identities. - Keep additive chunks without IDs distinct. Preserve ID-based event deduplication. - Clear current streaming part metadata when a new turn starts. Reject mismatched session/message identities and unknown parts or fields. - Disable automatic updates, remote model refreshes, and default plugins in each isolated OpenCode process. Keep assigned MCP tools and default model reasoning intact. - Retain the latest cumulative text/reasoning part while its message role is unknown, including completion metadata after more than 100 chunks. - Expose `startTurn({ message, reasoningMode: "disabled" })` for one OpenCode/OpenRouter turn. Carry the selection in the durable command through Rust and the proxy. An omitted/default option restores provider defaults on the next turn, including warm and recovered sessions. There is no environment-level setting or automatic selection policy. - Group optional Rust turn inputs in `CodexTurnOptions` and dispatch through `start_turn_with_options`; retain `start_turn` for defaults. Future turn settings can be added without another feature-specific method. - Expose a native-session capability for per-turn reasoning and reject explicit unsupported selections before dispatch, including direct ACPX and Codex backends. Runnerd checks the durable selected model as well as the provider. - Remove duplicate agent instructions from the shared OpenCode proxy task envelope for all users. Send one system copy on every turn, including prepared and resumed turns, because OpenCode reads the latest user message’s system field. Preserve task constraints and completion contracts. - Document startup and streaming behavior. Add regression coverage for role ordering, repeated chunks, final snapshot deduplication, and stale turns. ## Verification ### Lower-load browser retry — 2026-10-10 **All 8/8 trials succeeded. In this batch, median Send → first visible reply fell from 30.586 s to 12.476 s (59.2% lower).** Four fresh-agent trials per variant, phase order before/after/after/before with two trials per phase. Every adjacent-phase pair favored the candidate, although one difference was only 0.301 s. This is evidence for the readiness/probe changes on this host, not a production percentile or a guarantee across providers. | Phase/trial | Variant | Send → reply (s) | Observed ready → run (s) | Session open → turn start (s) | Model requests | |---|---|---:|---:|---:|---:| | 0/0 | before | 45.042 | 25.337 | 3.629 | 2 | | 0/1 | before | 13.689 | 0.000 | 2.443 | 2 | | 1/0 | after | 14.696 | 0.000 | 2.526 | 1 | | 1/1 | after | 13.389 | 0.000 | 2.470 | 2 | | 2/0 | after | 10.871 | 0.000 | 2.706 | 2 | | 2/1 | after | 11.564 | 0.004 | 2.443 | 2 | | 3/0 | before | 41.582 | 21.518 | 2.376 | 2 | | 3/1 | before | 19.589 | 9.753 | 2.707 | 1 | Observed ready-to-run median: **15.636 s → 0 s**; zero means execution began by the first ready observation, not literally zero latency. Polling interval is nominally 250 ms plus HTTP time. Session-open-to-turn-start includes controller handling, not just child startup. All eight ran within the existing startup timeouts; no timeout or reasoning policy was changed. Host load at fixture creation stayed **5.47–8.73 on 18 CPUs**, versus 68.9–100.8 in the prior campaign. No builds/tests ran concurrently with measurement. Same merged source **b0a0158729a7913882a0e756fe7e27228d72bf53**, release runner, rebuilt static UI, OpenCode **1.18.34**, model **openrouter/deepseek/deepseek-v4-flash-0731**, provider-default reasoning, and empty isolated workspaces in both variants. Only the readiness worker and explicit-model verification source were toggled; the launch-lifetime fix and all proxy streaming changes were held constant. Thus this does not separately measure the original streaming changes. One browser per phase, two fresh companies/agents/chats per phase. Measurement begins at capture-phase Send click and ends on visible Hi text plus two animation frames. Status/thinking is excluded; emoji can arrive just after the first text. Successful screenshots inspected. Each variant had three two-request turns using `paperclip_finish` and one one-request turn; no status lookup was observed. Reported usage includes settlement after the first visible reply. Runner SHA-256 `57f658c220eecd2370ffd3333cd7e5d06d16bb2798af0cf43c84e01c8207c1c3`; proxy `bcc02fda2bf35b38a116cd880841aa6c534f6e0b497a96d97369a1e06dcc72ac`; rebuilt UI tree `eee238f2eda628c4e8c3660707986da4f8dd9100c814e26938fce3756df2eae4` (sorted relative path, NUL, binary per-file SHA-256). Worker/probe variant source hashes are unchanged from the preserved comparison below. Measured turn cost **$0.00612574**, plus verification probes not included in that figure. All eight attempts retained; all source/proxy files restored and test supervisors exited. Current commit validation: **53 successful CI checks, 4 skipped, zero failing; Greptile 5/5 on b0a0158729 with no actionable findings.** The previous 20 targeted post-merge tests passed. No source edit was made for this rerun. Earlier failed and inconclusive cohorts below remain historical evidence and are not pooled with this batch. ### Master update and failure diagnosis — 2026-10-09 Merged `origin/master` at `417db14e7d` into this branch without conflicts; pushed merge commit `b0a0158729a7913882a0e756fe7e27228d72bf53`. The incoming changes do not alter OpenCode process startup or its timeout budgets. Historical measurements below remain attributed to their original commits. The four last-campaign failures never issued `turn.start`, so these were **cold-session admission failures, not model replies failing**. Two retained `session.open` commands eventually completed after controller timeout/cleanup had begun. The other two ended with the inner `thread/start` response timeout, and their diagnostics recorded OpenCode startup retries. Startup event timestamps are buffered publication times and must not be interpreted as exact process-spawn times. On the merged source, two credential-free startup-only checks used the real pinned OpenCode 1.18.34, empty temporary workspaces, a placeholder key, and **no model turns**: - Existing pinned executable: spawn at 1.083 s; failed health attempt/retry at 11.311 s and 21.914 s; third process healthy at 32.210 s; session opened at **35.168 s**. - Verified private executable path: snapshot preparation took **0.497 s**; process spawned at 1.588 s; healthy at 9.967 s; session opened at **11.120 s**. These are sequential diagnostic samples, not evidence that one executable path is faster. Host load was **56.5–64.5 on 18 logical CPUs**. The local health wait permits 10 s per attempt and the driver retries up to three times; the outer controller's cold admission budget is 30 s, and the Rust facade also waits 30 s for a response. Thus retries plus setup/session creation can outlast the caller's budget. The isolated 35.168 s success demonstrates that a session can eventually open yet fail the app's admission deadline. Heavy load aggravates the timing; its exact contribution is not isolated. No timeout was increased and no new model-latency claim is made. Merge validation: **20/20 targeted lifecycle-policy and heartbeat run-completion tests passed**, including temporary PostgreSQL cases. The first sandbox invocation skipped database cases; the authorized local-DB rerun executed all 20. The package-manager launcher could not verify its version via the unreachable registry, so validation used the already-installed Vitest binary. Full CI/review subsequently passed on this merge commit; see the October 10 retry above. Older results below apply to their named commits. ### Follow-up: reproduced macOS launch failure — 2026-10-09 The retry exposed a startup bug rather than only host-load noise. The macOS proxy deleted the private verified executable path immediately after asynchronous spawn. A credential-free test of the actual pinned OpenCode `--version` reproduced a fast SIGKILL and a 15-second timeout in three immediate-cleanup attempts; three deferred-cleanup controls succeeded. The timeout was killed by the diagnostic's own deadline. This experiment does not establish the OS-internal reason for the fast SIGKILL. Commit `c30451e0ebf3f74e9ef70d37bbe4d9fe7fb56359` retains the private launch path until child exit, cleans failed spawns, and waits for forced shutdown before reuse. Source descriptor authentication, ownership, mode, inode checks, and the prohibition on ambient executable fallback remain. Overlapping reuse fails closed. The existing synchronous shell test was changed to exercise the actual asynchronous boundary. **Verification:** 59 focused proxy/driver tests passed; runner TypeScript check passed. All six subsequent real OpenCode version launches succeeded without timeout (4.11, 4.33, 5.73, 6.91, 4.95, 6.70 seconds). The native Rust binary is unchanged from the retry; no model calls were made by these diagnostics. Retained interrupted retry: before 42.56/18.69 seconds; after one 180-second timeout (`provider_process_exited` / health SIGKILL) and one 37.33-second success. The latter overlapped the credential-free diagnostic and is not a clean matched timing. The next phase was stopped before Send to diagnose the reproduced bug. Reported measured-turn cost: $0.0016874, verification probes additional. These attempts are not pooled with other campaigns. **Completed retry, stopped by the declared failure rule after 6 of 8 planned trials. No reliable end-to-end speedup demonstrated.** At commit `c30451e0ebf3f74e9ef70d37bbe4d9fe7fb56359`, the launch fix above was held constant in both variants; only the readiness handoff and explicit-model verification path changed. Planned phase order: before/after/after/before, two fresh agents/chats per phase, one new browser per phase. The second after phase had two consecutive failures, so the final before phase was not run. | Phase/trial | Variant | Send → first visible reply | Observed ready → run | Outcome | |---|---|---|---|---| | 0/0 | before | No reply within 180 s | 2.777 s | session.open timeout | | 0/1 | before | 47.654 s | 9.383 s | Succeeded | | 1/0 | after | 50.051 s | 0.087 s | Succeeded | | 1/1 | after | No reply within 180 s | 0.151 s | session.open timeout | | 2/0 | after | No reply within 180 s | 3.337 s | session.open timeout | | 2/1 | after | No reply within 180 s | 1.434 s | session.open timeout | Both successful screenshots show “Hi 👋”. Timing starts at the browser's capture-phase Send click and ends at the first visible answer text plus two animation frames; the first fragment can be “Hi” before the emoji arrives. Thinking/status text is not counted. All four failures were `runner_local_connect_failed: PRP command session.open timed out` (`native_session_interrupted`), before a reply; none reported the earlier health-stage SIGKILL. They are retained, not discarded as outliers. One success per variant does not support a speedup estimate or median comparison. Observed ready-to-run median was **6.080 s before (n=2) vs 0.793 s after (n=4)**; this is sampled every nominal 250 ms plus HTTP latency and is a lower bound, not precise tracing. Host load at fixture creation ranged **68.9–100.8 on 18 logical CPUs**, a substantial confound that worsened over the campaign. No local build/test was run concurrently with this campaign. Provenance: pinned OpenCode **1.18.34**, model **openrouter/deepseek/deepseek-v4-flash-0731**, default reasoning in both variants, empty synthetic workspaces and isolated probe homes. Release runner SHA-256 `57f658c220eecd2370ffd3333cd7e5d06d16bb2798af0cf43c84e01c8207c1c3`; current proxy SHA-256 `bcc02fda2bf35b38a116cd880841aa6c534f6e0b497a96d97369a1e06dcc72ac`; built UI tree SHA-256 `7f0c2b7f586cb753408351574b2239e22311944add99af4f6f3562cd6d47bbcb`. This holds the rebuilt runner and static UI constant. Provider-reported measured turn cost **$0.00120649**; verification probes are additional. Including the earlier interrupted retry, this follow-up reports **$0.00289389** in measured turn usage; missing usage is not assumed free. Prior cohorts remain separate below. Validation on `c30451e0e`: **59 focused tests passed**, runner TypeScript check passed, **6/6 real credential-free pinned OpenCode launches passed**, fresh Greptile **5/5 with zero unresolved threads**. CI: **51 success, 4 skipped, 1 failed**. The failed native compilation job never reached compilation: Docker's auth endpoint timed out during Buildx setup, then failed the same way on one retry ([job](https://github.com/paperclipai/paperclip/actions/runs/37994181674/job/114039737503)). Full local checks were not rerun while measuring. The PR remains draft: the launch defect is reproduced and fixed, but cold session startup still exceeds the controller's admission budget on this host. Raising that budget would not establish a faster first response. ### Readiness investigation and live rerun — 2026-10-09 **Measured queue fix; no reliable end-to-end speedup claim. Keep this PR in draft.** Accepted chats could wait for the periodic queue sweep after verification had already committed `ready`. The lifecycle worker now dispatches through the existing scheduler immediately after that commit. The normal company, pause, budget, concurrency, and task-ownership checks still apply. Dispatch failure preserves readiness and leaves recovery to the queue sweep. Native OpenCode verification also avoids enumerating the full model catalog before running its real hello request. Identifier validation remains. Authentication failures, unavailable models, and timeouts still prevent readiness. An unspecified local probe directory uses an empty temporary directory. **Queue-only experiment:** four before and four after attempts, in before/after/after/before server phases, two fresh agents per phase. Median observed ready-to-start delay fell from **19.429 s to 0.058 s**. State polling was nominally 250 ms plus HTTP time, so the latter means within observation resolution; it is not a precise 58 ms latency claim. After timings were 100.20, 65.29, 63.78, and 76.97 seconds. Before had one 59.65-second success and three 120-second response timeouts (one provider health SIGKILL, one late success, one still running after settlement and canceled). These censored results do not support a success-only median speedup comparison. Focused tests overlapped the last baseline phase, which further limits total-time interpretation. Reported measured-turn cost: $0.0055383; verification probes and incomplete/canceled usage are additional. **Final matched experiment:** built static UI in both variants to avoid pre-Send Vite compilation; current proxy, release runner, selected model, default reasoning, and explicitly isolated empty probe cwd/HOME/XDG held constant. Both worker handoff and model-probe changes vary. Fresh browser/company/agent/chat per attempt. ABBA order, two attempts per variant, 180-second response limit declared before the batch, four-turn bound. Same Send-to-rendered-Hi metric as the earlier experiment. | Order | Variant | Send → visible Hi | Created → execution | Observed ready → execution | Outcome | | --- | --- | ---: | ---: | ---: | --- | | 1 | Before | 62.66 s | 23.467 s | 2.830 s | Succeeded | | 2 | After | >180 s | 14.200 s | Within polling resolution | Provider health SIGKILL | | 3 | After | >180 s | 13.877 s | Within polling resolution | Provider initialization timeout | | 4 | Before | >180 s | 144.256 s | 14.067 s | Still running after settlement; canceled | Only one of four attempts produced a timed reply. **There is no valid overall speedup estimate from this batch.** The dispatch delay is removed, but provider startup failures and heavy shared-host load remain. A recorded host sample was load 93.55 on 18 logical CPUs. This is a confound, not proof that every failure was caused by host load. Provider-reported measured-turn cost was $0.00032752, with no reported usage for the other three attempts; verification probes are additional. Retained earlier setup attempts: one shared-memory startup failure before model calls, and a separate development-UI batch with a 60.06-second baseline followed by a navigation timeout before Send. These attempts are retained separately, not counted as successful response timings. No live attempts were silently dropped. Provenance: source base `53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f`, measured patch committed as `40b54eafb`, then rebased as `c590770dcc2bb1debce14037935a7031e5c3c1ac`. **These live timings predate the rebase.** Source and binary hashes are retained below. The inner harness label `after` selects the identical current proxy in every attempt; the outer campaign handoff label is the actual before/after variant. Owned benchmark processes stopped and original proxy restored. Post-rebase validation at `c590770dcc2bb1debce14037935a7031e5c3c1ac`: - Full local `pnpm -r typecheck`: **passed**, including the native release build (1,098.6 seconds on the loaded host). - Focused lifecycle and adapter tests: **26 passed**. The 14 database integration tests did not run locally because their hard-coded 20-second database setup hook timed out. A retry with a CLI timeout override encountered the same explicit hook limit. The earlier pre-rebase focused batch passed 77 tests; these results are not substituted for post-rebase execution. - Local `pnpm test:run` was started, then stopped to avoid duplicating CI on the saturated host. Local `pnpm build` was not run separately. **Current-head CI passed Typecheck + Release Registry, Build, all general and serialized test groups, Rust checks, both runner Vitest groups, and all eight browser E2E shards.** The last general shard passed 1,179 tests (72 files). - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/15708#issuecomment-6085742217): **5/5**, reviewed this exact head, no actionable defect. - Final CI rollup: **49 success, 4 skipped, 4 failure**. The failures are [Compile isolated native Runner](https://github.com/paperclipai/paperclip/actions/runs/37990274515/job/114024329242), [Docker context integrity](https://github.com/paperclipai/paperclip/actions/runs/37990274623/job/114024196511), [Canary Dry Run](https://github.com/paperclipai/paperclip/actions/runs/37990274623/job/114024196424), and the dependent `ci / verify` aggregate. All three root failures are Docker Hub HTTP 429 / unauthenticated pull-rate limits while fetching Node base images. The native-image job was retried once and hit the same limit. No source compiler/test failure was reported by those gates. CI maintainers need authenticated pulls or to rerun after the registry limit clears. - Owned benchmark servers stopped. Local validation commands completed or were stopped. The PR remains a draft: user-visible speed/reliability is not qualified, and the Docker-dependent CI gates are not green.
Sanitized readiness evidence with all attempts and source hashes ```json {"campaign":{"order":["before","after","after","before"],"trialsPerPhase":1,"maxMeasuredTurns":4,"maxCampaignUsd":1,"sourceHead":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","reasoning":"provider default, no override","proxySha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","purpose":"Final matched comparison of both optimizations; distinct from earlier queue-only campaign.","responseTimeoutMs":180000,"timeoutRationale":"Declared before this final campaign because earlier unchanged baselines sometimes exceeded 120 seconds; earlier timeouts remain retained.","ui":{"mode":"built static UI","treeSha256":"7f0c2b7f586cb753408351574b2239e22311944add99af4f6f3562cd6d47bbcb"},"priorCampaign":"combined-proof stopped after a development-UI navigation timeout before Send; no failed attempts erased."},"complete":true,"statistics":{"before":{"attempted":2,"successfulMeasurements":1,"medianTtfrMs":62657.2000002861,"rangeTtfrMs":[62657.2000002861,62657.2000002861],"medianObservedReadyToRunMs":8448.5,"medianRunStartToVisibleMs":37290.199951171875},"after":{"attempted":2,"successfulMeasurements":0,"medianTtfrMs":null,"rangeTtfrMs":null,"medianObservedReadyToRunMs":0.0,"medianRunStartToVisibleMs":null}},"measuredTurnCostUsd":0.00032752,"costCoverage":"Provider-reported measured chat turns only; harness verification probes are additional and not included.","readinessMetricLimit":"First observed ready state sampled every 250 ms plus HTTP duration; clamped to zero if execution started before the observation. This is a lower bound on ready-to-start delay, not a precise sub-millisecond metric.","results":[{"phase":0,"trial":0,"handoff":"before","ttfrMs":62657.2000002861,"answer":"Hi","statuses":["succeeded"],"error":null,"runErrorCode":null,"providerHealthKilled":false,"queueMs":23467.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":26390.0,"observedReadyToRunMs":2830.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":37290.199951171875,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00032752,"costUsdExact":"0.000327520","inputTokens":26864,"outputTokens":46,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":1,"trial":0,"handoff":"after","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":"provider_process_exited","providerHealthKilled":true,"queueMs":14200.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":24028.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":0,"handoff":"after","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":"provider_initialize_timeout","providerHealthKilled":false,"queueMs":13877.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":18107.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":3,"trial":0,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["running"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":null,"providerHealthKilled":false,"queueMs":144256.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":136901.0,"observedReadyToRunMs":14067.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"}],"hostLoadSample":{"at":"2026-10-09T20:48:40.031Z","parallelism":18,"load":[93.546875,82.68017578125,75.9091796875]},"measuredChangesCommit":"40b54eafb","postRebaseCommit":"c590770dcc2bb1debce14037935a7031e5c3c1ac","postRebaseLimit":"Live benchmark predates the rebase. No post-rebase live result is claimed.","variantSourceHashes":{"before":{"server/src/modules/agent-lifecycle/application/worker.ts":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326","packages/adapters/opencode-local/src/server/test.ts":"4c10bacb55f2c7589e7b5c79d74781206ae075db8955f5b998065d6db3c0335a","server/src/adapters/registry.ts":"4a858763f69947d5745c7b8cb251a62fa404bc9683ade0ffb76282ed12761394"},"after":{"server/src/modules/agent-lifecycle/application/worker.ts":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7","packages/adapters/opencode-local/src/server/test.ts":"cd2ea3d51c3f1b4fb42261be701fb7f6b85704ad8e641e6db73d7c58e47364d3","server/src/adapters/registry.ts":"4a858763f69947d5745c7b8cb251a62fa404bc9683ade0ffb76282ed12761394"}},"model":"openrouter/deepseek/deepseek-v4-flash-0731","prompt":"Reply with Hi and an emoji and nothing else.","runnerBinary":{"profile":"release","sha256":"6134c17debda14a76da69ac9631a2d10dad482ebd58dbc9a4e4480ed652211c4"},"probeIsolation":"Explicit empty cwd and HOME/XDG folders under the synthetic fixture workspace in BOTH variants. No server checkout or ambient provider home is used for the hello request.","comparisonLabelNote":"Inner run.mts variant after selects the identical current proxy in every trial; outer campaign readinessHandoff controls before/after source changes."} ``` Queue-only campaign: ```json {"campaign":{"order":["before","after","after","before"],"trialsPerPhase":2,"maxMeasuredTurns":8,"maxCampaignUsd":1,"sourceHead":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","reasoning":"provider default, no override","proxySha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04"},"complete":true,"statistics":{"before":{"attempted":4,"successfulMeasurements":1,"medianTtfrMs":59654.5,"rangeTtfrMs":[59654.5,59654.5],"medianObservedReadyToRunMs":19429.0,"medianRunStartToVisibleMs":8843.5},"after":{"attempted":4,"successfulMeasurements":4,"medianTtfrMs":71128.7000002861,"rangeTtfrMs":[63778.90000009537,100195.59999990463],"medianObservedReadyToRunMs":58.0,"medianRunStartToVisibleMs":26542.849975585938}},"measuredTurnCostUsd":0.0055383,"costCoverage":"Provider-reported measured chat turns only; harness verification probes are additional and not included.","readinessMetricLimit":"First observed ready state sampled every 250 ms plus HTTP duration; clamped to zero if execution started before the observation. This is a lower bound on ready-to-start delay, not a precise sub-millisecond metric.","results":[{"phase":0,"trial":0,"handoff":"before","ttfrMs":59654.5,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":50590.0,"observedReadyToRunMs":25709.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":8843.5,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00087852,"costUsdExact":"0.000878520","inputTokens":25644,"outputTokens":264,"cachedInputTokens":28416,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":0,"trial":1,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":41651.0,"observedReadyToRunMs":14656.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":1,"trial":0,"handoff":"after","ttfrMs":100195.59999990463,"answer":"Hi 🙂","statuses":["succeeded"],"error":null,"queueMs":65893.0,"observedReadyToRunMs":116.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":33243.60009765625,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00075782,"costUsdExact":"0.000757820","inputTokens":27014,"outputTokens":175,"cachedInputTokens":26368,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":1,"trial":1,"handoff":"after","ttfrMs":65288.60000038147,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":43511.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":21227.60009765625,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00085326,"costUsdExact":"0.000853260","inputTokens":25166,"outputTokens":250,"cachedInputTokens":28160,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":0,"handoff":"after","ttfrMs":63778.90000009537,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":39218.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":23479.89990234375,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000839,"costUsdExact":"0.000839000","inputTokens":27068,"outputTokens":238,"cachedInputTokens":26368,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":1,"handoff":"after","ttfrMs":76968.80000019073,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":46257.0,"observedReadyToRunMs":185.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":29605.800048828125,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030582,"costUsdExact":"0.000305820","inputTokens":24694,"outputTokens":32,"cachedInputTokens":1792,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":3,"trial":0,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["succeeded"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":100979.0,"observedReadyToRunMs":3872.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00190388,"costUsdExact":"0.001903880","inputTokens":25524,"outputTokens":852,"cachedInputTokens":55808,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":3,"trial":1,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["running"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":129391.0,"observedReadyToRunMs":24202.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"}]} ```
### Matched live browser latency evidence — 2026-10-09 **Conclusion: the overall first-response speedup is not established.** Twelve successful real browser runs produced a lower after median, but the updated path won only 3 of 6 adjacent pairs. Median paired improvement was -0.17 seconds (effectively tied). Do not treat the deterministic streaming-forwarding improvement as proof that this PR fixes end-to-end first-response latency. - **Actual path:** local Paperclip browser → real server/database → current optimized Rust runner → pinned OpenCode 1.18.34 → OpenRouter `deepseek/deepseek-v4-flash-0731`. Default reasoning in both variants; no disabled-reasoning override. - **Prompt:** `Reply with Hi and an emoji and nothing else.` Fresh company, agent, chat, and per-turn native provider process for each trial; same QA fixture instructions and selected model. - **Comparison:** current product/runner at `53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f` held constant. The baseline substitutes the exact `opencode-app-server-proxy.ts`, `opencode-proxy-task-envelope.ts`, and `opencode-server-driver.ts` contents from pre-PR `41fec8c8921f9fb19c13fe11e2c9b71bf1017317`. This isolates those changed modules; it is not an entire old-checkout comparison. - **Metric:** browser capture-phase Send click to a rendered `Hi` markdown answer plus two animation frames, with a screenshot of the answer. Thinking, tool labels, and status indicators do not count. This is a DOM/render measurement, not instrumented compositor pixel timing. - **Order/bound:** predeclared `before, after, after, before` repeated three times; six trials per variant; 120-second response timeout; $1 campaign guard. All 12 measured trials returned `Hi` and an emoji and finished successfully; none omitted. - **Result:** before median **48.08 s** (37.42–66.40); after median **42.50 s** (27.68–63.32). The 11.6% lower median alone is not persuasive evidence of a reliable improvement given the mixed paired results and broad overlap. - **Remaining delay:** retained queue timestamps for 10/12 trials show **9.76–41.65 seconds from run creation to execution start**. This is before the OpenCode path changed here. Its cause is not established by this benchmark. The final two queue reads were unavailable after automatic server shutdown; all twelve primary browser timings were retained. - **Cost:** $0.010517369 provider-reported across the 12 measured turns; $0.011371499 including the successful setup pilot. This is separate from the older diagnostics listed below. - **Retained setup attempts:** two failures before Send (composer locator and transpiler helper); one baseline startup failure before any observed model request (`SIGKILL` while probing OpenCode health); one successful current-code pilot whose selector missed the visible reply. These are retained separately, not silently included/excluded as valid timings. The measured batch used the corrected selector and the same release binary for both variants. - **Limits:** local development server and headless Chromium on macOS arm64, small sample, variable queue timing/provider routing/cache. This does not qualify production/cloud performance or isolate the contributions of each changed module. Server, browser, and owned runs were stopped; original proxy build artifact restored; no product source files changed for this experiment. | Pair | Before: Send → visible Hi | After: Send → visible Hi | Faster variant | | --- | ---: | ---: | --- | | 1 | 66.40 s | 27.68 s | After | | 2 | 37.42 s | 28.00 s | After | | 3 | 39.75 s | 55.22 s | Before | | 4 | 47.32 s | 63.32 s | Before | | 5 | 48.85 s | 29.79 s | After | | 6 | 51.31 s | 61.07 s | Before |
Sanitized measurement records, source/binary hashes, usage, and retained attempt classification ```json {"manifest":{"candidate":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","baseline":"41fec8c8921f9fb19c13fe11e2c9b71bf1017317","comparison":"Same current product and runner; replace the three OpenCode proxy/driver source modules with exact pre-PR contents for baseline. Reasoning uses defaults in both.","model":"openrouter/deepseek/deepseek-v4-flash-0731","prompt":"Reply with Hi and an emoji and nothing else.","samplesPerVariant":6,"order":["before","after","after","before","before","after","after","before","before","after","after","before"],"maxTurnMs":120000,"maxCostUsd":1,"variants":{"before":{"sources":{"src/cli/opencode-app-server-proxy.ts":"e9b46c937d7ce3b51dc44181a2a3ad82befda9ebe184adf80c0a6b30da140fd6","src/cli/opencode-proxy-task-envelope.ts":"fc23f8a515c2c4d1551f6b95e5b38e6254849dec23f7396e76b646e170a6e006","src/drivers/opencode/opencode-server-driver.ts":"99de415e8fc3bb5fb6f8d7512d751c4c37d2aa77a18b0546b9fb232f1a6fb6c5"},"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b"},"after":{"sources":{"src/cli/opencode-app-server-proxy.ts":"fa800e9bccce02f05f1af83e225d3b5a1300d0a7d4c35ac548f400d6c0a83f79","src/cli/opencode-proxy-task-envelope.ts":"5c387614fc9832322686326b8482eb00d88699d491b05cf59cadcd94a41194ea","src/drivers/opencode/opencode-server-driver.ts":"a944f3bf7f1bf897daec3a5dc2aa34ee9d1a2133efcb3cf150b002095a0ec586"},"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04"}},"runnerBinary":{"profile":"release","sha256":"6134c17debda14a76da69ac9631a2d10dad482ebd58dbc9a4e4480ed652211c4"},"metric":"Browser Send capture-phase click to visible Hi answer, sampled on animation frames, plus two paint opportunities. Excludes reasoning/tool/status text.","setupAttempts":["setup-failure-1","setup-failure-2","startup-failure-3","pilot-selector-failure"]},"summary":{"before":{"attempts":6,"measured":6,"medianMs":48083.050000190735,"minMs":37420.5,"maxMs":66397.69999980927,"totalCostUsd":0.004945229,"toolCalls":[4,1,0,0,0,1]},"after":{"attempts":6,"measured":6,"medianMs":42504.049999952316,"minMs":27680.099999904633,"maxMs":63320.90000009537,"totalCostUsd":0.00557214,"toolCalls":[0,1,0,4,0,1]},"medianReductionPercent":11.602841334350234,"medianPairedImprovementMs":-168.59999990463257,"reportedCampaignCostUsdIncludingPilot":0.011371498999999998},"pairs":[{"indices":[0,1],"beforeMs":66397.69999980927,"afterMs":27680.099999904633,"improvementMs":38717.59999990463},{"indices":[2,3],"beforeMs":37420.5,"afterMs":27997,"improvementMs":9423.5},{"indices":[4,5],"beforeMs":39751.60000038147,"afterMs":55217.199999809265,"improvementMs":-15465.599999427795},{"indices":[6,7],"beforeMs":47318.90000009537,"afterMs":63320.90000009537,"improvementMs":-16002.0},{"indices":[8,9],"beforeMs":48847.2000002861,"afterMs":29790.900000095367,"improvementMs":19056.300000190735},{"indices":[10,11],"beforeMs":51311.7000002861,"afterMs":61072.40000009537,"improvementMs":-9760.699999809265}],"trials":[{"index":0,"variant":"before","startedAt":"2026-10-09T19:15:23.172Z","measurementError":null,"ttfrMs":66397.69999980927,"domMs":66390.59999990463,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.002612595,"costUsdExact":"0.002612595","inputTokens":55962,"outputTokens":1344,"cachedInputTokens":83456,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_get_task_context","paperclip_comment_on_task","paperclip_finish","paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":4,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:15:58.194Z","turn.submitted":"2026-10-09T19:15:58.311Z","turn.started":"2026-10-09T19:15:59.063Z","firstTextDelta":"2026-10-09T19:16:36.039Z","turn.completed":"2026-10-09T19:16:36.118Z"},"completedRunDurationsMs":[42915.0]},{"index":1,"variant":"after","startedAt":"2026-10-09T19:16:36.598Z","measurementError":null,"ttfrMs":27680.099999904633,"domMs":27636.599999904633,"answer":"Hi","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030429,"costUsdExact":"0.000304290","inputTokens":26845,"outputTokens":28,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":3,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:17:00.717Z","turn.submitted":"2026-10-09T19:17:00.841Z","turn.started":"2026-10-09T19:17:01.508Z","firstTextDelta":"2026-10-09T19:17:05.934Z","turn.completed":"2026-10-09T19:17:06.205Z"},"completedRunDurationsMs":[14435.0]},{"index":2,"variant":"after","startedAt":"2026-10-09T19:17:08.762Z","measurementError":null,"ttfrMs":27997,"domMs":27984.599999904633,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00101016,"costUsdExact":"0.001010160","inputTokens":27544,"outputTokens":366,"cachedInputTokens":26624,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:17:26.192Z","turn.submitted":"2026-10-09T19:17:26.305Z","turn.started":"2026-10-09T19:17:26.936Z","firstTextDelta":"2026-10-09T19:17:40.688Z","turn.completed":"2026-10-09T19:17:40.826Z"},"completedRunDurationsMs":[17763.0]},{"index":3,"variant":"before","startedAt":"2026-10-09T19:17:41.376Z","measurementError":null,"ttfrMs":37420.5,"domMs":37364.40000009537,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000693018,"costUsdExact":"0.000693018","inputTokens":24236,"outputTokens":282,"cachedInputTokens":27648,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:18:12.381Z","turn.submitted":"2026-10-09T19:18:12.616Z","turn.started":"2026-10-09T19:18:13.441Z","firstTextDelta":"2026-10-09T19:18:21.746Z","turn.completed":"2026-10-09T19:18:22.262Z"},"completedRunDurationsMs":[29216.0]},{"index":4,"variant":"before","startedAt":"2026-10-09T19:18:23.000Z","measurementError":null,"ttfrMs":39751.60000038147,"domMs":39705.7000002861,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000265747,"costUsdExact":"0.000265747","inputTokens":26123,"outputTokens":77,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:19:01.000Z","turn.submitted":"2026-10-09T19:19:01.163Z","turn.started":"2026-10-09T19:19:01.893Z","firstTextDelta":"2026-10-09T19:19:07.169Z","turn.completed":"2026-10-09T19:19:07.384Z"},"completedRunDurationsMs":[15843.0]},{"index":5,"variant":"after","startedAt":"2026-10-09T19:19:10.050Z","measurementError":null,"ttfrMs":55217.199999809265,"domMs":55204.89999961853,"answer":"Hi 🙂","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030421,"costUsdExact":"0.000304210","inputTokens":4294,"outputTokens":28,"cachedInputTokens":22543,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:19:59.877Z","turn.submitted":"2026-10-09T19:19:59.990Z","turn.started":"2026-10-09T19:20:00.713Z","firstTextDelta":"2026-10-09T19:20:07.290Z","turn.completed":"2026-10-09T19:20:07.658Z"},"completedRunDurationsMs":[15104.0]},{"index":6,"variant":"after","startedAt":"2026-10-09T19:20:09.550Z","measurementError":null,"ttfrMs":63320.90000009537,"domMs":63313.09999990463,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00260723,"costUsdExact":"0.002607230","inputTokens":137971,"outputTokens":951,"cachedInputTokens":1024,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_comment_on_task","paperclip_get_task_context","paperclip_comment_on_task","paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":4,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:20:59.730Z","turn.submitted":"2026-10-09T19:20:59.859Z","turn.started":"2026-10-09T19:21:00.501Z","firstTextDelta":"2026-10-09T19:21:19.416Z","turn.completed":"2026-10-09T19:21:19.608Z"},"completedRunDurationsMs":[26519.0]},{"index":7,"variant":"before","startedAt":"2026-10-09T19:21:20.519Z","measurementError":null,"ttfrMs":47318.90000009537,"domMs":47246.800000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000257965,"costUsdExact":"0.000257965","inputTokens":4700,"outputTokens":71,"cachedInputTokens":21407,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:22:03.924Z","turn.submitted":"2026-10-09T19:22:04.439Z","turn.started":"2026-10-09T19:22:04.772Z","firstTextDelta":"2026-10-09T19:22:09.915Z","turn.completed":"2026-10-09T19:22:10.343Z"},"completedRunDurationsMs":[16702.0]},{"index":8,"variant":"before","startedAt":"2026-10-09T19:22:11.124Z","measurementError":null,"ttfrMs":48847.2000002861,"domMs":48802.59999990463,"answer":"Hi 🎉","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000327654,"costUsdExact":"0.000327654","inputTokens":44796,"outputTokens":32,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:22:57.182Z","turn.submitted":"2026-10-09T19:22:57.281Z","turn.started":"2026-10-09T19:22:57.988Z","firstTextDelta":"2026-10-09T19:23:02.915Z","turn.completed":"2026-10-09T19:23:03.035Z"},"completedRunDurationsMs":[9541.0]},{"index":9,"variant":"after","startedAt":"2026-10-09T19:23:03.579Z","measurementError":null,"ttfrMs":29790.900000095367,"domMs":29748.800000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00034054,"costUsdExact":"0.000340540","inputTokens":24710,"outputTokens":59,"cachedInputTokens":1792,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":2,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:23:28.840Z","turn.submitted":"2026-10-09T19:23:28.954Z","turn.started":"2026-10-09T19:23:29.666Z","firstTextDelta":"2026-10-09T19:23:35.306Z","turn.completed":"2026-10-09T19:23:35.531Z"},"completedRunDurationsMs":[12732.0]},{"index":10,"variant":"after","startedAt":"2026-10-09T19:23:37.247Z","measurementError":null,"ttfrMs":61072.40000009537,"domMs":61061.300000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00100571,"costUsdExact":"0.001005710","inputTokens":25307,"outputTokens":368,"cachedInputTokens":28160,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:24:33.721Z","turn.submitted":"2026-10-09T19:24:33.832Z","turn.started":"2026-10-09T19:24:34.601Z","firstTextDelta":"2026-10-09T19:24:43.135Z","turn.completed":"2026-10-09T19:24:43.278Z"},"completedRunDurationsMs":[17387.0]},{"index":11,"variant":"before","startedAt":"2026-10-09T19:24:44.710Z","measurementError":null,"ttfrMs":51311.7000002861,"domMs":51300.5,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00078825,"costUsdExact":"0.000788250","inputTokens":24316,"outputTokens":356,"cachedInputTokens":27648,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:25:31.347Z","turn.submitted":"2026-10-09T19:25:31.463Z","turn.started":"2026-10-09T19:25:32.208Z","firstTextDelta":"2026-10-09T19:25:39.405Z","turn.completed":"2026-10-09T19:25:39.563Z"},"completedRunDurationsMs":[15446.0]}],"additionalAttempts":[{"attempt":"setup-1","stage":"before_send","result":"non-editable composer locator","modelRequestObserved":false},{"attempt":"setup-2","stage":"before_send","result":"transpiler helper absent in browser","modelRequestObserved":false},{"attempt":"startup-pilot-before","runnerProfile":"debug","result":"no visible reply within 120s; OpenCode health startup SIGKILL; retained infrastructure failure","modelRequestObserved":false},{"attempt":"selector-pilot-after","runnerProfile":"debug","result":"Hi reply succeeded; browser selector missed it; no valid TTFR","providerReportedCostUsd":0.00085413}],"queueTimingCoverage":{"measuredTrials":10,"totalTrials":12,"records":[{"index":0,"runs":[{"createdAt":"2026-10-09T19:15:28.998Z","startedAt":"2026-10-09T19:15:53.558Z","queuedMs":24560.0}]},{"index":1,"runs":[{"createdAt":"2026-10-09T19:16:38.520Z","startedAt":"2026-10-09T19:16:54.157Z","queuedMs":15637.0}]},{"index":2,"runs":[{"createdAt":"2026-10-09T19:17:10.587Z","startedAt":"2026-10-09T19:17:23.561Z","queuedMs":12974.0}]},{"index":3,"runs":[{"createdAt":"2026-10-09T19:17:43.807Z","startedAt":"2026-10-09T19:17:53.564Z","queuedMs":9757.0}]},{"index":4,"runs":[{"createdAt":"2026-10-09T19:18:28.113Z","startedAt":"2026-10-09T19:18:54.011Z","queuedMs":25898.0}]},{"index":5,"runs":[{"createdAt":"2026-10-09T19:19:12.343Z","startedAt":"2026-10-09T19:19:53.993Z","queuedMs":41650.0}]},{"index":6,"runs":[{"createdAt":"2026-10-09T19:20:14.258Z","startedAt":"2026-10-09T19:20:53.866Z","queuedMs":39608.0}]},{"index":7,"runs":[{"createdAt":"2026-10-09T19:21:23.019Z","startedAt":"2026-10-09T19:21:54.298Z","queuedMs":31279.0}]},{"index":8,"runs":[{"createdAt":"2026-10-09T19:22:14.671Z","startedAt":"2026-10-09T19:22:53.933Z","queuedMs":39262.0}]},{"index":9,"runs":[{"createdAt":"2026-10-09T19:23:06.155Z","startedAt":"2026-10-09T19:23:23.941Z","queuedMs":17786.0}]}],"limitation":"The server shut down before a final API read; queue timing is retained for the first ten trials only."},"conclusion":"No reliable end-to-end speedup established. The after median is lower, but only three of six adjacent matched pairs were faster; median paired improvement is approximately zero. The deterministic streaming-forwarding fix remains separately supported.","limitations":["Local dev server and headless Chromium on macOS arm64; six fresh chats per variant; not a production/cloud latency guarantee.","Current product and current release runner held constant; only three OpenCode modules replaced by their exact pre-PR contents for baseline.","Default model reasoning unchanged; no no-reasoning override or decision policy.","Upstream routing/cache and queue time vary; trials run serially in predeclared counterbalanced ABBA order.","Render proxy: capture-phase Send click to visible nonempty Hi markdown, then two animation frames; screenshot verifies the rendered answer. This is not instrumented compositor pixel timing.","Two setup-only failures, one startup failure, and one invalid-measurement successful pilot are retained separately, not silently treated as successful timings."],"reproducibility":{"harnessSha256":"f93996b306c7d31c19bfcb9cc197653f867944ffb900d81a23a34f831e17c0c5","variantBuilderSha256":"4e88a7bbdc0de5d72601194ebe3b96d9c6f6eda5a33d8e0f00cf2d5d14db34c2","analyzerSha256":"2d129177a1b8033f768aad3c82dc8fff70ba0ba76e7270b2d8612ee760d4aaf4"}} ```
- Latest options-struct refactor: 33 provider tests and 86 backend tests passed. Four port-dependent fixtures initially failed inside the sandbox and passed with loopback access. Latest check snapshot: 51 successes, 4 skips, and 2 failures (`ci / e2e` and `ci / e2e shard (5/8)`). This latency experiment does not make the PR merge-ready. - Focused TypeScript suites passed: OpenCode driver, per-turn reasoning, prepared-context recovery, proxy envelope, live sessions, Codex context control, and native instruction measurements. The real runnerd/proxy integration test confirms disabled followed by default in the same session. A stale staged binary initially hid the new parameter; rebuilding the release runner resolved it. - Capability-guard regression suites passed all 53 tests; the real runner/proxy boundary still accepts the per-turn selection. Both OpenRouter and non-OpenRouter real runner/proxy cases passed, along with 62 existing OpenCode fixture tests. Runner TypeScript build passed after the final correction. - Affected Rust modules passed: 33 provider tests and 86 durable backend tests. Malformed or unsupported reasoning selections do not dispatch a provider turn or poison admission state. - Pinned OpenCode 1.18.34 with a local OpenRouter-compatible fixture sent three successive provider requests with reasoning omitted, `enabled: false`, then omitted. Each request contained one copy of the agent instructions. No paid provider call was needed. - The reconstructed first-turn task envelope shrank from 4,356 to 460 characters. This is a payload-size measurement, not a claimed token or latency reduction. Existing stored conversation history is not rewritten. - Prior head `8c07fe2e70267a7c8a3118f1071eca73ecdc6c8a`: all 53 checks passed, with four intentional skips. Greptile is 5/5 with no open findings. One bounded retry cleared first-test timeouts, a legacy signoff lease failure, and CI runner shutdowns. The timed-out suites also passed locally (14 auth-signal tests, 40 device-login tests, and the runner resume case). - Known reported diagnostic usage totals about $0.0029. Two prematurely closed baseline warm-turn attempts are excluded from latency comparisons and their usage is not fully captured; background provider usage is also not fully accounted. No further paid runs are active. - `pnpm -r typecheck`: passed. Runner TypeScript typechecks and build also passed after the review correction. - `pnpm test:run`: attempted and stopped after a reproducible failure in the untouched Codex recovery test `archives a damaged prior epoch before completing guarded same-task replacement with real runnerd` (`server/src/services/native-runtime/native-session-resume.test.ts:1072`). It returns `needs_review` / failed instead of done / succeeded, including when rerun alone. The broad local suite is not green. The two initial collection failures disappeared after building; their isolated rerun passed all 1,565 tests. - `pnpm build`: passed. - Pinned real OpenCode plus a delayed local provider: first native delta to Paperclip delta fell from 1,496 ms to less than 1 ms at millisecond measurement resolution. This isolates forwarding delay, not network/model inference. - Two live driver samples after the patch reached first text in 2.28–2.62 seconds including session startup. Baseline first responses took 12.37–13.43 seconds. Small unmatched samples had different reasoning/cache outcomes, so these are observations, not an attributed speedup ratio. - Full compiled runnerd protocol diagnostic: completed with `Hi 👋`; first assistant delta at 9.19 seconds and cleanup complete at 9.44 seconds. The seeded mock-control-plane harness used 27,512 input tokens across two provider requests, split into 13,521 and 13,991. The extra `answer_status_question` call posted `Hi 👋` to the mock task in 12 ms, then caused another model request. A reconstruction contains 25 tools (35.3k JSON characters), 12.8k system characters, and a 4.4k task envelope; 3.9k instruction characters appear in both system and task context. Provider-reported cost was $0.00073976; the pricing-table estimate was $0.00395332. This is one integration diagnostic, not a production latency claim. - A separate full-runner sample with existing lazy tool discovery also completed, but took 14.64 seconds through cleanup and called `paperclip_finish`. Its usage ledger recorded 12,888 input and 11,776 cached-input tokens, with $0.00096344 provider-reported cost. This did not establish a latency improvement, so the default tool exposure was not changed. - Four direct OpenRouter requests separately measured default reasoning at 2.45/3.90 seconds to first text and disabled reasoning at 0.66/0.53 seconds. Default reasoning remains unchanged; disabling it requires the explicit per-turn option. ## Risks - Immediate readiness dispatch changes scheduling timing. It uses existing admission checks and preserves the periodic sweep as recovery. - Native model verification relies on the real request rather than catalog enumeration. The final live batch had provider startup failures; overall speed and reliability are not qualified. - Incremental chunks without IDs cannot safely be deduplicated by content. Repeated identical chunks are valid. Known-ID duplicates remain suppressed. - OpenCode uses its bundled model catalog plus the explicit selected-model entry. New remote catalog metadata will not refresh during isolated startup. - Default provider plugins are disabled. This runner supplies credentials and assigned MCP tools through its own controlled configuration. - Disabled reasoning is supported only on the OpenCode/OpenRouter path and requires a compatible model. It applies to one turn and is not an automatic latency policy. - The matched live browser sample is small and does not establish a reliable end-to-end improvement. It includes local queueing/startup and browser rendering, but does not qualify production or remote sandbox latency. ## Model Used OpenAI GPT-6 (Codex). Tool use, code editing, and command execution were used. The session does not expose a more specific model ID or context-window size. Live diagnostic target: DeepSeek `deepseek/deepseek-v4-flash-0731` through OpenRouter and OpenCode 1.18.34. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/AGENT-LIFECYCLE.md | 13 +++ .../src/server/test.remote.test.ts | 59 +++++++++++- .../opencode-local/src/server/test.ts | 34 ++++++- packages/paperclip-runner/README.md | 31 ++++++ .../crates/runner-core/src/codex_provider.rs | 27 +++++- .../runner-core/src/provider_backend.rs | 49 +++++++++- .../backends/harness-driver-backend.test.ts | 28 ++++++ .../src/backends/harness-driver-backend.ts | 4 + .../native-instruction-measurement.test.ts | 3 +- .../src/cli/opencode-app-server-proxy.test.ts | 42 +++++---- .../src/cli/opencode-app-server-proxy.ts | 2 + .../src/cli/opencode-proxy-command.ts | 11 ++- .../src/cli/opencode-proxy-task-envelope.ts | 3 +- .../src/contracts/harness-driver.ts | 3 + .../src/contracts/native-session-backend.ts | 2 + .../paperclip-runner/src/contracts/types.ts | 2 + .../src/drivers/codex/app-server-transport.ts | 1 + ...-app-server-driver.context-control.test.ts | 13 +++ .../drivers/codex/codex-harness-session.ts | 10 ++ .../opencode/opencode-server-driver.test.ts | 75 ++++++++++++++- .../opencode/opencode-server-driver.ts | 94 +++++++++++++++---- .../prepared-context-transport.test.ts | 4 + .../drivers/opencode/reasoning-mode.test.ts | 63 +++++++++++++ .../src/drivers/opencode/reasoning-mode.ts | 8 ++ .../paperclip-runner/src/live/live-session.ts | 3 +- .../src/live/runnerd-codex-transport.test.ts | 23 ++++- .../src/live/runnerd-codex-transport.ts | 20 ++++ .../test/fixtures/fake-opencode-server.mjs | 50 +++++++++- ...artbeat-responsible-user-invariant.test.ts | 22 +++++ .../__tests__/diagnostics.test.ts | 48 +++++++++- .../agent-lifecycle/application/ports.ts | 4 +- .../agent-lifecycle/application/worker.ts | 8 ++ server/src/services/agent-lifecycle-driver.ts | 3 + server/src/services/heartbeat.ts | 6 ++ 34 files changed, 707 insertions(+), 61 deletions(-) create mode 100644 packages/paperclip-runner/src/drivers/opencode/reasoning-mode.test.ts create mode 100644 packages/paperclip-runner/src/drivers/opencode/reasoning-mode.ts diff --git a/doc/AGENT-LIFECYCLE.md b/doc/AGENT-LIFECYCLE.md index 96767c83131..87e0967316f 100644 --- a/doc/AGENT-LIFECYCLE.md +++ b/doc/AGENT-LIFECYCLE.md @@ -54,6 +54,19 @@ An old result cannot complete a newer operation. Deletion requires completed termination or a rejected hire. The existing restrictions for built-in agents and accounting still apply. +Messages accepted during setup remain queued until verification completes. +After the worker commits `ready`, its driver immediately asks the scheduler to +dispatch that agent's saved work. The scheduler rechecks company scope, +invokability, budgets, concurrency, and task ownership through normal admission. +If dispatch fails, readiness remains committed and the periodic queue sweep +retries; setup is not repeated because of a scheduler failure. + +Native OpenCode verification checks the selected model with the real hello +probe. It validates the model identifier but does not first enumerate the full model +catalog. Authentication failures, unavailable models, and probe timeouts still +prevent readiness. Without an explicit local working directory, the native +probe uses a temporary empty directory instead of the server's checkout. + ## Commands and transactions `requestHire` writes the agent record before preparation starts. diff --git a/packages/adapters/opencode-local/src/server/test.remote.test.ts b/packages/adapters/opencode-local/src/server/test.remote.test.ts index aebe6cfa0a9..b9a63a48b97 100644 --- a/packages/adapters/opencode-local/src/server/test.remote.test.ts +++ b/packages/adapters/opencode-local/src/server/test.remote.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp, rm } from "node:fs/promises"; +import { mkdtemp, rm, stat } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -72,6 +72,13 @@ vi.mock("@paperclipai/adapter-utils/execution-target", async () => { }); import { testEnvironment } from "./test.js"; +import { discoverOpenCodeModels, ensureOpenCodeModelConfiguredAndAvailable } from "./models.js"; + +vi.mock("./models.js", async () => ({ + ...await vi.importActual("./models.js"), + discoverOpenCodeModels: vi.fn().mockRejectedValue(new Error("catalog should not be needed")), + ensureOpenCodeModelConfiguredAndAvailable: vi.fn().mockRejectedValue(new Error("catalog should not be needed")), +})); describe("opencode remote environment diagnostics", () => { const configHomes: string[] = []; @@ -89,6 +96,56 @@ describe("opencode remote environment diagnostics", () => { await Promise.all(configHomes.splice(0).map(dir => rm(dir, { recursive: true, force: true }))); }); + it.each(["passed", "auth_required", "model_unavailable", "timed_out"])( + "validates a native model with the real probe path and no catalog scan (%s)", async outcome => { + const successfulProbe = await runAdapterExecutionTargetProcess.getMockImplementation()!(); + runAdapterExecutionTargetProcess.mockResolvedValueOnce(outcome === "passed" ? successfulProbe : { + ...successfulProbe, exitCode: 1, stdout: "", timedOut: outcome === "timed_out", + stderr: outcome === "auth_required" ? "invalid API key" : "ProviderModelNotFoundError", + }); + const result = await testEnvironment({ companyId: "company-1", adapterType: "paperclip_runner", + config: { model: "openrouter/deepseek/deepseek-v4-flash-0731", env: { XDG_CONFIG_HOME: configHome } } }); + expect(discoverOpenCodeModels).not.toHaveBeenCalled(); + expect(ensureOpenCodeModelConfiguredAndAvailable).not.toHaveBeenCalled(); + expect(runAdapterExecutionTargetProcess).toHaveBeenCalledExactlyOnceWith( + expect.any(String), null, "opencode", ["run", "--format", "json", "--model", "openrouter/deepseek/deepseek-v4-flash-0731"], + expect.objectContaining({ stdin: "Respond with hello." }), + ); + expect(result.status).toBe(outcome === "passed" ? "pass" : "warn"); + expect(result.checks).toContainEqual(expect.objectContaining({ code: `opencode_hello_probe_${outcome}` })); + const call = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as [unknown, unknown, unknown, unknown, { cwd: string }]; + expect(path.basename(call[4].cwd)).toMatch(/^paperclip-opencode-native-probe-/); + await expect(stat(call[4].cwd)).rejects.toMatchObject({ code: "ENOENT" }); + }, + ); + + it("rejects a malformed native model before starting the probe", async () => { + const result = await testEnvironment({ companyId: "company-1", adapterType: "paperclip_runner", + config: { model: "missing-provider", env: { XDG_CONFIG_HOME: configHome } } }); + expect(result.status).toBe("fail"); + expect(runAdapterExecutionTargetProcess).not.toHaveBeenCalled(); + expect(discoverOpenCodeModels).not.toHaveBeenCalled(); + }); + + it("preserves an explicitly configured native probe directory", async () => { + await testEnvironment({ companyId: "company-1", adapterType: "paperclip_runner", + config: { model: "provider/model", cwd: configHome, env: { XDG_CONFIG_HOME: configHome } } }); + const call = runAdapterExecutionTargetProcess.mock.calls[0] as unknown as [unknown, unknown, unknown, unknown, { cwd: string }]; + expect(call[4].cwd).toBe(configHome); + expect((await stat(configHome)).isDirectory()).toBe(true); + }); + + it("keeps catalog validation for the legacy local adapter", async () => { + vi.mocked(discoverOpenCodeModels).mockResolvedValueOnce([{ id: "provider/model", label: "Model" }]); + vi.mocked(ensureOpenCodeModelConfiguredAndAvailable).mockResolvedValueOnce([{ id: "provider/model", label: "Model" }]); + const result = await testEnvironment({ companyId: "company-1", adapterType: "opencode_local", + config: { model: "provider/model", env: { XDG_CONFIG_HOME: configHome } } }); + expect(result.status).toBe("pass"); + expect(discoverOpenCodeModels).toHaveBeenCalledOnce(); + expect(ensureOpenCodeModelConfiguredAndAvailable).toHaveBeenCalledOnce(); + expect(runAdapterExecutionTargetProcess).toHaveBeenCalledOnce(); + }); + it.each([false, true])("stages remote runtime config assets for sandbox hello probes (managed=%s)", async (managed) => { const configHome = await mkdtemp(path.join(os.tmpdir(), "opencode-remote-test-config-")); configHomes.push(configHome); diff --git a/packages/adapters/opencode-local/src/server/test.ts b/packages/adapters/opencode-local/src/server/test.ts index 7f2480b9c90..628745e9772 100644 --- a/packages/adapters/opencode-local/src/server/test.ts +++ b/packages/adapters/opencode-local/src/server/test.ts @@ -25,7 +25,7 @@ import { prepareAdapterExecutionTargetRuntime, overrideAdapterExecutionTargetRemoteCwd, } from "@paperclipai/adapter-utils/execution-target"; -import { discoverOpenCodeModels, ensureOpenCodeModelConfiguredAndAvailable } from "./models.js"; +import { discoverOpenCodeModels, ensureOpenCodeModelConfiguredAndAvailable, requireOpenCodeModelId } from "./models.js"; import { parseOpenCodeJsonl } from "./parse.js"; import { SANDBOX_INSTALL_COMMAND } from "../index.js"; import { prepareOpenCodeRuntimeConfig, prepareManagedOpenCodeRemoteHomes } from "./runtime-config.js"; @@ -141,9 +141,16 @@ export async function testEnvironment( // still has the path available for cleanup in `finally` — otherwise the // `fs.mkdtemp` directory leaks on the early-throw path. let preparedRuntimeWorkspaceLocalDir: string | null = null; + let nativeProbeWorkspaceLocalDir: string | null = null; try { let runtimeTarget: AdapterExecutionTarget | null = target ?? null; let runtimeCwd = cwd; + // Native workspaces are assigned at execution time. A configuration probe + // without an explicit workspace must not run in the server's checkout. + if (!targetIsRemote && ctx.adapterType === "paperclip_runner" && !asString(config.cwd, "").trim()) { + nativeProbeWorkspaceLocalDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-opencode-native-probe-")); + runtimeCwd = nativeProbeWorkspaceLocalDir; + } if (targetIsRemote) { preparedRuntimeWorkspaceLocalDir = await fs.mkdtemp(path.join(os.tmpdir(), `paperclip-opencode-envtest-${runId}-`)); const preparedExecutionTargetRuntime = await prepareAdapterExecutionTargetRuntime({ @@ -223,12 +230,30 @@ export async function testEnvironment( let modelValidationPassed = false; const configuredModel = asString(config.model, "").trim(); + // The native runner already selects a concrete model. Its real hello probe + // verifies model access and credentials; a full catalog scan adds another + // cold CLI launch (and discovery retries) without verifying that request. + const nativeModelProbe = ctx.adapterType === "paperclip_runner" && Boolean(configuredModel); + // Model discovery and validation use local child processes against // OpenCode's `models` subcommand and JSON config; these are not yet // wired through the execution target. When probing a remote env, skip // discovery/validation and rely on the remote hello probe to surface // model/auth issues directly. - if (targetIsRemote && configuredModel) { + if (nativeModelProbe) { + try { + requireOpenCodeModelId(configuredModel); + modelValidationPassed = true; + checks.push({ + code: "opencode_model_validation_via_probe", + level: "info", + message: "The configured native-runner model will be validated by the hello probe.", + }); + } catch (error) { + checks.push({ code: "opencode_model_invalid", level: "error", + message: error instanceof Error ? error.message : "Configured model is invalid." }); + } + } else if (targetIsRemote && configuredModel) { checks.push({ code: "opencode_model_validation_skipped_remote", level: "info", @@ -305,7 +330,7 @@ export async function testEnvironment( const modelUnavailable = checks.some((check) => check.code === "opencode_hello_probe_model_unavailable"); if (!configuredModel && !modelUnavailable) { // No model configured – skip model requirement if no model-related checks exist - } else if (!targetIsRemote && configuredModel && canRunProbe) { + } else if (!nativeModelProbe && !targetIsRemote && configuredModel && canRunProbe) { try { await ensureOpenCodeModelConfiguredAndAvailable({ model: configuredModel, @@ -430,6 +455,9 @@ export async function testEnvironment( } } } finally { + if (nativeProbeWorkspaceLocalDir) { + await fs.rm(nativeProbeWorkspaceLocalDir, { recursive: true, force: true }).catch(() => {}); + } await restoreWorkspace?.(); if (!restoreWorkspace && preparedRuntimeWorkspaceLocalDir) { // Reached when `prepareAdapterExecutionTargetRuntime` threw before diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index f87a1d07652..a7f0647a39f 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -238,6 +238,37 @@ provider-native operations. Full auto is resolved consistently for agent creation, adapter conversion, direct driver launches, and fresh/resumed turns. Explicitly stored restrictive modes still apply. +Native OpenCode forwards incremental text and reasoning parts as they arrive; +the final full snapshot selects the final response without repeating streamed +text. Delta parts must match an observed message and part identity. Identical +chunks without event IDs remain distinct tokens. Each isolated launch disables +automatic updates, remote model-catalog refreshes, and default plugins; the +runner supplies the pinned executable, selected model, and assigned MCP tools. +Provider inference and reasoning still contribute to time to first text. These +startup settings do not disable model reasoning. + +On macOS, the verified OpenCode executable keeps its private launch pathname +until the provider process exits. Removing that pathname immediately after +spawn can kill or stall the signed binary before its health check succeeds. +Ownership, permissions, and file-identity checks remain enforced; cleanup also +runs on failed launches, and retries create a fresh launch snapshot. + +Call `session.startTurn({ message, reasoningMode: "disabled" })` to disable +reasoning for one OpenCode/OpenRouter turn. `CapabilityLiveSession.sendMessage` +accepts the same option in its second argument. The choice travels in the durable +`turn.start` command and selects an OpenCode model variant for that prompt only. +Omitting it (or passing `"default"`) on the next turn restores provider defaults, +including on a warm or recovered session. It is not an environment variable or +agent-wide setting. No automatic decision policy selects it. The model must +support disabling reasoning; other runner providers reject an explicit selection +before starting work. Native sessions expose support through the +`perTurnReasoning` capability. + +OpenCode's proxy sends agent instructions once per turn through the system prompt. The +task envelope retains task-specific constraints and the completion contract, +without repeating the system instructions. This applies to all users of the +proxy; already-recorded conversation history is not rewritten. + The runner's authenticated bridge and controller still enforce company access, action claims, task modes, and governed approvals. Provider permission defaults do not change workspace isolation or grant credentials or connection access. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs index 29494a4e21a..61432b9dbe0 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs @@ -386,6 +386,13 @@ pub struct CodexSkillInput { pub path: String, } +/// Optional inputs for one turn; omitted fields preserve provider defaults. +#[derive(Clone, Copy, Debug, Default)] +pub struct CodexTurnOptions<'a> { + pub skills: &'a [CodexSkillInput], + pub reasoning_mode: Option<&'a str>, +} + impl CodexSkillInput { pub fn validate(&self) -> Result<(), LocalRunnerError> { if self.input_type != "skill" @@ -1670,15 +1677,26 @@ impl CodexProvider { } pub fn start_turn(&mut self, message: &str, cwd: &str) -> Result { - self.start_turn_with_skills(message, cwd, &[]) + self.start_turn_with_options(message, cwd, CodexTurnOptions::default()) } - pub fn start_turn_with_skills( + pub fn start_turn_with_options( &mut self, message: &str, cwd: &str, - skills: &[CodexSkillInput], + options: CodexTurnOptions<'_>, ) -> Result { + let CodexTurnOptions { + skills, + reasoning_mode, + } = options; + if reasoning_mode.is_some_and(|mode| { + self.config.provider != "opencode" || !matches!(mode, "default" | "disabled") + }) { + return Err(LocalRunnerError::invalid( + "turn.reasoningMode requires OpenCode and must be default or disabled", + )); + } if skills.len() > 64 || (self.config.provider != "codex" && !skills.is_empty()) { return Err(LocalRunnerError::invalid( "explicit skills require Codex and at most 64 selections", @@ -1727,6 +1745,9 @@ impl CodexProvider { let turn_params_object = turn_params .as_object_mut() .expect("Codex turn parameters are an object"); + if let Some(mode) = reasoning_mode { + turn_params_object.insert("reasoningMode".to_owned(), json!(mode)); + } if self.permission_profile == "paperclip-runner-external-sandbox" { turn_params_object.insert( "sandboxPolicy".to_owned(), diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index 23268296bd6..329c3dc7bf4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -13,7 +13,7 @@ use serde_json::{json, Value}; use sha2::{Digest, Sha256}; use crate::codex_provider::{ - CodexProvider, CodexProviderConfig, CodexProviderEvent, CodexSkillInput, + CodexProvider, CodexProviderConfig, CodexProviderEvent, CodexSkillInput, CodexTurnOptions, ProviderStartupObservation, ProviderStartupStage, RejectedAcceptedTurn, MAX_SETTLED_PROVIDER_TURN_IDS, }; @@ -2903,7 +2903,24 @@ impl CodexCommandExecutor { .validate() .map_err(|error| DurableRunnerError::invalid(error.to_string()))?; } + let reasoning_mode = match payload.get("reasoningMode") { + None => None, + Some(Value::String(mode)) if matches!(mode.as_str(), "default" | "disabled") => { + Some(mode.as_str()) + } + Some(_) => return Err(DurableRunnerError::invalid("invalid turn.reasoningMode")), + }; self.restore_provider_if_needed()?; + if reasoning_mode.is_some() + && self + .state + .as_ref() + .is_none_or(|state| state.config.provider != "opencode") + { + return Err(DurableRunnerError::invalid( + "turn.reasoningMode requires OpenCode", + )); + } if self .state .as_ref() @@ -2986,7 +3003,14 @@ impl CodexCommandExecutor { rejected_accepted_turn, ) = { let provider = self.ensure_provider()?; - let result = provider.start_turn_with_skills(text, &cwd, &skills); + let result = provider.start_turn_with_options( + text, + &cwd, + CodexTurnOptions { + skills: &skills, + reasoning_mode, + }, + ); ( result, provider.completed_turn_authority().is_some(), @@ -6068,6 +6092,27 @@ mod tests { executor.state = Some(state); executor.restore_checked = true; + for mode in [ + json!(null), + json!(false), + json!("bad"), + json!("disabled"), + json!("default"), + ] { + let error = executor + .start_turn(&json!({"text": "must not reach the provider", "reasoningMode": mode})) + .unwrap_err(); + assert!(error.to_string().contains("reasoningMode")); + assert!(executor.provider.is_none()); + assert!( + !executor + .state + .as_ref() + .unwrap() + .ambiguous_turn_start_pending + ); + } + let error = executor .start_turn(&json!({"text": "must not reach the provider"})) .expect_err("the exact regular backlog limit must reject admission"); diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts index e1542606a24..a5cc423e60e 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.test.ts @@ -97,6 +97,34 @@ const driver: HarnessDriver = { }; describe("HarnessDriverBackend", () => { + it.each(["default", "disabled"] as const)("rejects unsupported reasoning %s before starting provider work", async reasoningMode => { + const startTurn = vi.fn(async () => ({ turnId: "turn-1" })); + const session = Object.assign(new FakeHarnessSession(), { startTurn }); + const backend = new HarnessDriverBackend({ ...driver, openSession: async () => session }); + const opened = await backend.openSession({ identity: { + runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1", + } }); + expect((await opened.capabilities()).perTurnReasoning).toBe(false); + const message = { role: "user" as const, text: "Hi" }; + await expect(opened.startTurn({ message, reasoningMode })).rejects.toThrow("Per-turn reasoning is not supported"); + expect(startTurn).not.toHaveBeenCalled(); + await expect(opened.startTurn({ message })).resolves.toEqual({ turnId: "turn-1" }); + expect(startTurn).toHaveBeenCalledExactlyOnceWith({ message }); + }); + + it("forwards reasoning when the provider explicitly supports it", async () => { + const startTurn = vi.fn(async () => ({ turnId: "turn-1" })); + const session = Object.assign(new FakeHarnessSession(), { startTurn, supportsTurnReasoning: () => true }); + const backend = new HarnessDriverBackend({ ...driver, openSession: async () => session }); + const opened = await backend.openSession({ identity: { + runId: "run-1", sessionId: "session-1", companyId: "company-1", issueId: "issue-1", agentId: "agent-1", + } }); + expect((await opened.capabilities()).perTurnReasoning).toBe(true); + const input = { message: { role: "user" as const, text: "Hi" }, reasoningMode: "disabled" as const }; + await opened.startTurn(input); + expect(startTurn).toHaveBeenCalledExactlyOnceWith(input); + }); + it("refreshes negotiated controls after a warm handshake and separates follow-ups", async () => { let controls = { steering: false, queuedFollowUp: false }; const steer = vi.fn(async () => {}); diff --git a/packages/paperclip-runner/src/backends/harness-driver-backend.ts b/packages/paperclip-runner/src/backends/harness-driver-backend.ts index 6e3897e4e57..fcdb43e16b7 100644 --- a/packages/paperclip-runner/src/backends/harness-driver-backend.ts +++ b/packages/paperclip-runner/src/backends/harness-driver-backend.ts @@ -454,6 +454,7 @@ class HarnessNativeSession implements NativeSession { return { resume: true, typedEvents: true, + perTurnReasoning: this.#session.supportsTurnReasoning?.() === true, steering: (negotiated?.steering ?? this.#steeringSupported) && this.#session.steer !== undefined, queuedFollowUp: negotiated?.queuedFollowUp === true && this.#session.steer !== undefined, interruption: this.#session.interrupt !== undefined, @@ -710,6 +711,9 @@ class HarnessNativeSession implements NativeSession { // The provider has no active turn to interrupt yet. Do not launch work // whose events cancellation would suppress and leave the owner waiting. if (this.#explicitlyCancelled) throw new Error("native_session_cancelled"); + if (input.reasoningMode !== undefined && this.#session.supportsTurnReasoning?.() !== true) { + throw new Error("Per-turn reasoning is not supported by this provider"); + } this.#terminal = null; try { const started = await this.#session.startTurn(input); diff --git a/packages/paperclip-runner/src/backends/native-instruction-measurement.test.ts b/packages/paperclip-runner/src/backends/native-instruction-measurement.test.ts index 98507b24338..2c5592af47e 100644 --- a/packages/paperclip-runner/src/backends/native-instruction-measurement.test.ts +++ b/packages/paperclip-runner/src/backends/native-instruction-measurement.test.ts @@ -190,11 +190,10 @@ describe("direct OpenCode HTTP instruction boundary", () => { const request = requests.at(-1)!; expect(request.providerID).toBe("openrouter"); expect(request.modelID).toBe("deepseek/deepseek-v4-flash-0731"); const text = request.parts[0].text as string; + expect(request.system).toBe(nativeSystemInstructions(value)); if (schema === "v4" && phase === "start") { - expect(request.system).toBe(nativeSystemInstructions(input)); expect(JSON.parse(text).task.constraints).toEqual(nativeTaskConstraints(input)); } else { - expect(request.system).toBeUndefined(); expect(JSON.parse(text)).toEqual(envelope); } directOpenCodeReceipts.push({ provider: "opencode", schema, phase, request: measure(request), diff --git a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.test.ts b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.test.ts index 0c672b356af..65250a0f8d5 100644 --- a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.test.ts +++ b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.test.ts @@ -9,7 +9,7 @@ import { rmSync, writeFileSync, } from "node:fs"; -import { spawnSync } from "node:child_process"; +import { spawn, spawnSync } from "node:child_process"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -28,7 +28,7 @@ describe("OpenCode runnerd proxy executable", () => { ])).toEqual({ command: "/proc/self/fd/7", commandFd: 7 }); }); - it("executes the inherited artifact through the nested child fd mapping", () => { + it("executes the inherited artifact through the nested child fd mapping", async () => { if (process.platform === "win32") return; if (process.platform === "darwin") { const root = mkdtempSync(join(tmpdir(), "paperclip-opencode-nested-")); @@ -45,22 +45,27 @@ describe("OpenCode runnerd proxy executable", () => { TRUSTED_OPENCODE_EXECUTABLE_ARG, command, ]); - binding.commandLifecycle?.beforeSpawn(); - const child = spawnSync(binding.command, [], { encoding: "utf8" }); - expect(child.error).toBeUndefined(); - expect(child.status).toBe(0); - expect(child.stdout).toBe("verified-nested-spawn"); - binding.commandLifecycle?.afterSpawn(); - expect(() => lstatSync(command)).toThrow(); - expect(() => lstatSync(directory)).toThrow(); - binding.commandLifecycle?.beforeSpawn(); - const retriedChild = spawnSync(binding.command, [], { encoding: "utf8" }); - expect(retriedChild.error).toBeUndefined(); - expect(retriedChild.status).toBe(0); - expect(retriedChild.stdout).toBe("verified-nested-spawn"); - binding.commandLifecycle?.afterSpawn(); - expect(() => lstatSync(command)).toThrow(); - expect(() => lstatSync(directory)).toThrow(); + for (let attempt = 0; attempt < 2; attempt += 1) { + binding.commandLifecycle!.beforeSpawn(); + const child = spawn(binding.command, [], { stdio: ["ignore", "pipe", "pipe"] }); + let stdout = ""; + child.stdout!.on("data", chunk => { stdout += chunk; }); + const exited = new Promise((resolve, reject) => { + child.once("error", reject); + child.once("close", resolve); + }); + binding.commandLifecycle!.afterSpawn(); + // Exercise the production async boundary, not spawnSync followed by + // cleanup after the executable has already finished. + expect(lstatSync(command).isFile()).toBe(true); + expect(() => binding.commandLifecycle!.beforeSpawn()).toThrow("still in use"); + expect(await exited).toBe(0); + expect(stdout).toBe("verified-nested-spawn"); + binding.commandLifecycle!.afterExit(); + binding.commandLifecycle!.afterExit(); + expect(() => lstatSync(command)).toThrow(); + expect(() => lstatSync(directory)).toThrow(); + } } finally { rmSync(root, { recursive: true, force: true }); } @@ -131,6 +136,7 @@ describe("OpenCode runnerd proxy task envelope", () => { criterionIds: ["objective", "verification"], }, })).toMatchObject({ + constraints: ["Use Paperclip MCP tools for semantic operations."], completionContract: { revision: "17", criteria: [ diff --git a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts index 11cc112dea7..0e60ac1f245 100644 --- a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts +++ b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { parseOpenCodeReasoningMode } from "../drivers/opencode/reasoning-mode.js"; import { createInterface } from "node:readline"; import { readFileSync } from "node:fs"; import { resolve } from "node:path"; @@ -365,6 +366,7 @@ async function handle(message: RpcMessage): Promise { .join("\n"); const turn = await session.startTurn({ message: { role: "user", text: messageText }, + ...(params.reasoningMode === undefined ? {} : { reasoningMode: parseOpenCodeReasoningMode(params.reasoningMode) }), }); activeTurnId = turn.turnId; // OpenCode normally publishes session/turn startup over SSE, but a fast diff --git a/packages/paperclip-runner/src/cli/opencode-proxy-command.ts b/packages/paperclip-runner/src/cli/opencode-proxy-command.ts index 2da3349bf34..3d73ce8d22b 100644 --- a/packages/paperclip-runner/src/cli/opencode-proxy-command.ts +++ b/packages/paperclip-runner/src/cli/opencode-proxy-command.ts @@ -39,6 +39,7 @@ export function trustedOpenCodeLaunchBinding( commandLifecycle?: { beforeSpawn(): void; afterSpawn(): void; + afterExit(): void; }; } { const command = args.length === 2 && args[0] === TRUSTED_OPENCODE_EXECUTABLE_ARG @@ -119,8 +120,7 @@ export function trustedOpenCodeLaunchBinding( let materializedIdentity: { dev: number; ino: number } | undefined; const materializeForSpawn = () => { if (materialized) { - validateMacSnapshot(materializedIdentity); - return; + throw new Error("Verified OpenCode executable is still in use"); } let destinationFd: number | undefined; try { @@ -184,10 +184,15 @@ export function trustedOpenCodeLaunchBinding( // attackers are outside the documented local-host trust boundary in // docs/durable-recovery.md. Keep the verified source on an unlinked // descriptor, rematerialize only at the syscall boundary (including - // retries), then remove the executable pathname immediately. + // retries). Keep its private pathname until the child exits: deleting + // a signed executable during macOS startup can kill or stall the child. beforeSpawn: materializeForSpawn, afterSpawn() { validateMacSnapshot(materializedIdentity); + }, + afterExit() { + if (!materialized) return; + validateMacSnapshot(materializedIdentity); unlinkSync(command); rmdirSync(dirname(command)); materialized = false; diff --git a/packages/paperclip-runner/src/cli/opencode-proxy-task-envelope.ts b/packages/paperclip-runner/src/cli/opencode-proxy-task-envelope.ts index dd30c666d90..535b64d6d30 100644 --- a/packages/paperclip-runner/src/cli/opencode-proxy-task-envelope.ts +++ b/packages/paperclip-runner/src/cli/opencode-proxy-task-envelope.ts @@ -30,7 +30,8 @@ export function openCodeProxyTaskEnvelope(params: Record) { } : {}), constraints: [ - text(params.baseInstructions, "Complete only the supplied task."), + // baseInstructions already travels in the system prompt. Keep task + // constraints here without duplicating the agent instruction bundle. "Use Paperclip MCP tools for semantic operations.", ], }); diff --git a/packages/paperclip-runner/src/contracts/harness-driver.ts b/packages/paperclip-runner/src/contracts/harness-driver.ts index 410496afce5..a520a146431 100644 --- a/packages/paperclip-runner/src/contracts/harness-driver.ts +++ b/packages/paperclip-runner/src/contracts/harness-driver.ts @@ -498,6 +498,7 @@ export interface HarnessSessionRecoveryResult { } export interface HarnessSession { + supportsTurnReasoning?(): boolean; turnControlCapabilities?(): NativeTurnControlCapabilities | null; ids(): { driverSessionId: string; @@ -511,6 +512,8 @@ export interface HarnessSession { /** Set by orchestration only after successful provider-session recovery. */ continuation?: true; requestedCollaborationMode?: "default" | "plan"; + /** OpenCode/OpenRouter only. Applies to this turn, never subsequent turns. */ + reasoningMode?: "default" | "disabled"; }): Promise<{ turnId: string; effectiveCollaborationMode?: "default" | "plan"; diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.ts index f473b56a03e..251d10748e7 100644 --- a/packages/paperclip-runner/src/contracts/native-session-backend.ts +++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts @@ -193,6 +193,8 @@ export interface NativeSession { /** Set by orchestration only after successful provider-session recovery. */ continuation?: true; requestedCollaborationMode?: "default" | "plan"; + /** OpenCode/OpenRouter only. Applies to this turn, never subsequent turns. */ + reasoningMode?: "default" | "disabled"; }): Promise<{ turnId: string; effectiveCollaborationMode?: "default" | "plan"; diff --git a/packages/paperclip-runner/src/contracts/types.ts b/packages/paperclip-runner/src/contracts/types.ts index 81bc3d24723..8a5dd2a49bd 100644 --- a/packages/paperclip-runner/src/contracts/types.ts +++ b/packages/paperclip-runner/src/contracts/types.ts @@ -41,6 +41,8 @@ export interface NativeSessionCapabilities { runtimeRequestHandoff?: boolean; goals?: boolean; threadLineage?: boolean; + /** Can select reasoning for one turn without changing subsequent turns. */ + perTurnReasoning?: boolean; collaborationModes?: Array<"default" | "plan">; unsupported?: string[]; } diff --git a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts index 3e08de1f22f..a65138a435b 100644 --- a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts +++ b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts @@ -35,6 +35,7 @@ export type CodexServerRequestHandler = ( ) => Promise>; export interface CodexAppServerTransport { + supportsTurnReasoning?(): boolean; /** Runner-owned live capability projection; absent on native Codex transports. */ turnControlCapabilities?(): NativeTurnControlCapabilities | null; request( diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts index cd9ee34129d..021c0ab99f7 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts @@ -45,6 +45,19 @@ import { import { rehydrateRunnerdItemNotification } from "../../live/runnerd-codex-transport.js"; describe("Codex app-server Codex driver", () => { + it.each(["default", "disabled"] as const)("rejects unsupported reasoning %s without sending a turn RPC", async reasoningMode => { + const transport = new FakeCodexTransport(); + const session = await makeDriver([transport]).openSession({ + runId: "run-reasoning", normalizedSessionId: "normalized-reasoning", workingDirectory: WORKSPACE, + }); + const message = { role: "user" as const, text: "Hi" }; + await expect(session.startTurn({ message, reasoningMode })).rejects.toThrow("Per-turn reasoning is not supported"); + expect(transport.calls.filter(call => call.method === "turn/start")).toHaveLength(0); + await session.startTurn({ message }); + expect(transport.calls.filter(call => call.method === "turn/start")).toHaveLength(1); + await session.close({ reason: "fixture complete" }); + }); + it("retains one correlation-bound steering acknowledgement after the rehydrated runnerd echo", async () => { const transport = new FakeCodexTransport("thread-root"); const session = await makeDriver([transport]).openSession({ diff --git a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts index 4efc9eec23d..983d9322eee 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts @@ -82,6 +82,10 @@ export class CodexHarnessSession } } + supportsTurnReasoning(): boolean { + return this.transport.supportsTurnReasoning?.() === true; + } + turnControlCapabilities() { if (this.driverKind === "acpx_runtime") { return this.transport.turnControlCapabilities?.() ?? { steering: false, queuedFollowUp: false }; @@ -157,11 +161,16 @@ export class CodexHarnessSession /** Set by orchestration only after successful provider-session recovery. */ continuation?: true; requestedCollaborationMode?: "default" | "plan"; + /** OpenCode/OpenRouter only. Applies to this turn, never subsequent turns. */ + reasoningMode?: "default" | "disabled"; }): Promise<{ turnId: string; effectiveCollaborationMode: "default" | "plan"; }> { this.assertProtocolIntegrity(); + if (input.reasoningMode !== undefined && !this.supportsTurnReasoning()) { + throw new Error("Per-turn reasoning is not supported by this provider"); + } if (this.protocolFailed && this.protocolFailureCode) { throw new NativeProviderTerminalFailure(this.protocolFailureCode, false, this.protocolFailureMessage ?? undefined); } @@ -231,6 +240,7 @@ export class CodexHarnessSession try { response = await this.transport.request("turn/start", { threadId: this.opened.threadId, + ...(input.reasoningMode === undefined ? {} : { reasoningMode: input.reasoningMode }), ...(this.reasoningEffort ? { effort: this.reasoningEffort } : {}), cwd: this.opened.context.workingDirectory, permissions: diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index 59d740d9940..75032bd82c7 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -906,6 +906,8 @@ describe("OpenCodeServerDriver", () => { expect(environment.keys).not.toContain("UNRELATED_SECRET"); expect(environment.keys).not.toContain("PAPERCLIP_PROVIDER_TRACE_PATH"); expect(environment.projectConfigDisabled).toBe("true"); + expect(environment.modelsFetchDisabled).toBe("true"); + expect(environment.defaultPluginsDisabled).toBe("true"); expect(mcpEvidence.tools).toEqual( expect.arrayContaining(["paperclip_finish", "paperclip_block"]), ); @@ -920,6 +922,46 @@ describe("OpenCodeServerDriver", () => { expect(diagnostics.join("\n")).toContain("[REDACTED]"); }); + it.each(["role-first", "role-last", "role-last-long"])("preserves incremental text and completion with identity checks (%s)", async (roleOrder) => { + await chmod(fixture, 0o755); + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-stream-")); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-opencode-stream-workspace-")); + roots.push(root, workspace); + const driver = new OpenCodeServerDriver({ + model: "openrouter/deepseek/deepseek-v4-flash-0731", + runtimeDirectory: root, + command: fixture, + environment: { PATH: process.env.PATH }, + }); + const session = await driver.openSession({ runId: "stream", normalizedSessionId: "stream", workingDirectory: workspace }); + try { + await session.startTurn({ message: { role: "user", text: `incremental-stream ${roleOrder}` } }); + const deltas: string[] = []; + const reasoning: string[] = []; + const events: PrpEvent[] = []; + for await (const event of session.events()) { + events.push(event); + if (event.eventType === "item.delta" && event.payload.kind === "reasoning") reasoning.push(String(event.payload.text)); + if (event.eventType === "item.delta" && event.payload.kind === "agentMessage") { + deltas.push(String(event.payload.text)); + // The fixture cannot finish its turn until a chunk reaches us. + if (deltas.length === 1) { + expect(await readFile(join(root, "stream", "data", "stream-released"), "utf8").catch(() => null)).toBeNull(); + } + await writeFile(join(root, "stream", "data", "release-stream"), "ready"); + } + if (TURN_TERMINAL_EVENT_TYPES.has(event.eventType)) break; + } + const expectedText = `Hi ${"👋".repeat(roleOrder === "role-last-long" ? 120 : 2)}`; + expect(deltas).toEqual(roleOrder === "role-first" ? ["Hi ", "👋", "👋"] : [expectedText]); + expect(reasoning).toEqual(["Thinking"]); + expect(events.filter(event => event.eventType === "item.completed" && event.payload.kind === "agentMessage") + .map(event => event.payload.text)).toEqual([expectedText]); + } finally { + await session.close({ reason: "stream test complete" }); + } + }); + it("maps OpenCode's normal abort error to cancellation without a false provider failure notice", async () => { await chmod(fixture, 0o755); const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-abort-")); @@ -1497,7 +1539,7 @@ describe("OpenCodeServerDriver", () => { await session.close({ reason: "test" }); }); - it("sends only the authoritative wake envelope when resuming an existing provider session", async () => { + it("resends system instructions without repeating the task envelope on recovery", async () => { await chmod(fixture, 0o755); const root = await mkdtemp( join(tmpdir(), "paperclip-opencode-resume-context-"), @@ -1553,7 +1595,7 @@ describe("OpenCodeServerDriver", () => { ], }); expect(submittedPrompt).not.toHaveProperty("tools"); - expect(submittedPrompt).not.toHaveProperty("system"); + expect(submittedPrompt).toHaveProperty("system", "large original system context"); await recovered!.session!.close({ reason: "recovery-test" }); }); @@ -1942,9 +1984,11 @@ describe("OpenCodeServerDriver", () => { "*": permissionMode, external_directory: { "*": "deny", [`${workspace}/**`]: "allow" }, }); + expect(config.autoupdate).toBe(false); expect(config.provider.openrouter.models).toHaveProperty( "deepseek/deepseek-v4-flash-0731", ); + expect(config.provider.openrouter.models["deepseek/deepseek-v4-flash-0731"].options).toBeUndefined(); expect(config.permission.paperclip_finish).toBeUndefined(); expect(config.permission["paperclip_*"]).toBe("allow"); await session.close({ reason: "permission mode test complete" }); @@ -2440,6 +2484,9 @@ describe("OpenCodeServerDriver", () => { afterSpawn: () => { commandLifecycle.push("after"); }, + afterExit: () => { + commandLifecycle.push("exit"); + }, }, fetch: async (input, init) => { if ( @@ -2465,9 +2512,31 @@ describe("OpenCodeServerDriver", () => { expect(spawns.length).toBeGreaterThanOrEqual(2); expect(spawns.length).toBeLessThanOrEqual(3); expect(commandLifecycle).toEqual( - Array.from({ length: spawns.length }, () => ["before", "after"]).flat(), + [...Array.from({ length: spawns.length - 1 }, () => ["before", "after", "exit"]).flat(), "before", "after"], ); await session.close({ reason: "test" }); + expect(commandLifecycle).toEqual( + Array.from({ length: spawns.length }, () => ["before", "after", "exit"]).flat(), + ); + }); + + it("releases the executable binding when spawning fails before health checks", async () => { + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-spawn-error-")); + roots.push(root); + const lifecycle: string[] = []; + const driver = new OpenCodeServerDriver({ + model: "openrouter/deepseek/deepseek-v4-flash-0731", + runtimeDirectory: root, + command: join(root, "missing-executable"), + commandLifecycle: { + beforeSpawn: () => { lifecycle.push("before"); }, + afterSpawn: () => { lifecycle.push("spawned"); }, + afterExit: () => { lifecycle.push("released"); }, + }, + }); + await expect(driver.openSession({ runId: "spawn-error", normalizedSessionId: "spawn-error", + workingDirectory: root })).rejects.toThrow("ENOENT"); + expect(lifecycle).toEqual(["before", "released"]); }); it("reports startup exit details with stderr captured after spawn and redacted", async () => { diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index 81abbfd3f06..4ce6a5b9dd9 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -19,6 +19,8 @@ import { getCACertificates } from "node:tls"; import { pipeline } from "node:stream/promises"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; +import { parseOpenCodeReasoningMode, type OpenCodeReasoningMode } from "./reasoning-mode.js"; + import { CODEX_SKILLLESS_BASE_INSTRUCTIONS, createCodexTaskEnvelope, @@ -176,6 +178,7 @@ export interface OpenCodeServerDriverOptions { commandLifecycle?: { beforeSpawn(): void; afterSpawn(): void; + afterExit?(): void; }; runtimeDirectory: string; systemInstructions?: string; @@ -479,6 +482,7 @@ class OpenCodeHarnessSession implements HarnessSession { Array> >(); readonly #partText = new Map(); + readonly #streamingParts = new Map>(); readonly #completedTextPartIds = new Set(); readonly #completedReasoningPartIds = new Set(); readonly #completedTextParts: Array<{ @@ -640,12 +644,22 @@ class OpenCodeHarnessSession implements HarnessSession { .finally(() => this.#pumpEvents()); } + supportsTurnReasoning(): boolean { + return this.#model.startsWith("openrouter/"); + } + async startTurn(input: { message: NativeUserMessage; + reasoningMode?: OpenCodeReasoningMode; }): Promise<{ turnId: string }> { if (this.#activeTurnId !== null) throw new Error("OpenCode session already has an active turn"); + const reasoningMode = parseOpenCodeReasoningMode(input.reasoningMode); + if (input.reasoningMode !== undefined && !this.#model.startsWith("openrouter/")) { + throw new Error("Per-turn OpenCode reasoning is supported only for OpenRouter models"); + } const turnId = `turn-${randomBytes(12).toString("hex")}`; + this.#streamingParts.clear(); this.#activeTurnId = turnId; this.#emit("turn.submitted", { envelopeSchema: this.#taskEnvelope.schema, @@ -655,12 +669,9 @@ class OpenCodeHarnessSession implements HarnessSession { this.#emit("turn.started", { status: "inProgress" }, { turnId }); const [providerID, ...modelParts] = this.#model.split("/"); const modelID = modelParts.join("/"); - // A resumed OpenCode provider session already retains the original system - // instructions and task envelope in its conversation. Repeating both on - // every Paperclip continuation can overflow smaller context windows and - // OpenCode then completes with `finish: unknown` and zero tokens. The - // native model envelope still carries the authoritative wake delta, - // interaction responses, completion contract, and current issue context. + // Keep the task envelope only on the initial task-mode wake. OpenCode + // rebuilds system instructions from the latest user message, so system + // instructions must still accompany every prompt (including recovery). const prompt = this.#sendFullContext ? JSON.stringify({ task: this.#taskEnvelope, @@ -682,12 +693,15 @@ class OpenCodeHarnessSession implements HarnessSession { // at this HTTP boundary. providerID, modelID, + // Always select the empty default variant on an ordinary turn so a + // previous disabled turn cannot change this turn's provider defaults. + ...(providerID === "openrouter" + ? { variant: reasoningMode === "disabled" ? "paperclip-no-reasoning" : "paperclip-default" } + : {}), // Keep question enabled in the isolated config. OpenCode 1.18.32 // turns a prompt's deprecated `tools` map into replacement session // permissions, so a sparse override here discards the session policy. - ...(this.#sendFullContext - ? { system: this.#systemInstructions } - : {}), + system: this.#systemInstructions, parts: [{ type: "text", text: prompt }], }), }, @@ -1470,9 +1484,11 @@ class OpenCodeHarnessSession implements HarnessSession { const event = record(value); const properties = record(event.properties); const type = text(event.type); - const eventId = text(event.id, canonicalJson(value)); - if (this.#seenProviderEvents.has(eventId)) return; - this.#seenProviderEvents.add(eventId); + // Id-less deltas are additive: two identical chunks can be legitimate + // adjacent tokens. Content deduplication would silently lose the second. + const eventId = text(event.id) || (type === "message.part.delta" ? null : canonicalJson(value)); + if (eventId !== null && this.#seenProviderEvents.has(eventId)) return; + if (eventId !== null) this.#seenProviderEvents.add(eventId); if (this.#seenProviderEvents.size > 10_000) this.#seenProviderEvents.delete( this.#seenProviderEvents.values().next().value!, @@ -1565,8 +1581,16 @@ class OpenCodeHarnessSession implements HarnessSession { ); return; } - if (type === "message.part.updated") { - const part = record(properties.part); + if (type === "message.part.updated" || type === "message.part.delta") { + let part = record(properties.part); + if (type === "message.part.delta") { + const previous = this.#streamingParts.get(text(properties.partID)); + // A delta has no role or part type. Only stream text for an observed + // part with matching message identity; never infer assistant identity. + if (!previous || properties.field !== "text" || typeof properties.delta !== "string" + || text(previous.messageID, text(previous.messageId)) !== text(properties.messageID)) return; + part = { ...previous, text: text(previous.text) + properties.delta }; + } const messageId = text(part.messageID, text(part.messageId)); if (!messageId) return; // Resolve the turn this message actually belongs to, not whichever @@ -1574,11 +1598,20 @@ class OpenCodeHarnessSession implements HarnessSession { // must stay attributed to the turn that created that message. const owningTurnId = this.#messageTurnIds.get(messageId) ?? turnId; if (!owningTurnId) return; + if (owningTurnId === turnId && text(part.id) && ["text", "reasoning"].includes(text(part.type))) { + this.#streamingParts.set(text(part.id), part); + } const role = this.#messageRoles.get(messageId); if (role === "assistant") this.#emitAssistantPart(part, owningTurnId); else if (role === undefined) { const pending = this.#pendingMessageParts.get(messageId) ?? []; - if (pending.length < 100) pending.push(part); + // Text deltas are cumulative by this point. Keep the newest snapshot + // (including completion metadata), not the first 100 token chunks. + const previousIndex = ["text", "reasoning"].includes(text(part.type)) && text(part.id) + ? pending.findIndex(candidate => candidate.id === part.id && candidate.type === part.type) + : -1; + if (previousIndex >= 0) pending[previousIndex] = part; + else if (pending.length < 100) pending.push(part); this.#pendingMessageParts.set(messageId, pending); } return; @@ -2233,12 +2266,23 @@ async function startRuntime(input: { input.trace?.addSensitiveValues([providerProxy.token]); } let child: ChildProcess | undefined; + let launchPrepared = false; + const releaseExecutable = () => { + if (!launchPrepared) return; + launchPrepared = false; + try { + input.options.commandLifecycle?.afterExit?.(); + } catch (error) { + input.options.onDiagnostic?.(redact(`OpenCode executable cleanup failed: ${String(error)}`, sensitiveValues)); + } + }; try { const config = { $schema: "https://opencode.ai/config.json", model: input.options.model, small_model: input.options.model, share: "disabled", + autoupdate: false, // The configured entry is already composed exactly once into the session // system prompt; siblings remain available through the read-only root. instructions: [], @@ -2257,7 +2301,15 @@ async function startRuntime(input: { }, } : {}), models: { - [providerModelId]: { name: providerModelId }, + [providerModelId]: { + name: providerModelId, + ...(modelProvider === "openrouter" + ? { variants: { + "paperclip-default": {}, + "paperclip-no-reasoning": { reasoning: { enabled: false } }, + } } + : {}), + }, }, }, }, @@ -2307,6 +2359,11 @@ async function startRuntime(input: { XDG_DATA_HOME: dataHome, XDG_CACHE_HOME: cacheHome, OPENCODE_DISABLE_PROJECT_CONFIG: "true", + // The runner pins the executable, model and assigned MCP tools. A + // fresh isolated cache must not fetch a catalog or install unrelated + // default plugins before it can submit the first prompt. + OPENCODE_DISABLE_MODELS_FETCH: "true", + OPENCODE_DISABLE_DEFAULT_PLUGINS: "true", OPENCODE_SERVER_USERNAME: username, OPENCODE_SERVER_PASSWORD: password, ...(providerProxy ? { NO_PROXY: [input.options.environment?.no_proxy ?? input.options.environment?.NO_PROXY, "127.0.0.1", "localhost"].filter(Boolean).join(",") } : {}), @@ -2319,6 +2376,7 @@ async function startRuntime(input: { stdio[input.options.commandFd] = input.options.commandFd; } input.options.commandLifecycle?.beforeSpawn(); + launchPrepared = true; child = spawn( input.options.command ?? resolvePinnedOpenCodeCommand(), ["serve", "--hostname", "127.0.0.1", "--port", String(port)], @@ -2329,6 +2387,7 @@ async function startRuntime(input: { detached: globalThis.process.platform !== "win32" && isolateProcessGroup, }, ); + child.once("exit", releaseExecutable); if (child.pid !== undefined) { try { input.options.commandLifecycle?.afterSpawn(); @@ -2425,6 +2484,7 @@ async function startRuntime(input: { } catch { providerChild.kill("SIGKILL"); } + await waitForExit(providerChild, 2_000); } await rm(join(configHome, "opencode", "opencode.json"), { force: true, @@ -2441,6 +2501,8 @@ async function startRuntime(input: { await providerProxy?.close(); await bridge.close().catch(() => {}); child?.kill("SIGKILL"); + if (child?.pid) await waitForExit(child, 2_000); + if (!child?.pid || child.exitCode !== null || child.signalCode !== null) releaseExecutable(); await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); diff --git a/packages/paperclip-runner/src/drivers/opencode/prepared-context-transport.test.ts b/packages/paperclip-runner/src/drivers/opencode/prepared-context-transport.test.ts index dd6f34e8677..c4710a971e9 100644 --- a/packages/paperclip-runner/src/drivers/opencode/prepared-context-transport.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/prepared-context-transport.test.ts @@ -30,6 +30,7 @@ describe("prepared OpenCode context transport", () => { runtimeDirectory: root, command: fixture, conversationMode: "prepared", + systemInstructions: "Prepared agent instructions.", environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "fixture-key" }, fetch: async (input, init) => { if (String(input).endsWith("/prompt_async")) @@ -45,6 +46,7 @@ describe("prepared OpenCode context transport", () => { }); await session.startTurn({ message: { role: "user", text: "prepared OpenCode initial wake" }, + reasoningMode: "disabled", }); for await (const event of session.events()) { if (event.eventType === "turn.completed") break; @@ -67,6 +69,8 @@ describe("prepared OpenCode context transport", () => { [{ type: "text", text: "completion-only OpenCode continuation" }], ]); expect(JSON.stringify(submitted)).not.toContain('"task"'); + expect(submitted.map(body => body.variant)).toEqual(["paperclip-no-reasoning", "paperclip-default"]); + expect(submitted.map(body => body.system)).toEqual(["Prepared agent instructions.", "Prepared agent instructions."]); await recovered!.session!.close({ reason: "prepared-test-recovery" }); }); }); diff --git a/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.test.ts b/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.test.ts new file mode 100644 index 00000000000..db5e747d46a --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.test.ts @@ -0,0 +1,63 @@ +import { chmod, mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { describe, expect, it } from "vitest"; +import { OpenCodeServerDriver } from "./opencode-server-driver.js"; +import { parseOpenCodeReasoningMode } from "./reasoning-mode.js"; +import { openCodeProxyTaskEnvelope } from "../../cli/opencode-proxy-task-envelope.js"; + +describe("per-turn OpenCode reasoning", () => { + it.each([undefined, "default"])("preserves provider defaults for %s", (value) => { + expect(parseOpenCodeReasoningMode(value)).toBe("default"); + }); + it.each(["", "false", null, false, 0])("rejects malformed mode %s", (value) => { + expect(() => parseOpenCodeReasoningMode(value)).toThrow("turn.reasoningMode"); + }); + it("selects reasoning independently on successive turns and sends instructions once", async () => { + const root = await mkdtemp(join(tmpdir(), "paperclip-turn-reasoning-")); + const fixture = resolve("test/fixtures/fake-opencode-server.mjs"); + await chmod(fixture, 0o755); + const submitted: Array<{ variant?: string; system?: string; parts: Array<{ text: string }> }> = []; + const instructions = "Unique shared agent instructions. Keep the requested scope."; + const driver = new OpenCodeServerDriver({ + model: "openrouter/deepseek/deepseek-v4-flash-0731", + runtimeDirectory: root, + command: fixture, + systemInstructions: instructions, + taskEnvelope: openCodeProxyTaskEnvelope({ baseInstructions: instructions, + completionContract: { revision: "17", criterionIds: ["objective"] } }), + environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "fixture-key" }, + fetch: async (input, init) => { + if (String(input).endsWith("/prompt_async")) submitted.push(JSON.parse(String(init?.body))); + return fetch(input, init); + }, + }); + let session: Awaited> | undefined; + try { + session = await driver.openSession({ runId: "turn-reasoning", normalizedSessionId: "reasoning", workingDirectory: root }); + await expect(session.startTurn({ message: { role: "user", text: "invalid" }, reasoningMode: "bad" as never })).rejects.toThrow("turn.reasoningMode"); + expect(submitted).toHaveLength(0); + for (const reasoningMode of [undefined, "disabled", undefined] as const) { + await session.startTurn({ message: { role: "user", text: "Reply Hi." }, reasoningMode }); + for await (const event of session.events()) if (event.eventType === "turn.completed") break; + } + expect(submitted.map(body => body.variant)).toEqual(["paperclip-default", "paperclip-no-reasoning", "paperclip-default"]); + expect(submitted[0]!.system).toBe(instructions); + for (const body of submitted) { + expect(body.system).toBe(instructions); + expect(JSON.stringify(body).split(instructions)).toHaveLength(2); + } + const envelope = JSON.parse(submitted[0]!.parts[0].text); + expect(envelope.task.completionContract.revision).toBe("17"); + expect(envelope.task.constraints).toEqual(["Use Paperclip MCP tools for semantic operations."]); + const config = JSON.parse(await readFile(join(root, "reasoning/config/opencode/opencode.json"), "utf8")); + expect(config.provider.openrouter.models["deepseek/deepseek-v4-flash-0731"]).toMatchObject({ + variants: { "paperclip-default": {}, "paperclip-no-reasoning": { reasoning: { enabled: false } } }, + }); + expect(config.provider.openrouter.models["deepseek/deepseek-v4-flash-0731"].options).toBeUndefined(); + } finally { + await session?.close(); + await rm(root, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.ts b/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.ts new file mode 100644 index 00000000000..003589ead39 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/reasoning-mode.ts @@ -0,0 +1,8 @@ +export type OpenCodeReasoningMode = "default" | "disabled"; + +/** Opt-in only. Omitting the setting preserves the provider's own defaults. */ +export function parseOpenCodeReasoningMode(value: unknown): OpenCodeReasoningMode { + if (value === undefined || value === "default") return "default"; + if (value === "disabled") return "disabled"; + throw new Error("turn.reasoningMode must be default or disabled"); +} diff --git a/packages/paperclip-runner/src/live/live-session.ts b/packages/paperclip-runner/src/live/live-session.ts index b79755a6685..c36b8036346 100644 --- a/packages/paperclip-runner/src/live/live-session.ts +++ b/packages/paperclip-runner/src/live/live-session.ts @@ -1514,7 +1514,7 @@ export class CapabilityLiveSession { async sendMessage( message: string, /** Launch-only diagnostics may opt out; qualification campaigns must not. */ - options: { allowMissingUsage?: boolean } = {}, + options: { allowMissingUsage?: boolean; reasoningMode?: "default" | "disabled" } = {}, ): Promise { const value = message.trim(); if (value.length === 0) throw new Error("Capability live messages cannot be empty"); @@ -1617,6 +1617,7 @@ export class CapabilityLiveSession { permissions: CODEX_PERMISSION_PROFILE, runtimeWorkspaceRoots: [this.#config.workingDirectory], input: [userInput(value)], + ...(options.reasoningMode === undefined ? {} : { reasoningMode: options.reasoningMode }), }); } catch (error) { if (this.#pendingTurnAdmission !== admission) { diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index a024eb20d92..debd0c6e3fe 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -7951,7 +7951,10 @@ it.each([true, false])("preserves prepared OpenCode cleanup errors (primary fail expect((failure as Error).message).not.toContain("fixture-secret"); }); -it("preserves prepared input and completion feedback through runnerd and the real OpenCode proxy boundary", async () => { +it.each([ + { model: "openrouter/deepseek/deepseek-v4-flash-0731", supportsReasoning: true }, + { model: "anthropic/claude-sonnet-4", supportsReasoning: false }, +])("preserves prepared input and completion feedback through runnerd and the real OpenCode proxy boundary ($model)", async ({ model, supportsReasoning }) => { const root = await mkdtemp(join(tmpdir(), "runnerd-prepared-opencode-")); // GitHub-hosted Linux toolcache Node can be group-writable, unlike the AWS // fleet. Qualify an owned copy with strict permissions, never chmod the host @@ -8006,7 +8009,7 @@ it("preserves prepared input and completion feedback through runnerd and the rea const driver = new CodexAppServerDriver({ taskEnvelope: task, conversationMode: "prepared", - model: "openrouter/deepseek/deepseek-v4-flash-0731", + model, transportFactory: () => bundle.transport, workingDirectoryAuthority: "remote_runner", environment: { PAPERCLIP_WORKSPACE_CWD: root }, @@ -8025,7 +8028,15 @@ it("preserves prepared input and completion feedback through runnerd and the rea await withPreparedOpenCodeCleanup({ run: async () => { session = await driver.openSession({ runId: "prepared-opencode", normalizedSessionId: "prepared-opencode", workingDirectory: root }); - await session.startTurn({ message: { role: "user", text: prepared } }); + expect(bundle.transport.supportsTurnReasoning?.()).toBe(supportsReasoning); + const firstTurn = { message: { role: "user" as const, text: prepared } }; + if (!supportsReasoning) { + await expect(session.startTurn({ ...firstTurn, reasoningMode: "disabled" })) + .rejects.toThrow("Per-turn reasoning is not supported"); + await expect(bundle.transport.request("turn/start", { input: [{ type: "text", text: prepared }], reasoningMode: "disabled" })) + .rejects.toThrow("requires an OpenCode/OpenRouter model"); + } + await session.startTurn({ ...firstTurn, ...(supportsReasoning ? { reasoningMode: "disabled" as const } : {}) }); const events: PrpEvent[] = []; for await (const event of session.events()) { events.push(event); @@ -8037,10 +8048,16 @@ it("preserves prepared input and completion feedback through runnerd and the rea expect(sessionRoots).toHaveLength(1); const requests = (await readFile(join(runtime, sessionRoots[0]!.name, "data/fake-prompt-requests.ndjson"), "utf8")).trim().split("\n").map((line) => JSON.parse(line)); expect(requests.map((request) => request.parts)).toEqual([[{ type: "text", text: prepared }]]); + expect(requests[0].variant).toBe(supportsReasoning ? "paperclip-no-reasoning" : undefined); const outcomes = JSON.parse(await readFile(join(runtime, sessionRoots[0]!.name, "data/fake-completion-feedback.json"), "utf8")); expect(outcomes).toHaveLength(2); expect(outcomes[0].result).toMatchObject({ isError: true, content: [{ text: expect.stringContaining("required document link") }] }); expect(outcomes[1].result).toMatchObject({ content: [{ text: expect.stringContaining(feedback) }] }); + await session.startTurn({ message: { role: "user", text: "Reply Hi." } }); + for await (const event of session.events()) if (event.eventType === "turn.completed") break; + const allRequests = (await readFile(join(runtime, sessionRoots[0]!.name, "data/fake-prompt-requests.ndjson"), "utf8")).trim().split("\n").map((line) => JSON.parse(line)); + expect(allRequests.map(request => request.variant)).toEqual(supportsReasoning + ? ["paperclip-no-reasoning", "paperclip-default"] : [undefined, undefined]); }, closeSession: async () => { await session?.close(); }, closeTransport: () => bundle.transport.close(), diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 08a8886f3b9..71103812f2a 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -3538,6 +3538,20 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { #providerIdentity: Record | null = null; #turnControls: NativeTurnControlCapabilities = { steering: false, queuedFollowUp: false }; + supportsTurnReasoning(): boolean { + if (this.options.provider !== "opencode") return false; + // Use the durable selected model on both fresh and recovered sessions. + // Older states may retain the provider only in the preparation command. + const state = this.#core?.store.state; + const template = this.#runAttachTemplate ?? state?.runAttachTemplate + ?? state?.commands.findLast(command => + (command.type === "run.prepare" || command.type === "run.attach") + && record(command.payload).provider !== undefined, + )?.payload; + const model = record(record(template).provider).model; + return typeof model === "string" && model.startsWith("openrouter/"); + } + turnControlCapabilities(): NativeTurnControlCapabilities | null { if (this.options.provider !== "acpx") return null; if (this.#closed || this.#failure) return { steering: false, queuedFollowUp: false }; @@ -5883,6 +5897,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { .filter((item) => item.type !== "skill") .map((item) => (typeof item.text === "string" ? item.text : "")) .join("\n"); + const reasoningMode = params.reasoningMode; + if (reasoningMode !== undefined && ( + !this.supportsTurnReasoning() + || (reasoningMode !== "default" && reasoningMode !== "disabled") + )) throw new Error("turn.reasoningMode requires an OpenCode/OpenRouter model and must be default or disabled"); const skills = resolveRunnerdCodexSkillInputs( input.filter((item) => item.type === "skill"), this.options.runtimeContext ?? null, @@ -5916,6 +5935,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { "turn.start", { text: message, + ...(reasoningMode === undefined ? {} : { reasoningMode }), ...(skills.length ? { skills } : {}), turnId: pendingTurnId, }, diff --git a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs index 69bd35a8e4e..692fd9c2ebd 100644 --- a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs +++ b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs @@ -99,6 +99,8 @@ await writeFile( home: process.env.HOME, configHome: process.env.XDG_CONFIG_HOME, projectConfigDisabled: process.env.OPENCODE_DISABLE_PROJECT_CONFIG, + modelsFetchDisabled: process.env.OPENCODE_DISABLE_MODELS_FETCH, + defaultPluginsDisabled: process.env.OPENCODE_DISABLE_DEFAULT_PLUGINS, }), ); @@ -400,8 +402,7 @@ const server = createServer(async (request, response) => { const promptPayload = JSON.parse(Buffer.concat(chunks).toString("utf8")); await appendFile(join(process.env.XDG_DATA_HOME, "fake-prompt-requests.ndjson"), JSON.stringify(promptPayload) + "\n"); if ( - promptPayload.providerID !== "openrouter" || - promptPayload.modelID !== "deepseek/deepseek-v4-flash-0731" || + `${promptPayload.providerID}/${promptPayload.modelID}` !== runtimeConfig.model || "model" in promptPayload ) { return json(response, 400, { @@ -427,11 +428,56 @@ const server = createServer(async (request, response) => { }, }, }); + emit({ + type: "message.part.delta", + properties: { + sessionID: session.id, + messageID: "message-late-source", + partID: "part-late-straggler", + field: "text", + delta: "late straggler text must not reach the next turn", + }, + }); } } setTimeout(async () => { await callFirstPaperclipTool(); const parsedPrompt = parsedPromptText(promptPayload); + if (String(parsedPrompt.message ?? "").includes("incremental-stream")) { + const messageID = `stream-message-${promptTurnSeq}`; + const part = { id: `stream-part-${promptTurnSeq}`, messageID, sessionID: session.id, type: "text", text: "" }; + const reasoning = { ...part, id: `${part.id}-reasoning`, type: "reasoning" }; + const update = (value) => emit({ type: "message.part.updated", properties: { sessionID: session.id, part: value } }); + const delta = (value, text, extra = {}) => emit({ type: "message.part.delta", properties: { sessionID: session.id, messageID, partID: value.id, field: "text", delta: text, ...extra } }); + const announceRole = () => emit({ type: "message.updated", properties: { sessionID: session.id, info: { id: messageID, role: "assistant" } } }); + const roleFirst = String(parsedPrompt.message).includes("role-first"); + const longResponse = String(parsedPrompt.message).includes("role-last-long"); + if (roleFirst) announceRole(); + // Parts and deltas can precede the message's role announcement. + update(part); + update(reasoning); + delta(reasoning, "Thinking"); + delta(part, "Hi "); + const emojiCount = longResponse ? 120 : 2; + for (let i = 0; i < emojiCount; i++) delta(part, "👋"); + delta(part, "wrong message", { messageID: "unrelated-message" }); + delta(part, "wrong session", { sessionID: "unrelated-session" }); + delta(part, "wrong field", { field: "output" }); + delta(part, "unknown part", { partID: "unknown-part" }); + const finalPart = { ...part, text: `Hi ${"👋".repeat(emojiCount)}`, time: { start: 1, end: 2 } }; + if (longResponse) update(finalPart); + if (!roleFirst) announceRole(); + // The consumer must observe a delta before the completed snapshot + // exists. This handshake makes the streaming regression deterministic. + for (let i = 0; i < 500; i++) { + if (await readFile(join(process.env.XDG_DATA_HOME, "release-stream"), "utf8").catch(() => "") === "ready") break; + await new Promise(resolve => setTimeout(resolve, 10)); + } + await writeFile(join(process.env.XDG_DATA_HOME, "stream-released"), "ready"); + update(finalPart); + emit({ type: "session.idle", properties: { sessionID: session.id } }); + return; + } if (String(parsedPrompt.message ?? "").includes("native-question")) { pendingQuestion = nativeQuestion(); emit({ diff --git a/server/src/__tests__/heartbeat-responsible-user-invariant.test.ts b/server/src/__tests__/heartbeat-responsible-user-invariant.test.ts index 59c3cb2c419..54bcebe2005 100644 --- a/server/src/__tests__/heartbeat-responsible-user-invariant.test.ts +++ b/server/src/__tests__/heartbeat-responsible-user-invariant.test.ts @@ -23,6 +23,7 @@ import { import { heartbeatService } from "../services/heartbeat.ts"; import { runningProcesses } from "../adapters/index.ts"; import { drainHeartbeatRunsToQuiescence } from "./helpers/drain-heartbeat-runs.js"; +import { createLifecycleDriver } from "../services/agent-lifecycle-driver.js"; const mockAdapterExecute = vi.hoisted(() => vi.fn(async () => ({ @@ -159,6 +160,27 @@ describeEmbeddedPostgres("heartbeat responsible-user invariant", () => { return { companyId, ownerUserId, agentId }; } + it("dispatches saved work on readiness with company scoping and normal admission checks", async () => { + const { companyId, agentId, ownerUserId } = await seedCompany(); + await db.update(agents).set({ status: "paused", lifecycleState: "verifying" }).where(eq(agents.id, agentId)); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "queued", + invocationSource: "on_demand", contextSnapshot: { responsibleUserId: ownerUserId } }).returning(); + const driver = createLifecycleDriver(db, {} as never); + const currentAgent = async () => (await db.select().from(agents).where(eq(agents.id, agentId)))[0]; + await driver.onReady!(await currentAgent()); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("queued"); + + await db.update(agents).set({ status: "idle", lifecycleState: "ready" }).where(eq(agents.id, agentId)); + await heartbeat.resumeQueuedRunsForAgent(randomUUID(), agentId); + expect(mockAdapterExecute).not.toHaveBeenCalled(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("queued"); + + await driver.onReady!(await currentAgent()); + expect(await waitForRun(db, run.id)).toMatchObject({ status: "succeeded", responsibleUserId: ownerUserId }); + expect(mockAdapterExecute).toHaveBeenCalledOnce(); + }); + it("dispatches an interrupted queue under the clicking operator through the real startup path", async () => { const { companyId, agentId, ownerUserId } = await seedCompany(); const operatorId = `operator-${randomUUID()}`, issueId = randomUUID(), commentId = randomUUID(), queueId = randomUUID(); diff --git a/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts b/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts index 4f7a8db7214..06b7e246d49 100644 --- a/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts @@ -13,14 +13,58 @@ function fixture() { const store = { get: vi.fn(), claim: vi.fn().mockResolvedValue(agent), renew: vi.fn().mockResolvedValue(true), setRequiredPlugins: vi.fn().mockResolvedValue(true), completeHost: vi.fn().mockResolvedValue(true), completePlugin: vi.fn().mockResolvedValue(true), defer: vi.fn().mockResolvedValue(true), - change: vi.fn().mockResolvedValue({ ...agent, lifecycleState: "ready" }), pending: vi.fn().mockResolvedValue([agent.id]) } satisfies LifecycleStore; + change: vi.fn().mockResolvedValue({ ...agent, lifecycleState: "ready", lifecycleVersion: 4, lifecycleOperation: null }), pending: vi.fn().mockResolvedValue([agent.id]) } satisfies LifecycleStore; const driver = { requiredPluginIds: vi.fn().mockResolvedValue(["plugin"]), runHost: vi.fn().mockResolvedValue("complete"), - runPlugin: vi.fn().mockResolvedValue("complete") } satisfies LifecycleDriver; + runPlugin: vi.fn().mockResolvedValue("complete"), onReady: vi.fn().mockResolvedValue(undefined) } satisfies LifecycleDriver; const report = vi.fn(); return { store, driver, report, worker: createLifecycleWorker(store, driver, report) }; } afterEach(() => { vi.restoreAllMocks(); vi.useRealTimers(); }); +describe("lifecycle readiness handoff", () => { + it("notifies the scheduler only after host, plugins, and readiness have committed", async () => { + const { store, driver, worker } = fixture(); + let commit!: (ready: LifecycleAgent) => void; + store.change.mockReturnValue(new Promise(resolve => { commit = resolve; })); + driver.onReady.mockImplementation(async ready => { + expect(store.completeHost).toHaveBeenCalledOnce(); + expect(store.completePlugin).toHaveBeenCalledOnce(); + expect(store.change).toHaveBeenCalledOnce(); + expect(ready).toMatchObject({ lifecycleState: "ready", lifecycleVersion: 4, lifecycleOperation: null }); + }); + const work = worker.process(agent.id); + await vi.waitFor(() => expect(store.change).toHaveBeenCalledOnce()); + expect(driver.onReady).not.toHaveBeenCalled(); + commit({ ...agent, lifecycleState: "ready", lifecycleVersion: 4, lifecycleOperation: null }); + await work; + expect(driver.onReady).toHaveBeenCalledOnce(); + await worker.stop(); + }); + + it.each(["host", "plugin", "transition", "hold"])("does not dispatch when %s prevents readiness", async stage => { + const { store, driver, worker } = fixture(); + if (stage === "host") driver.runHost.mockResolvedValue("pending"); + if (stage === "plugin") driver.runPlugin.mockResolvedValue("pending"); + if (stage === "transition") store.change.mockResolvedValue(null); + if (stage === "hold") store.change.mockResolvedValue({ ...agent, lifecycleState: "paused" }); + await worker.process(agent.id); + expect(driver.onReady).not.toHaveBeenCalled(); + await worker.stop(); + }); + + it("preserves committed readiness after a dispatch failure for the queue sweep to recover", async () => { + const { store, driver, report, worker } = fixture(); + const error = new Error("queue unavailable"); + driver.onReady.mockRejectedValue(error); + await worker.process(agent.id); + expect(report).toHaveBeenCalledWith(expect.objectContaining({ stage: "ready", phase: "ready", version: 4 }), error); + expect(store.defer).not.toHaveBeenCalled(); + expect(driver.runHost).toHaveBeenCalledOnce(); + expect(store.change).toHaveBeenCalledOnce(); + await worker.stop(); + }); +}); + describe("lifecycle failure diagnostics", () => { it.each(["host", "plugin", "claim", "renew", "scan", "defer"] as const)("reports %s failures with the available context", async stage => { const { store, driver, report, worker } = fixture(); diff --git a/server/src/modules/agent-lifecycle/application/ports.ts b/server/src/modules/agent-lifecycle/application/ports.ts index 1845ad8b7d1..11442b70b88 100644 --- a/server/src/modules/agent-lifecycle/application/ports.ts +++ b/server/src/modules/agent-lifecycle/application/ports.ts @@ -27,11 +27,13 @@ export interface LifecycleDriver { requiredPluginIds(agent: LifecycleAgent): Promise; runHost(agent: LifecycleAgent): Promise<"complete" | "pending">; runPlugin(agent: LifecycleAgent, pluginId: string): Promise<"complete" | "pending">; + /** Notify the scheduler only after readiness has committed. */ + onReady?(agent: LifecycleAgent): Promise; } export interface LifecycleFailure { stage: "claim" | "renew" | "select_plugins" | "save_plugins" | "host" | "plugin" - | "complete_host" | "complete_plugin" | "transition" | "defer" | "scan" | "policy_scan"; + | "complete_host" | "complete_plugin" | "transition" | "defer" | "scan" | "policy_scan" | "ready"; agentId?: string; companyId?: string; phase?: AgentLifecycleState; diff --git a/server/src/modules/agent-lifecycle/application/worker.ts b/server/src/modules/agent-lifecycle/application/worker.ts index ad1c143dd96..3b90b227673 100644 --- a/server/src/modules/agent-lifecycle/application/worker.ts +++ b/server/src/modules/agent-lifecycle/application/worker.ts @@ -56,6 +56,14 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr if (!await runStep("complete_plugin", () => store.completePlugin(agent, owner, pluginId), pluginId)) continue phase; } const changed = await runStep("transition", () => store.change(id, "complete", { version: agent.lifecycleVersion, owner })); + if (changed?.lifecycleState === "ready" && !stopped && canRun()) { + // Accepted messages may already be waiting for verification. Do not + // leave them until the periodic queue sweep. Readiness is committed; + // a dispatch failure must not replay setup or defer the old phase. + await attempt({ ...context, stage: "ready", phase: changed.lifecycleState, + version: changed.lifecycleVersion, operationId: changed.lifecycleOperation?.id }, + async () => { await driver.onReady?.(changed); }).catch(() => {}); + } if (changed && !isLifecycleWorkPending(changed.lifecycleState)) return; } catch { if (!await runStep("defer", () => store.defer(agent, owner, "The lifecycle step failed. Retry the operation.", new Date()))) continue; diff --git a/server/src/services/agent-lifecycle-driver.ts b/server/src/services/agent-lifecycle-driver.ts index 68cbea29999..11963d997fa 100644 --- a/server/src/services/agent-lifecycle-driver.ts +++ b/server/src/services/agent-lifecycle-driver.ts @@ -10,6 +10,9 @@ export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): Lif const { verify } = agentHarnessVerificationService(db, manager); const heartbeat = heartbeatService(db, { pluginWorkerManager: manager }); return { + async onReady(agent) { + await heartbeat.resumeQueuedRunsForAgent(agent.companyId, agent.id); + }, async requiredPluginIds(agent) { if (agent.lifecycleRequiredPluginIds !== null) return agent.lifecycleRequiredPluginIds; const installed = await db.select().from(plugins).where(inArray(plugins.status, ["ready", "error", "installed", "upgrade_pending"])); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index a549ee7ce42..961019eca34 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -8088,6 +8088,12 @@ export function heartbeatService( resumeQueuedRuns, + resumeQueuedRunsForAgent: async (companyId: string, agentId: string) => { + const agent = await getAgent(agentId); + if (!agent || agent.companyId !== companyId) return []; + return startNextQueuedRunForAgent(agentId); + }, + scheduleBoundedRetry, reconcileStrandedAssignedIssues, From 2a2071340d39e7ec9eeeec8615191df365b4dbac Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 10 Oct 2026 07:31:15 -0500 Subject: [PATCH 3/3] refactor(server): extract heartbeat runtime selection (#15746) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution prepares a runtime, dispatches work, and records the result. > - Earlier refactors separated workspace services, scheduling, and completion. > - The executor still builds native inputs and records runtime selection inline. > - This preparation must preserve admitted inputs and serialize with Stop. > - This pull request moves that phase into a typed runtime-selection module. > - Reviewers can inspect runtime preparation separately from dispatch and cleanup. ## Linked Issues or Issue Description Refs: #15732. **What existing behavior does this improve?** Maintainability of heartbeat runtime preparation and selection. **Subsystem affected** server/ — orchestration services. **Current behavior** The main heartbeat service has 8,128 lines. `executeRun` builds or restores native inputs, selects sessions and sandbox lifecycle, and records native or legacy selection inline. **Proposed behavior** Move this phase into `server/src/services/heartbeat/runtime-selection.ts`. The executor calls `selectHeartbeatRuntime` at the existing boundary and receives an explicit selection result. The main file becomes 7,275 lines, a reduction of 853 lines. **Reason and benefit** This gives runtime selection its own typed boundary and direct tests. It preserves existing decisions, database transactions, and cleanup ordering. **Breaking changes** None intended. All public heartbeat exports remain available. ## What Changed - Move native input assembly, completion contract selection, checkpoint lookup, runtime persistence, and native workspace-sync preparation into `selectHeartbeatRuntime`. - Preserve the row lock and controller lease check that serialize native selection with cancellation. - Report lifecycle and ownership changes to the executor immediately, including when a later preparation step throws. - Move sandbox lifecycle and model fallback helpers with compatible public re-exports. - Add 12 database-backed boundary tests. Update the existing session-recovery source test and developer guide. ## Verification - Structural AST comparison against current `master` confirms the 784-line selection block, moved helpers, and remaining service bodies are unchanged after normalizing boundary callbacks. All 140 public exports are preserved. - On final head `2eddaa485f`, all 12 new database-backed tests and 83 existing lifecycle, selection, fallback-warning, and session regression tests pass (95 total). - Five executor integration cases passed before the rebase. Their final-head rerun could not start because the local checkout was removed during verification; no assertions ran in that last attempt. - `pnpm -r typecheck` and `pnpm build` both pass on final head. - Local verification limitations: the full `pnpm test:run` was attempted, then stopped after database/process startup timeouts in concurrent local runs. Two unchanged optional real-runner session cases timed out on the initial run and one isolated retry; they were excluded from the 95-test focused run. CI also skips those two binary-dependent cases. - All CI gates pass on final head `2eddaa485f`: typecheck, build, general and serialized tests, runner checks, E2E, and canary dry run. - Greptile reviewed final head at 5/5 with no actionable findings. ## Risks This boundary controls native admission and cleanup ownership. Losing an update after a later exception could change resource cleanup. Immediate callbacks preserve those updates. Tests cover Stop races, lost controller leases, immutable admitted inputs, competing runtime selection, and workspace-sync failures. The change adds no schema or API behavior. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code editing, and command execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (95 focused tests on final head; broader local limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 10 + server/src/services/heartbeat.ts | 1003 ++-------------- .../heartbeat/runtime-selection.test.ts | 229 ++++ .../services/heartbeat/runtime-selection.ts | 1053 +++++++++++++++++ .../native-session-resume.test.ts | 2 +- 5 files changed, 1368 insertions(+), 929 deletions(-) create mode 100644 server/src/services/heartbeat/runtime-selection.test.ts create mode 100644 server/src/services/heartbeat/runtime-selection.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 26ad54fa183..1e9be2453b8 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1351,6 +1351,16 @@ trace finalization immediately so a later error cannot lose cleanup state. Existing public failure helpers remain re-exported by `heartbeat.ts`. Keep completion policy changes separate from this extraction and dispatch. +Runtime selection is in `server/src/services/heartbeat/runtime-selection.ts`. +`selectHeartbeatRuntime` constructs or restores native execution inputs, selects +completion contracts and sessions, and persists native or legacy runtime choices. +Native selection retains the row lock that serializes it with cancellation and +controller lease renewal. The executor receives an explicit selection result; +lifecycle and ownership callbacks update its teardown state immediately, including +when preparation later throws. Public sandbox lifecycle and model fallback helpers +remain re-exported by `heartbeat.ts`. Provider dispatch and final cleanup stay in +the executor. + ## Wake Context Delivery Built-in adapters deliver wake context through the run prompt, including structured diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 961019eca34..10cbb2b81fe 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { selectHeartbeatRuntime } from "./heartbeat/runtime-selection.js"; import { createHeartbeatRunCompletion } from "./heartbeat/run-completion.js"; export { MAX_TURN_CONTINUATION_WAKE_REASON, @@ -244,7 +245,6 @@ export { import { buildPaperclipTaskMarkdown } from "./heartbeat/task-markdown.js"; export { buildPaperclipTaskMarkdown } from "./heartbeat/task-markdown.js"; - import { recordLegacyWorkspaceRestoreFailure } from "./legacy-execution-recovery.js"; import { configuredEnvironmentProjection, @@ -259,7 +259,6 @@ import { prepareConnectionInstructionDelivery } from "./connection-instructions. import { resolveAssignedConnectionInstructionsForRun } from "./native-runtime/assigned-mcp-tools.js"; import { externalObjectService } from "./external-objects.js"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; -import { dotRunnerBroker } from "./dot-runner-broker.js"; import { startupCancellationFence, @@ -273,7 +272,6 @@ import { AgentDirectoryReuseInvalidatedError, isAgentDirectoryCopy } from "./age import type { PaperclipTurnContext } from "@paperclipai/adapter-utils/server-utils"; import { restoreNativeWorkspaceBestEffort } from "./native-runtime/native-workspace-best-effort.js"; - import { withNativeWorkspaceFinalizationOwnership, NativeWorkspaceFinalizationBusyError, @@ -306,7 +304,6 @@ import { import { legacyControllerBootId, - renewLegacyControllerLease, watchLegacyControllerLease, } from "./legacy-controller-lease.js"; @@ -333,9 +330,6 @@ import { claimedAdapterType, } from "./conversation-continuation.js"; -import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js"; - -import { buildNativeReviewRequest } from "./native-runtime/native-review-prompt.js"; import { settleInterruptedNativeBootstrap, } from "./legacy-execution-recovery.js"; @@ -350,8 +344,6 @@ export { buildHeartbeatRunStatusLiveEventPayload } from "./heartbeat-run-status- import { buildExecutionContinuation, StaleExecutionContinuationError } from "./execution-continuation.js"; import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils"; - - import { initializeRunIdentity, } from "./run-identity.js"; @@ -372,7 +364,9 @@ import fs from "node:fs/promises"; import path from "node:path"; import { execFile as execFileCallback } from "node:child_process"; import { promisify } from "node:util"; -import { createHash, randomUUID } from "node:crypto"; +import { + createHash, +} from "node:crypto"; import { and, asc, @@ -397,8 +391,6 @@ import { agentConfigRevisions, agentWakeupRequests, chatActions, - completionContracts, - heartbeatRunEvents, heartbeatRuns, environmentLeases, issueComments, @@ -429,50 +421,28 @@ import { publishLiveEvent } from "./live-events.js"; import { queuedCommentIdsFromRunContext, } from "./issue-queued-comment-queue.js"; -import { documentService } from "./documents.js"; + import { getTaskPlanContext } from "./task-plan-context.js"; -import { managedAgentProfileService } from "./managed-agent-profiles.js"; -import { remoteAgentProfileService } from "./remote-agent-profiles.js"; + import { buildNativeProviderEnvironment, - buildNativeExecutionInput, - buildNativeExecutionWithCheckpoint, - buildNativeRuntimeContext, claimWarmNativeInstructionCopy, nativeSessionWorkspaceScope, reserveWarmNativeInstructionDirectory, dispatchNativeSessionResumptions, - ensureNativeCompletionContract, executePaperclipNativeSession, finalizeNativeRun, - findNativeSessionResumeRun, - isNativeSessionId, - isUnusedNativeSessionBootstrap, - isUnusedLegacyNativeRetryReplacement, isRunnerIngressAuthorized, materializeLegacyQuestionResponseWakeProjection, - materializeNativeInteractionResponses, - nativeCompletionRequestsWithSources, - nativeCompletionSource, - nativeImmediateObjectiveSource, NativeCancellationPendingRecoveryError, NativeControllerDetachedForRestartError, - nativeToolContractFingerprintForTarget, - prepareNativeSessionBootstrapPersistence, prepareNativeWorkspaceSync, readNativeWorkspaceSyncReference, recordNativeFinalizationFailure, type NativeRestartRecoveryClaim, - rebindNativeSessionCheckpoint, resolveHeartbeatNativeRuntimeMode, } from "./native-runtime/index.js"; -import { - assertAgentCoreProfileRecoveryBinding, - assertManagedProfileRecoveryBinding, - projectPaperclipRunnerTaskConfig, - resolvePaperclipRunnerNativeProviderInput, -} from "./native-runtime/provider-profile.js"; -import { readRemoteCodexModelCliVersion } from "./native-runtime/codex-model-fallback.js"; + import { buildNativeHeartbeatPreparationSpans, buildNativeWakeIngressSpan, @@ -480,7 +450,6 @@ import { type NativeRunHistoricalSpan, } from "./native-runtime/native-run-trace.js"; import { - describeRunnerdNativeSessionBackend, parseNativeExecutionInput, type NativeExecutionInput, type NativeSessionBackend, @@ -508,7 +477,7 @@ import { import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY, } from "./native-runtime/external-chat-question-response.js"; -import { materializeExternalChatQuestionResponseInput } from "./native-runtime/external-chat-question-response-input.js"; + import { NativeRunnerOwnershipUnverifiedError, isNativeRunnerOwnershipHeld, @@ -545,10 +514,6 @@ import { buildHeartbeatRunIssueComment, } from "./heartbeat-run-summary.js"; -import { - CHAT_CONTROL_RECOVERY_ADMISSION_KEY, -} from "./chat-control-recovery-stop.js"; - import { logActivity, type LogActivityInput, @@ -583,7 +548,6 @@ import { settleLiveRunnerGoalBeforeInterrupt, } from "./runner-goals.js"; - import { parseIssueExecutionState, } from "./issue-execution-policy.js"; @@ -594,9 +558,7 @@ import { continuationSummaryParksExecutor, getIssueContinuationSummaryDocument, } from "./issue-continuation-summary.js"; -import { - buildPlanReviewContext, -} from "./plan-review-context.js"; + import { executionWorkspaceService, } from "./execution-workspaces.js"; @@ -669,7 +631,6 @@ import { type RunSnapshot as WakeQueueRunSnapshot, } from "../modules/wake-queue/index.js"; - import { taskWatchdogService } from "./task-watchdogs.js"; import { @@ -686,15 +647,11 @@ import { } from "../log-redaction.js"; import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js"; -import { - resolvePaperclipRunnerIdleTimeoutMs, -} from "@paperclipai/adapter-utils"; + import { readPaperclipSkillSyncPreference, - selectPaperclipTaskMarkdown, } from "@paperclipai/adapter-utils/server-utils"; - import { environmentService } from "./environments.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; @@ -835,64 +792,11 @@ export function getTaskDrainStatus(): { }; } -export interface NativeSandboxLifecycle { - runnerProcess: "per_turn" | "warm"; - sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; - failoverBackup: "verified"; -} +export { type NativeSandboxLifecycle } from "./heartbeat/runtime-selection.js"; -export function resolveReusableSandboxLifecycle(input: { - lifecyclePolicy: - | { mode: "per_turn"; idleTimeoutMs: null } - | { mode: "warm"; idleTimeoutMs: number }; - target: { - kind: "local" | "remote"; - transport?: string; - reusableLeaseConfigured?: boolean; - effectiveCapabilities?: { reusableLeases: boolean } | null; - } | null; -}): NativeSandboxLifecycle | null { - if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") { - return null; - } - const reusableLease = - input.target.reusableLeaseConfigured === true && - input.target.effectiveCapabilities?.reusableLeases === true; - if (input.lifecyclePolicy.mode === "warm" && !reusableLease) { - throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease"); - } - return { - runnerProcess: input.lifecyclePolicy.mode, - sandboxResource: - input.lifecyclePolicy.mode === "warm" - ? "keep_running" - : reusableLease - ? "stop_and_reuse" - : "destroy_after_turn", - failoverBackup: "verified", - }; -} +export { resolveReusableSandboxLifecycle } from "./heartbeat/runtime-selection.js"; -export function resolveNativeSandboxLifecycle(input: { - adapterType: string; - lifecyclePolicy: - | { mode: "per_turn"; idleTimeoutMs: null } - | { mode: "warm"; idleTimeoutMs: number }; - target: { - kind: "local" | "remote"; - transport?: string; - reusableLeaseConfigured?: boolean; - effectiveCapabilities?: { reusableLeases: boolean } | null; - } | null; -}): NativeSandboxLifecycle | null { - if ( - input.adapterType !== "paperclip_runner" || - input.target?.kind !== "remote" || - input.target.transport !== "sandbox" - ) - return null; - return resolveReusableSandboxLifecycle(input); -} +export { resolveNativeSandboxLifecycle } from "./heartbeat/runtime-selection.js"; interface ParsedIssueAssigneeAdapterOverrides { adapterConfig: Record | null; @@ -1111,38 +1015,7 @@ export function postWorkspaceReadyComment(input: { ); } -export async function postNativeModelFallbackWarning(input: { - issuesSvc: Pick, "addComment">; - onEvent: (event: AdapterRuntimeEvent) => Promise; - issueId: string; - runId: string; - requestedModel: string | null; - effectiveModel: string | null; - codexCliVersion: string; -}): Promise { - const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`; - await input.onEvent({ - eventType: "runner.model_fallback", - stream: "system", - level: "warn", - message, - payload: { - requestedModel: input.requestedModel, - effectiveModel: input.effectiveModel, - codexCliVersion: input.codexCliVersion, - }, - }); - await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, { - authorType: "system", - presentation: { - kind: "system_notice", - tone: "warning", - title: `Using ${input.effectiveModel}`, - density: "compact", - detailsDefaultOpen: false, - }, - }); -} +export { postNativeModelFallbackWarning } from "./heartbeat/runtime-selection.js"; function isTruthyRuntimeEnvValue(value: string | undefined) { return value === "true" || value === "1" || value === "yes" || value === "on"; @@ -5722,793 +5595,67 @@ export function heartbeatService( } } } - let nativeExecution: NativeExecutionInput | null = null; - let getNativeFreshSessionHandoff: (() => Promise) | undefined; - let nativeRunnerInstanceId: string | null = null; - if (nativeRuntimeResolution.kind === "native") { - if (!issueRef) { - throw new Error("native_runtime_ineligible: issue is required"); - } - const nativeExecutionWorkspaceId = - persistedExecutionWorkspace?.id ?? run.id; - const nativeReviewContext = readNativeReviewAssignmentContext(context); - const nativeReview = nativeReviewContext ? await getNativeReviewAssignment(db, { - companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, - contextSnapshot: nativeReviewContext, - }) : null; - if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available"); - const nativeReviewRequest = nativeReview - ? buildNativeReviewRequest({ - title: nativeReview.interaction.title, - summary: nativeReview.interaction.summary, - payload: nativeReview.interaction.payload, - }) - : null; - const persistedContract = run.completionContractId - ? await db - .select() - .from(completionContracts) - .where( - and( - eq(completionContracts.id, run.completionContractId), - eq(completionContracts.companyId, agent.companyId), - eq(completionContracts.issueId, issueRef.id), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null) - : null; - // Only a server-verified human resolution may supply a current answer - // reference. Tool/agent results and generated summaries stay evidence. - const currentHumanResponseId = !nativeReviewRequest - ? executionContinuation?.humanResponses?.find( - (response) => response.id === executionContinuation.trigger.interactionId, - )?.id - : undefined; - const immediateCompletion = (() => { - if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] }; - const { requests, sources } = nativeCompletionRequestsWithSources( - safeWakeComments.length > 0 - ? safeWakeComments - : safeWakeCommentContext?.body - ? [safeWakeCommentContext] - : [], - { - requiredFullWakeCommentCount: - paperclipWakePayload?.fallbackFetchNeeded === true && - CHAT_PROVIDERS.some( - (provider) => - provider === - paperclipWakePayload.externalChatProvider, - ) && - Array.isArray(paperclipWakePayload.commentIds) - ? paperclipWakePayload.commentIds.length - : undefined, - }, - ); - // Preserve every admitted pending chat request while also - // retaining newer user direction materialized by recovery. - // A file-only wake must not inherit an old task objective. - const latestComment = - executionContinuation?.messages.findLast( - (message) => - message.authorType === "user" && - !message.createdByRunId && - !message.deleted && - message.body.trim().length > 0, - ); - const latestRequest = latestComment?.body; - if ( - latestRequest && - !requests.some( - (request) => request === latestRequest.trim(), - ) - ) { - requests.push(latestRequest.trim()); - sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest)); - } - return { requests: requests.length > 0 ? requests : undefined, sources }; - })(); - // Rebuilding a default contract is not a change in user direction. - // In particular, an upgraded checkpoint may have an intentionally - // authored contract and no continuation envelope yet. - const completionContract = - persistedContract && persistedNativeExecutionInput - ? { - row: persistedContract, - contract: persistedContract.contractJson as never, - } - : await ensureNativeCompletionContract({ - db, - companyId: agent.companyId, - issue: issueRef, - actorId: agent.id, - immediateRequest: - nativeReviewRequest ?? (currentHumanResponseId - ? null - : executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null), - // The ordinary initial objective is selected by the server-owned - // continuation envelope. Carry its explicit description source - // through the singular-request compatibility path; do not infer - // provenance for review requests, answers, or wake fallbacks. - immediateRequestSource: nativeImmediateObjectiveSource({ - issueId: issueRef.id, - objectiveSource: executionContinuation?.objectiveSource, - excluded: Boolean(nativeReviewRequest || currentHumanResponseId), - }), - humanResponseId: currentHumanResponseId, - immediateRequests: immediateCompletion.requests, - immediateRequestSources: immediateCompletion.sources, - }); - const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString( - taskSessionDecodedParams?.sessionId, - ); - // Compatibility for native retry rows created before same-run restart - // recovery existed. Only an entirely unused replacement row may - // inherit its source checkpoint; any process/provider evidence on the - // replacement makes the ownership ambiguous and therefore ineligible. - const legacyRetrySource = - run.retryOfRunId && !isFailedChatRunRetry - ? await db - .select({ - id: heartbeatRuns.id, - companyId: heartbeatRuns.companyId, - agentId: heartbeatRuns.agentId, - runnerInstanceId: heartbeatRuns.runnerInstanceId, - nativeSessionId: heartbeatRuns.nativeSessionId, - runnerProfileJson: heartbeatRuns.runnerProfileJson, - runtimeMode: heartbeatRuns.runtimeMode, - status: heartbeatRuns.status, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, run.retryOfRunId), - eq(heartbeatRuns.companyId, agent.companyId), - eq(heartbeatRuns.agentId, agent.id), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null) - : null; - const nativeBootstrapHasProviderEvidence = - legacyRetrySource || - run.nativeSessionId || - persistedNativeExecutionInput - ? await db - .select({ id: heartbeatRunEvents.id }) - .from(heartbeatRunEvents) - .where( - and( - eq(heartbeatRunEvents.runId, run.id), - inArray(heartbeatRunEvents.eventType, [ - "harness.ready", - "session.started", - "session.resumed", - "session.updated", - "turn.started", - "provider.event", - "provider.rpc_result", - ]), - ), - ) - .limit(1) - .then((rows) => rows.length > 0) - : false; - const compatibleLegacyRetrySource = - !managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({ - replacement: run, - source: legacyRetrySource, - hasProviderEvents: nativeBootstrapHasProviderEvidence, - }) - ? legacyRetrySource - : null; - const legacyRetrySessionId = - compatibleLegacyRetrySource?.nativeSessionId; - const taskResumeRunId = - taskSessionForRun?.lastRunId && - taskSessionForRun.lastRunId !== run.id && - isNativeSessionId(taskNativeSessionId) - ? taskSessionForRun.lastRunId - : null; - const resumableTaskSessionId = isDotRun ? null : taskResumeRunId - ? taskNativeSessionId - : (legacyRetrySessionId ?? null); - const requestedNativeSessionId = - run.nativeSessionId ?? resumableTaskSessionId; - // A task-session lastRunId can lag a failed turn or point at an older - // normalized session. Find the newest exact-session authority instead. - // Rows that already acquired provider authority are progress barriers, - // even when they do not contain a usable checkpoint. - const previousNativeRun = - requestedNativeSessionId && - isUnusedNativeSessionBootstrap( - run, - nativeBootstrapHasProviderEvidence, - ) - ? await findNativeSessionResumeRun(db, { - companyId: agent.companyId, - agentId: agent.id, - issueId: issueRef.id, - normalizedSessionId: requestedNativeSessionId, - currentRunId: run.id, - beforeCreatedAt: run.createdAt, - }) - : null; - nativeRunnerInstanceId = - previousNativeRun?.runnerInstanceId && - previousNativeRun.nativeSessionId === - (run.nativeSessionId ?? resumableTaskSessionId) - ? previousNativeRun.runnerInstanceId - : (run.runnerInstanceId ?? randomUUID()); - let nativeSessionId = - run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID(); - let nativeResumeCheckpoint: ReturnType< - typeof rebindNativeSessionCheckpoint - > = null; - const agentLifecyclePolicy = - parseObject(agent.adapterConfig).lifecycleMode === "warm" - ? { - mode: "warm" as const, - idleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs( - parseObject(agent.adapterConfig).idleTimeoutMs, - ), - } - : { mode: "per_turn" as const, idleTimeoutMs: null }; - const environmentLifecyclePolicy = - executionTarget?.kind === "remote" && - executionTarget.transport === "sandbox" - ? (executionTarget.runnerLifecyclePolicy ?? null) - : null; - // Native Codex owns a durable, session-scoped home. It flushes refreshed - // auth into each invocation's private home before that home is removed. - // Other managed harnesses still require per-turn credential cleanup. - const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" && - nativeRuntimeResolution.profile.backend === "codex_app_server"; - const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ?? - (nativeRuntimeResolution.profile.backend === "openai_dot_mcp" || managedAiRuntime && !supportsManagedWarmSession - ? { mode: "per_turn" as const, idleTimeoutMs: null } - : environmentLifecyclePolicy ?? agentLifecyclePolicy); - if ( - effectiveLifecyclePolicy.mode === "warm" && - executionTarget?.kind === "remote" && - executionTarget.transport === "sandbox" && - (executionTarget.reusableLeaseConfigured !== true || - executionTarget.effectiveCapabilities?.reusableLeases !== true) - ) { - throw new Error("runner_warm_environment_requires_reusable_lease"); - } - const persistedProfile = persistedRunnerProfile; - if (persistedNativeExecutionInput) { - nativeExecution = persistedNativeExecutionInput; - if ( - nativeExecution.binding.companyId !== agent.companyId || - nativeExecution.binding.runId !== run.id || - nativeExecution.binding.issueId !== issueRef.id || - nativeExecution.binding.agentId !== agent.id || - nativeExecution.binding.executionWorkspaceId !== - nativeExecutionWorkspaceId || - nativeExecution.completionContract.id !== - completionContract.row.id || - nativeExecution.completionContract.sha256 !== - completionContract.row.canonicalSha256 - ) - throw new Error( - "native_execution_input_persisted_binding_mismatch", - ); - // Recover only the originally admitted request. A stored idle - // checkpoint can precede an already-started provider turn, so even - // apparent idleness is not authority to rewrite its contract. - // Newer user direction retains its separate durable wakeup cause. - // A failed pre-bootstrap attempt may have persisted its immutable - // input before discovering that lastRunId no longer names this - // session. Restore only an exactly compatible prior checkpoint; - // never rewrite the admitted input or skip current provider work. - if ( - previousNativeRun && - isUnusedNativeSessionBootstrap( - run, - nativeBootstrapHasProviderEvidence, - ) - ) { - nativeResumeCheckpoint = rebindNativeSessionCheckpoint({ - previousRun: previousNativeRun, - currentExecution: nativeExecution, - executionTargetKind: executionTarget?.kind ?? "local", - }); - } - if (nativeExecution.provider.kind === "claude_managed") { - const recoveryProfile = await managedAgentProfileService( - db, - ).requireQualified( - agent.companyId, - nativeExecution.provider.managedProfile.profileId, - ); - assertManagedProfileRecoveryBinding({ - adapterConfig: agent.adapterConfig, - snapshot: nativeExecution.provider.managedProfile, - stored: recoveryProfile, - }); - } - if (nativeExecution.provider.kind === "aws_agentcore") { - const recoveryProfile = await remoteAgentProfileService( - db, - ).requireQualified( - agent.companyId, - nativeExecution.provider.agentCoreProfile.profileId, - "aws_bedrock_agentcore_harness", - ); - assertAgentCoreProfileRecoveryBinding({ - snapshot: nativeExecution.provider.agentCoreProfile, - stored: recoveryProfile, - }); - } - } else { - const interactionId = readNonEmptyString(context.interactionId); - const interactionResponses = context[ - EXTERNAL_CHAT_QUESTION_RESPONSE_KEY - ] - ? await materializeExternalChatQuestionResponseInput({ - db, - binding: { - companyId: agent.companyId, - issueId: issueRef.id, - runId: run.id, - agentId: agent.id, - }, - contextSnapshot: context, - }) - : await materializeNativeInteractionResponses({ - db, - companyId: agent.companyId, - issueId: issueRef.id, - runId: run.id, - agentId: agent.id, - interactionIds: Array.isArray(context.interactionIds) - ? [ - ...new Set([ - ...(interactionId ? [interactionId] : []), - ...context.interactionIds.filter( - (id): id is string => typeof id === "string", - ), - ]), - ] - : interactionId - ? [interactionId] - : [], - }); - const runnerAdapterConfig = parseObject(agent.adapterConfig); - const managedProfile = - nativeRuntimeResolution.profile.backend === - "claude_managed_agents_api" - ? await managedAgentProfileService(db).requireQualified( - agent.companyId, - readNonEmptyString(runnerAdapterConfig.managedProfileId) ?? - "", - ) - : null; - const agentCoreProfile = - nativeRuntimeResolution.profile.backend === - "aws_agentcore_harness_api" - ? await remoteAgentProfileService(db).requireQualified( - agent.companyId, - readNonEmptyString( - runnerAdapterConfig.agentCoreProfileId, - ) ?? "", - "aws_bedrock_agentcore_harness", - ) - : null; - if (managedProfile) { - const rawApiKeyBinding = parseObject( - runnerAdapterConfig.env, - ).ANTHROPIC_API_KEY; - const boundSecretId = - typeof rawApiKeyBinding === "object" && - rawApiKeyBinding !== null - ? readNonEmptyString( - (rawApiKeyBinding as Record).secretId, - ) - : null; - if (boundSecretId !== managedProfile.apiKeySecretId) { - throw new ConfigurationIncompleteFailure( - "configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY", - { - configurationIncomplete: { - reason: "managed_agent_profile_secret_binding_mismatch", - companyId: agent.companyId, - agentId: agent.id, - profileId: managedProfile.id, - requiredEnvKeys: ["ANTHROPIC_API_KEY"], - }, - }, - ); - } - } - const executionMode = - issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake - ? ("plan" as const) - : ("default" as const); - const pinnedPlan = - executionMode === "plan" - ? await documentService(db).getIssueDocumentByKey( - issueRef.id, - "plan", - ) - : null; - const pinnedReviewContext = - executionMode === "plan" - ? await buildPlanReviewContext({ - db, - companyId: agent.companyId, - issueId: issueRef.id, - issueWorkMode: issueRef.workMode, - interactionId: readNonEmptyString(context.interactionId), - }) - : null; - const pinnedPlanMarkdown = pinnedPlan?.body ?? ""; - const dotBinding = nativeRuntimeResolution.profile.backend === "openai_dot_mcp" - ? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).dotBindingId ?? "")) : undefined; - const nativeRuntimeContext = await buildNativeRuntimeContext({ - db, - agent, - runId: run.id, - runtimeConfig, - runtimeSkillEntries, - instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined, - }); - getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff; - const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "codex_app_server" - || nativeRuntimeResolution.profile.backend === "opencode_server" - ? projectPaperclipRunnerTaskConfig( - nativeRuntimeResolution.profile.backend, - agent.adapterConfig, - issueAssigneeOverrides?.adapterConfig, - managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined, - ) - : agent.adapterConfig; - const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({ - backend: nativeRuntimeResolution.profile.backend, - adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, - }); - const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex" - ? await readRemoteCodexModelCliVersion({ - model: requestedNativeProvider.model, - target: executionTarget, - remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH, - remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC, - }) : null; - const nativeProvider = codexCliVersion - ? resolvePaperclipRunnerNativeProviderInput({ - backend: nativeRuntimeResolution.profile.backend, - adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, - }) : requestedNativeProvider; - if (nativeProvider.model !== requestedNativeProvider.model) { - await postNativeModelFallbackWarning({ - issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id, - requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model, - codexCliVersion: codexCliVersion!, - }); - } - const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) => - buildNativeExecutionInput({ - agentKeyId: agentIdentity?.keyId, - companyId: agent.companyId, - runId: run.id, - issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef, - taskPrompt: [ - nativeReviewRequest ?? readNonEmptyString( - selectPaperclipTaskMarkdown(context, { - resumedSession: false, - includeCommunicationGuidance: false, - }), - ) ?? - `# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`, - nativeRuntimeResolution.profile.backend !== "openai_dot_mcp" && projectRepositoryPaths.length > 0 - ? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}` - : null, - ].filter(Boolean).join("\n\n"), - initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance), - wakePayload: context.paperclipWake, - turnContext: context.paperclipTurnContext, - githubInstructionSkillKeys: runScopedMentionedSkillKeys, - resumedSession, - previousTurn: (() => { - if (!previousNativeRun || nativeReviewRequest) return null; - try { - const previousTask = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput).task; - if (paperclipWakePayload?.externalChatProvider) { - // External native inputs use a neutral task title. Compare - // the saved canonical brief so old provider text is not - // repeated as a change, while genuine edits still arrive. - const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue); - if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" || - (savedIssue.description !== null && typeof savedIssue.description !== "string")) return null; - return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description } }; - } - return { runId: previousNativeRun.id, task: previousTask }; - } catch { - // An invalid prior snapshot must use the fresh bootstrap. - return null; - } - })(), - conversationMode: context.conversationMode === true, - agentId: agent.id, - workspace: { - // Projectless paperclip_runner tasks still have a resolved local cwd. Bind that - // transient workspace to the run id so the native input remains durable and replayable - // without fabricating a project-scoped execution_workspaces row. - id: nativeExecutionWorkspaceId, - cwd: executionWorkspace.cwd, - repoUrl: executionWorkspace.repoUrl, - repoRef: executionWorkspace.repoRef, - branchName: executionWorkspace.branchName, - }, - normalizedSessionId, - executionMode, - planningContext: - executionMode === "plan" - ? { - documentId: pinnedPlan?.id ?? null, - baseRevisionId: - pinnedPlan?.latestRevisionId ?? null, - baseRevisionNumber: - pinnedPlan?.latestRevisionNumber ?? 0, - markdown: pinnedPlanMarkdown, - sha256: createHash("sha256") - .update(pinnedPlanMarkdown) - .digest("hex"), - reviewContext: pinnedReviewContext - ? (structuredClone( - pinnedReviewContext, - ) as unknown as Record) - : {}, - } - : null, - ...nativeProvider, - lifecyclePolicy: effectiveLifecyclePolicy, - interactionResponses, - completionContract: { - id: completionContract.row.id, - sha256: completionContract.row.canonicalSha256, - schemaVersion: completionContract.row.schemaVersion, - contract: completionContract.contract, - sources: "sources" in completionContract ? completionContract.sources : undefined, - }, - runtimeContext: nativeRuntimeContext, - }); - const backendDescriptor = await describeRunnerdNativeSessionBackend(buildExecution({ - normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null, - })); - const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({ - previousRun: previousNativeRun, - normalizedSessionId: nativeSessionId, - executionTargetKind: executionTarget?.kind ?? "local", - toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true, - refreshTools: context.refreshTools === true, - buildExecution, - }); - nativeExecution = nativeExecutionWithCheckpoint.execution; - nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint; - if ( - nativeSessionId !== - nativeExecutionWithCheckpoint.normalizedSessionId - ) { - nativeRunnerInstanceId = randomUUID(); - } - nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId; - } - const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({ - adapterType: agent.adapterType, - lifecyclePolicy: nativeExecution.session.lifecyclePolicy, - target: executionTarget, - }); - if (nativeSandboxLifecycle) { - nativeLifecycleTelemetryForRun = { - provider: nativeExecution.provider.kind, - harness: nativeExecution.session.driverKind, - lifecycleMode: nativeExecution.session.lifecyclePolicy.mode, - sandboxResource: nativeSandboxLifecycle.sandboxResource, - }; - const selectedLifecycleSpan = getStartupTracer( - "paperclip.environment-lifecycle", - ).startSpan("sandbox.lifecycle.selected", { - attributes: { - "paperclip.native.span.provider": nativeExecution.provider.kind, - "paperclip.native.span.harness": - nativeExecution.session.driverKind, - "paperclip.native.span.lifecycle_mode": - nativeExecution.session.lifecyclePolicy.mode, - "paperclip.native.span.sandbox_resource": - nativeSandboxLifecycle.sandboxResource, - "paperclip.native.span.outcome": "selected", - "paperclip.native.span.bytes_transferred": 0, - }, - }); - selectedLifecycleSpan.end(); - } - providerResourceDispositionForRun = - nativeSandboxLifecycle?.sandboxResource === "keep_running" - ? "keep_running" - : nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse" - ? "stop_and_retain" - : nativeSandboxLifecycle?.sandboxResource === - "destroy_after_turn" - ? "destroy" - : undefined; - await options.beforeNativeRuntimeSelection?.(run.id); - const nativeSelected = await db.transaction(async (tx) => { - const lockedRun = await tx - .select() - .from(heartbeatRuns) - .where(eq(heartbeatRuns.id, run.id)) - .for("update") - .limit(1) - .then((rows) => rows[0] ?? null); - if (!lockedRun) throw new Error("native_runtime_run_missing"); - // Cancellation and runtime selection serialize on this row. A - // stopped preparation must never create a new native coordinator. - if (lockedRun.status !== "running" || lockedRun.resultJson?.startupCancellation) return false; - if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId && - !(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) { - nativeOwnershipHeld = true; - return false; - } - if ( - lockedRun.runtimeModeResolvedAt && - lockedRun.runtimeMode !== "native" - ) { - throw new Error("native_runtime_mode_conflict"); - } - const lockedProfile = parseObject(lockedRun.runnerProfileJson); - const persistedNativeSessionId = - await prepareNativeSessionBootstrapPersistence(tx, { - run: lockedRun, - selectedSessionId: nativeSessionId, - execution: nativeExecution!, - restoringCheckpoint: nativeResumeCheckpoint !== null, - }); - await tx - .update(heartbeatRuns) - .set({ - runtimeMode: "native", - runtimeModeResolverVersion: - lockedRun.runtimeModeResolverVersion ?? - nativeRuntimeResolution.resolverVersion, - runtimeModeReason: - lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason, - runtimeModeResolvedAt: - lockedRun.runtimeModeResolvedAt ?? new Date(), - runnerProfileJson: { - ...nativeRuntimeResolution.profile, - ...lockedProfile, - ...(lockedProfile.nativeExecutionInput - ? {} - : { recoveryEventInventoryVersion: 1 }), - ...(providerTraceRequested - ? { - providerTrace: { - mode: "raw", - traceId: providerTraceCapture?.metadata.id ?? null, - maxBytes: PROVIDER_TRACE_MAX_BYTES, - }, - } - : {}), - nativeExecutionInput: - lockedProfile.nativeExecutionInput ?? nativeExecution, - nativeToolContractFingerprint: - nativeToolContractFingerprintForTarget( - executionTarget?.kind ?? "local", - ), - ...(lockedProfile.sessionCheckpoint != null - ? { sessionCheckpoint: lockedProfile.sessionCheckpoint } - : nativeResumeCheckpoint - ? { - sessionCheckpoint: - nativeResumeCheckpoint as unknown as Record< - string, - unknown - >, - } - : {}), - }, - runnerInstanceId: - previousNativeRun?.runnerInstanceId && - persistedNativeSessionId === previousNativeRun.nativeSessionId - ? previousNativeRun.runnerInstanceId - : lockedRun.nativeSessionId !== persistedNativeSessionId - ? nativeRunnerInstanceId - : (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId), - nativeSessionId: persistedNativeSessionId, - processPid: - lockedRun.processPid ?? - (previousNativeRun?.nativeSessionId === nativeSessionId - ? previousNativeRun.processPid - : null), - processGroupId: - lockedRun.processGroupId ?? - (previousNativeRun?.nativeSessionId === nativeSessionId - ? previousNativeRun.processGroupId - : null), - processStartedAt: - lockedRun.processStartedAt ?? - (previousNativeRun?.nativeSessionId === nativeSessionId - ? previousNativeRun.processStartedAt - : null), - nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id, - driverKind: - lockedRun.driverKind ?? - nativeExecution?.session.driverKind ?? - "codex_app_server", - driverVersion: lockedRun.driverVersion ?? "phase6-v1", - completionContractId: - lockedRun.completionContractId ?? completionContract.row.id, - completionContractSha256: - lockedRun.completionContractSha256 ?? - completionContract.row.canonicalSha256, - nativePhase: lockedRun.nativePhase ?? "observed", - nativePhaseUpdatedAt: - lockedRun.nativePhaseUpdatedAt ?? new Date(), - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, run.id)); - await tx - .insert(nativeRunFinalizations) - .values({ - runId: run.id, - companyId: agent.companyId, - issueId: issueRef.id, - phase: "observed", - }) - .onConflictDoNothing(); - return true; - }); - if (!nativeSelected) return; - controllerLease.stop(); - nativeWorkspaceSync = await prepareNativeWorkspaceSync({ - db, - runId: run.id, - companyId: agent.companyId, - workspaceId: nativeExecutionWorkspaceId, - workspaceLocalDir: executionWorkspace.cwd, - target: executionTarget, - lease: activeEnvironmentLease.lease, - restartRecovery: runOptions.nativeRestartRecovery, - sameRunRecovery: Boolean(runOptions.nativeLeaseOwner), - resourceDisposition: providerResourceDispositionForRun, - }); - } else { - const legacyWarmLifecycle = - executionTarget?.kind === "remote" && - executionTarget.transport === "sandbox" && - executionTarget.runnerLifecyclePolicy?.mode === "warm" - ? resolveReusableSandboxLifecycle({ - lifecyclePolicy: executionTarget.runnerLifecyclePolicy, - target: executionTarget, - }) - : null; - if (legacyWarmLifecycle?.sandboxResource === "keep_running") { - providerResourceDispositionForRun = "keep_running"; - } - await db - .update(heartbeatRuns) - .set({ - runtimeMode: "legacy", - runtimeModeResolverVersion: - nativeRuntimeResolution.resolverVersion, - runtimeModeReason: nativeRuntimeResolution.reason, - runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(), - // Preserve server-owned admission and dispatch evidence on this - // row; never copy another run's execution profile. - runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY} - then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}) - else '{}'::jsonb end) - || (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch' - then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch') - else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, run.id)); - } + const runtimeSelection = await selectHeartbeatRuntime(db, { + run, + agent, + nativeRuntimeResolution, + task: { + issueRef, + issueContext, + context, + executionContinuation, + safeWakeComments, + safeWakeCommentContext, + paperclipWakePayload, + acceptedPlanContinuationWake, + }, + session: { + persistedNativeExecutionInput, + persistedRunnerProfile, + taskSessionCredentialCompatible, + taskSessionDecodedParams, + taskSessionForRun, + isFailedChatRunRetry, + getFreshSessionHandoff, + }, + workspace: { + persistedExecutionWorkspace, + executionWorkspace, + executionTarget, + activeEnvironmentLease, + isDotRun, + projectRepositoryPaths, + }, + config: { + runtimeConfig, + runtimeSkillEntries, + runScopedMentionedSkillKeys, + instructionCopy, + agentIdentity, + issueAssigneeOverrides, + resolvedConfig, + managedAiRuntime, + }, + recovery: runOptions, + trace: { providerTraceRequested, providerTraceCapture }, + issuesSvc, + runtimeEnv, + beforeNativeRuntimeSelection: options.beforeNativeRuntimeSelection, + onAdapterEvent, + onNativeLifecycleSelected: (telemetry) => { + nativeLifecycleTelemetryForRun = telemetry; + }, + onProviderResourceDisposition: (disposition) => { + providerResourceDispositionForRun = disposition; + }, + onNativeOwnershipHeld: () => { + nativeOwnershipHeld = true; + }, + stopControllerLease: () => controllerLease.stop(), + }); + if (!runtimeSelection.selected) return; + const { nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff } = runtimeSelection; + nativeWorkspaceSync = runtimeSelection.nativeWorkspaceSync; const localAgentJwtScope = issueRef?.workMode === "skill_test" ? { kind: "skill_test" as const, issueId: issueRef.id } diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts new file mode 100644 index 00000000000..dbd7bbcb878 --- /dev/null +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -0,0 +1,229 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { agents, companies, completionContracts, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js"; +import { buildNativeRuntimeContext } from "../native-runtime/runtime-context.js"; +import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js"; +import { prepareNativeWorkspaceSync } from "../native-runtime/native-workspace-sync.js"; +import { resolveHeartbeatNativeRuntimeMode } from "../native-runtime/runtime-mode.js"; +import { canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; +import { selectHeartbeatRuntime, type HeartbeatRuntimeSelectionInput } from "./runtime-selection.js"; + +// Keep the real contract, session, and runtime-selection transactions. Replace only +// filesystem materialization and sandbox synchronization at their external boundaries. +vi.mock("../native-runtime/runtime-context.js", async (original) => ({ + ...await original(), + buildNativeRuntimeContext: vi.fn(async () => nativeRuntimeContextFixture()), +})); +vi.mock("../native-runtime/native-workspace-sync.js", async (original) => ({ + ...await original(), + prepareNativeWorkspaceSync: vi.fn(async () => null), +})); + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () => { + let database: Awaited>; + let db: Db; + let instructionRoot: string; + beforeAll(async () => { + instructionRoot = await mkdtemp(join(tmpdir(), "paperclip-selection-instructions-")); + await writeFile(join(instructionRoot, "AGENTS.md"), "Complete the assigned task.\n"); + database = await startEmbeddedPostgresTestDatabase("paperclip-runtime-selection"); + db = createDb(database.connectionString); + await db.execute(sql`set client_min_messages = warning`); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); if (instructionRoot) await rm(instructionRoot, { recursive: true, force: true }); }); + beforeEach(async () => { + await db.execute(sql`truncate table companies restart identity cascade`); + vi.clearAllMocks(); + vi.mocked(prepareNativeWorkspaceSync).mockResolvedValue(null); + vi.mocked(buildNativeRuntimeContext).mockImplementation(async () => { + const context = nativeRuntimeContextFixture(); + context.instructions.bundle.rootPath = instructionRoot; + return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) }; + }); + }); + + async function fixture(adapterType = "paperclip_runner") { + const [company] = await db.insert(companies).values({ name: "Runtime selection", issuePrefix: "SEL" }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Selecting agent", status: "running", adapterType, adapterConfig: { provider: "codex", model: "gpt-5.6-luna" } }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Complete this task", description: "Preserve the requested work", assigneeAgentId: agent.id }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, issueId: issue.id, scopeKind: "issue", status: "running" }).returning(); + const now = new Date(); + const input: HeartbeatRuntimeSelectionInput = { + run, agent, + nativeRuntimeResolution: resolveHeartbeatNativeRuntimeMode({ persisted: run, enabled: true, runtimeConfig: {}, adapterConfig: agent.adapterConfig, agent, issue, target: null, workspaceId: null }), + task: { issueRef: issue, issueContext: null, context: { paperclipTaskMarkdown: "Complete this task" }, executionContinuation: null, safeWakeComments: [], safeWakeCommentContext: null, paperclipWakePayload: null, acceptedPlanContinuationWake: false }, + session: { persistedNativeExecutionInput: null, persistedRunnerProfile: {}, taskSessionCredentialCompatible: true, taskSessionDecodedParams: null, taskSessionForRun: null, isFailedChatRunRetry: false, getFreshSessionHandoff: vi.fn(async () => "Earlier context") }, + workspace: { persistedExecutionWorkspace: null, executionWorkspace: { cwd: "/tmp/paperclip-runtime-selection", repoUrl: null, repoRef: null, branchName: null }, executionTarget: null, isDotRun: false, projectRepositoryPaths: [], + activeEnvironmentLease: { lease: { id: randomUUID(), companyId: company.id, environmentId: null, executionWorkspaceId: null, issueId: issue.id, heartbeatRunId: run.id, status: "active", leasePolicy: "ephemeral", provider: null, providerLeaseId: null, acquiredAt: now, lastUsedAt: now, expiresAt: null, releasedAt: null, failureReason: null, cleanupStatus: null, metadata: null, createdAt: now, updatedAt: now } }, + }, + config: { runtimeConfig: {}, runtimeSkillEntries: [], runScopedMentionedSkillKeys: [], instructionCopy: null, agentIdentity: undefined, issueAssigneeOverrides: null, resolvedConfig: {}, managedAiRuntime: undefined }, + recovery: {}, trace: { providerTraceRequested: false, providerTraceCapture: null }, + issuesSvc: { addComment: vi.fn() }, runtimeEnv: {}, onAdapterEvent: vi.fn(async () => {}), + onNativeLifecycleSelected: vi.fn(), onProviderResourceDisposition: vi.fn(), onNativeOwnershipHeld: vi.fn(), stopControllerLease: vi.fn(), + }; + return { input, issue, run, agent }; + } + const read = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]!); + const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id)); + const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false }; + + it("persists the native input, completion contract, and coordinator before transferring cleanup ownership", async () => { + const { input, run } = await fixture(); + vi.mocked(prepareNativeWorkspaceSync).mockImplementationOnce(async args => { + expect(await read(run.id)).toMatchObject({ runtimeMode: "native", nativeIssueId: input.task.issueRef!.id }); + expect(await coordinators(run.id)).toHaveLength(1); + expect(input.stopControllerLease).toHaveBeenCalledOnce(); + expect(args).toMatchObject({ runId: run.id, workspaceId: run.id, workspaceLocalDir: input.workspace.executionWorkspace.cwd }); + return null; + }); + const selected = await selectHeartbeatRuntime(db, input); + expect(selected.selected).toBe(true); + if (!selected.selected) throw new Error("Expected selection"); + const saved = await read(run.id); + expect(saved).toMatchObject({ runtimeMode: "native", driverKind: "codex_app_server", nativePhase: "observed", nativeSessionId: expect.any(String), runnerInstanceId: selected.nativeRunnerInstanceId, + runnerProfileJson: { nativeExecutionInput: selected.nativeExecution, recoveryEventInventoryVersion: 1 } }); + expect(await db.select().from(completionContracts)).toHaveLength(1); + expect(selected.getNativeFreshSessionHandoff).toBe(input.session.getFreshSessionHandoff); + expect(input.onNativeOwnershipHeld).not.toHaveBeenCalled(); + }); + + it("persists configured GitHub instruction skills without admitting provider-mentioned skills", async () => { + const { input, issue, run } = await fixture(); + input.config.runScopedMentionedSkillKeys = ["company/test/configured"]; + input.task.context.paperclipWake = { + reason: "External chat message received", externalChatProvider: "github", checkedOutByHarness: true, + issue: { id: issue.id, workMode: "standard" }, + comments: [{ id: "review-comment", body: "Please use /provider-mentioned" }], + commentIds: ["review-comment"], latestCommentId: "review-comment", + commentWindow: { requestedCount: 1, includedCount: 1, missingCount: 0 }, fallbackFetchNeeded: false, + }; + vi.mocked(buildNativeRuntimeContext).mockImplementationOnce(async () => { + const context = nativeRuntimeContextFixture(); + context.instructions.bundle.rootPath = instructionRoot; + context.skills = ["configured", "provider-mentioned"].map(name => ({ + key: `company/test/${name}`, runtimeName: name, versionId: null, + bundle: { ...context.instructions.bundle }, + })); + return { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) }; + }); + expect(await selectHeartbeatRuntime(db, input)).toMatchObject({ + selected: true, nativeExecution: { task: { description: "Configured GitHub instruction skills:\n/configured" } }, + }); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ + nativeExecutionInput: { task: { description: "Configured GitHub instruction skills:\n/configured" } }, + }); + }); + + it("restores the immutable admitted input and saved checkpoint when newer context arrives", async () => { + const { input, run } = await fixture(); + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + const saved = await read(run.id); + const profile = { ...saved.runnerProfileJson, sessionCheckpoint: { retained: "checkpoint" } }; + await db.update(heartbeatRuns).set({ runnerProfileJson: profile }).where(eq(heartbeatRuns.id, run.id)); + input.run = await read(run.id); + input.session.persistedRunnerProfile = profile; + input.session.persistedNativeExecutionInput = selected.nativeExecution; + input.task.context.paperclipTaskMarkdown = "A newer unrelated request"; + input.task.safeWakeComments = [{ id: randomUUID(), body: "New direction belongs to a separate wake" }]; + vi.mocked(buildNativeRuntimeContext).mockClear(); + const resumed = await selectHeartbeatRuntime(db, input); + expect(resumed).toMatchObject({ selected: true, nativeExecution: selected.nativeExecution, nativeRunnerInstanceId: selected.nativeRunnerInstanceId }); + expect((await read(run.id)).runnerProfileJson).toMatchObject(profile); + expect(await db.select().from(completionContracts)).toHaveLength(1); + expect(await coordinators(run.id)).toHaveLength(1); + expect(buildNativeRuntimeContext).not.toHaveBeenCalled(); + }); + + it("rejects an admitted input bound to another run without replacing the saved runtime", async () => { + const { input, run } = await fixture(); + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected native selection"); + const saved = await read(run.id); + input.run = saved; + input.session.persistedNativeExecutionInput = { ...selected.nativeExecution, binding: { ...selected.nativeExecution.binding, runId: randomUUID() } }; + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_execution_input_persisted_binding_mismatch"); + expect((await read(run.id)).runnerProfileJson).toEqual(saved.runnerProfileJson); + }); + + it.each(["cancelled", "startup_cancellation"])("does not create a native coordinator when %s wins preparation", async winner => { + const { input, run } = await fixture(); + input.beforeNativeRuntimeSelection = async () => { + await db.update(heartbeatRuns).set(winner === "cancelled" ? { status: "cancelled" } : { resultJson: { startupCancellation: { beforeNativeSelection: true } } }).where(eq(heartbeatRuns.id, run.id)); + }; + expect(await selectHeartbeatRuntime(db, input)).toEqual({ selected: false }); + expect(await read(run.id)).toMatchObject({ runtimeMode: "legacy", nativeSessionId: null, completionContractId: null }); + expect(await coordinators(run.id)).toHaveLength(0); + expect(input.stopControllerLease).not.toHaveBeenCalled(); + expect(prepareNativeWorkspaceSync).not.toHaveBeenCalled(); + }); + + it("reports a lost controller lease before returning without native selection", async () => { + const { input, run } = await fixture(); + await db.update(heartbeatRuns).set({ controllerBootId: randomUUID(), controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, run.id)); + expect(await selectHeartbeatRuntime(db, input)).toEqual({ selected: false }); + expect(input.onNativeOwnershipHeld).toHaveBeenCalledOnce(); + expect(input.stopControllerLease).not.toHaveBeenCalled(); + expect(await coordinators(run.id)).toHaveLength(0); + }); + + it("rejects a competing persisted legacy selection atomically", async () => { + const { input, run } = await fixture(); + input.beforeNativeRuntimeSelection = async () => { + await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date(), runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, run.id)); + }; + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("native_runtime_mode_conflict"); + expect((await read(run.id)).runtimeMode).toBe("legacy"); + expect(await coordinators(run.id)).toHaveLength(0); + expect(input.stopControllerLease).not.toHaveBeenCalled(); + }); + + it("reports sandbox teardown policy even when a later preparation step throws", async () => { + const { input, run } = await fixture(); + input.workspace.executionTarget = sandbox; + input.beforeNativeRuntimeSelection = async () => { throw new Error("selection interrupted"); }; + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("selection interrupted"); + expect(input.onNativeLifecycleSelected).toHaveBeenCalledWith({ provider: "codex", harness: "codex_app_server", lifecycleMode: "per_turn", sandboxResource: "destroy_after_turn" }); + expect(input.onProviderResourceDisposition).toHaveBeenCalledWith("destroy"); + expect(input.stopControllerLease).not.toHaveBeenCalled(); + expect(await coordinators(run.id)).toHaveLength(0); + }); + + it("retains native admission and teardown state if workspace sync preparation fails", async () => { + const { input, run } = await fixture(); + input.workspace.executionTarget = sandbox; + vi.mocked(prepareNativeWorkspaceSync).mockRejectedValueOnce(new Error("sync preparation failed")); + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("sync preparation failed"); + expect(input.onProviderResourceDisposition).toHaveBeenCalledWith("destroy"); + expect(input.stopControllerLease).toHaveBeenCalledOnce(); + expect((await read(run.id)).runtimeMode).toBe("native"); + expect(await coordinators(run.id)).toHaveLength(1); + }); + + it("rejects a warm native sandbox that cannot reuse its lease", async () => { + const { input, run } = await fixture(); + input.workspace.executionTarget = { ...sandbox, runnerLifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } }; + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("runner_warm_environment_requires_reusable_lease"); + expect(input.onNativeLifecycleSelected).not.toHaveBeenCalled(); + expect(await coordinators(run.id)).toHaveLength(0); + }); + + it("preserves only server-owned admission and dispatch evidence when recording legacy selection", async () => { + const { input, run } = await fixture("process"); + const admission = { version: 1, kind: "fixture" }, dispatch = { entered: true }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: admission, adapterDispatch: dispatch, unrelated: "drop" } }).where(eq(heartbeatRuns.id, run.id)); + input.trace = { providerTraceRequested: true, providerTraceCapture: { metadata: { id: "trace-id" } } }; + expect(await selectHeartbeatRuntime(db, input)).toMatchObject({ selected: true, nativeExecution: null, nativeWorkspaceSync: null }); + expect((await read(run.id)).runnerProfileJson).toEqual({ [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: admission, adapterDispatch: dispatch, providerTrace: { mode: "raw", traceId: "trace-id", maxBytes: 64 * 1024 * 1024 } }); + expect(await coordinators(run.id)).toHaveLength(0); + expect(input.stopControllerLease).not.toHaveBeenCalled(); + expect(buildNativeRuntimeContext).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts new file mode 100644 index 00000000000..efe4bbe63e7 --- /dev/null +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -0,0 +1,1053 @@ +import { + ConfigurationIncompleteFailure, + type buildPaperclipWakePayload, + type createHeartbeatRunPreparation, +} from "./run-preparation.js"; +import { dotRunnerBroker } from "../dot-runner-broker.js"; +import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js"; +import { isConversation } from "../agent-conversations.js"; +import { renewLegacyControllerLease } from "../legacy-controller-lease.js"; +import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js"; +import { + getNativeReviewAssignment, + readNativeReviewAssignmentContext, +} from "../native-runtime/native-review-participant.js"; +import { buildNativeReviewRequest } from "../native-runtime/native-review-prompt.js"; +import type { buildExecutionContinuation } from "../execution-continuation.js"; +import type { agentInstructionWorkingCopyService } from "../agent-instruction-working-copies.js"; +import { createHash, randomUUID } from "node:crypto"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { CHAT_PROVIDERS } from "@paperclipai/shared"; +import { + type agents, + completionContracts, + heartbeatRunEvents, + heartbeatRuns, + nativeRunFinalizations, +} from "@paperclipai/db"; +import { getStartupTracer } from "../../instrumentation.js"; +import { documentService } from "../documents.js"; +import { managedAgentProfileService } from "../managed-agent-profiles.js"; +import { remoteAgentProfileService } from "../remote-agent-profiles.js"; +import { + buildNativeExecutionInput, + buildNativeExecutionWithCheckpoint, + buildNativeRuntimeContext, + ensureNativeCompletionContract, + findNativeSessionResumeRun, + isNativeSessionId, + isUnusedNativeSessionBootstrap, + isUnusedLegacyNativeRetryReplacement, + materializeNativeInteractionResponses, + nativeCompletionRequestsWithSources, + nativeCompletionSource, + nativeImmediateObjectiveSource, + nativeToolContractFingerprintForTarget, + prepareNativeSessionBootstrapPersistence, + prepareNativeWorkspaceSync, + type NativeRestartRecoveryClaim, + rebindNativeSessionCheckpoint, + type resolveHeartbeatNativeRuntimeMode, +} from "../native-runtime/index.js"; +import { + assertAgentCoreProfileRecoveryBinding, + assertManagedProfileRecoveryBinding, + projectPaperclipRunnerTaskConfig, + resolvePaperclipRunnerNativeProviderInput, +} from "../native-runtime/provider-profile.js"; +import { readRemoteCodexModelCliVersion } from "../native-runtime/codex-model-fallback.js"; +import { + describeRunnerdNativeSessionBackend, + parseNativeExecutionInput, + type NativeExecutionInput, +} from "../../vendor/paperclip-runner/index.js"; +import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js"; +import type { AdapterRuntimeEvent } from "../../adapters/index.js"; +import { parseObject } from "../../adapters/utils.js"; +import { EXTERNAL_CHAT_QUESTION_RESPONSE_KEY } from "../native-runtime/external-chat-question-response.js"; +import { + materializeExternalChatQuestionResponseInput, +} from "../native-runtime/external-chat-question-response-input.js"; +import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js"; +import type { issueService } from "../issues.js"; +import { buildPlanReviewContext } from "../plan-review-context.js"; +import { resolvePaperclipRunnerIdleTimeoutMs } from "@paperclipai/adapter-utils"; +import { selectPaperclipTaskMarkdown } from "@paperclipai/adapter-utils/server-utils"; +import type { ProviderResourceDisposition } from "../environment-runtime.js"; + +type Run = typeof heartbeatRuns.$inferSelect; +type Agent = typeof agents.$inferSelect; +type IssueContext = Awaited["getIssueExecutionContext"]>> | null; +type IssueRef = Parameters[0]["issue"] & Parameters[0]["issue"]; + +export interface NativeLifecycleTelemetry { + provider: string; + harness: string; + lifecycleMode: "per_turn" | "warm"; + sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; +} + +export interface HeartbeatRuntimeSelectionInput { + run: Run; + agent: Agent; + nativeRuntimeResolution: ReturnType; + task: { + issueRef: IssueRef | null; + issueContext: IssueContext; + context: Record; + executionContinuation: Awaited> | null; + safeWakeComments: Parameters[0]; + safeWakeCommentContext: { id: string; body: string } | null; + paperclipWakePayload: Awaited>; + acceptedPlanContinuationWake: boolean; + }; + session: { + persistedNativeExecutionInput: NativeExecutionInput | null; + persistedRunnerProfile: Record; + taskSessionCredentialCompatible: boolean; + taskSessionDecodedParams: Record | null; + taskSessionForRun: { lastRunId: string | null } | null; + isFailedChatRunRetry: boolean; + getFreshSessionHandoff: (() => Promise) | undefined; + }; + workspace: { + persistedExecutionWorkspace: { id: string } | null; + executionWorkspace: Omit[0]["workspace"], "id">; + executionTarget: Parameters[0]["target"]; + activeEnvironmentLease: Pick[0], "lease">; + isDotRun: boolean; + projectRepositoryPaths: string[]; + }; + config: { + runtimeConfig: Record; + runtimeSkillEntries: Parameters[0]["runtimeSkillEntries"]; + runScopedMentionedSkillKeys: string[]; + instructionCopy: Awaited["prepare"]>>; + agentIdentity: { keyId: string } | undefined; + issueAssigneeOverrides: { adapterConfig: Record | null } | null; + resolvedConfig: Record; + managedAiRuntime: Awaited> | undefined; + }; + recovery: { nativeLeaseOwner?: string; nativeRestartRecovery?: NativeRestartRecoveryClaim }; + trace: { providerTraceRequested: boolean; providerTraceCapture: { metadata: { id: string } } | null }; + issuesSvc: Pick, "addComment">; + runtimeEnv: Record; + beforeNativeRuntimeSelection?: (runId: string) => Promise; + onAdapterEvent: (event: AdapterRuntimeEvent) => Promise; + // Teardown must observe these updates even when a later preparation step throws. + onNativeLifecycleSelected: (telemetry: NativeLifecycleTelemetry) => void; + onProviderResourceDisposition: (disposition: ProviderResourceDisposition | undefined) => void; + onNativeOwnershipHeld: () => void; + stopControllerLease: () => void; +} + +export type HeartbeatRuntimeSelectionResult = { selected: false } | { + selected: true; + nativeExecution: NativeExecutionInput | null; + nativeRunnerInstanceId: string | null; + getNativeFreshSessionHandoff: (() => Promise) | undefined; + nativeWorkspaceSync: Awaited>; +}; + +export interface NativeSandboxLifecycle { + runnerProcess: "per_turn" | "warm"; + sandboxResource: "keep_running" | "stop_and_reuse" | "destroy_after_turn"; + failoverBackup: "verified"; +} + +export function resolveReusableSandboxLifecycle(input: { + lifecyclePolicy: + | { mode: "per_turn"; idleTimeoutMs: null } + | { mode: "warm"; idleTimeoutMs: number }; + target: { + kind: "local" | "remote"; + transport?: string; + reusableLeaseConfigured?: boolean; + effectiveCapabilities?: { reusableLeases: boolean } | null; + } | null; +}): NativeSandboxLifecycle | null { + if (input.target?.kind !== "remote" || input.target.transport !== "sandbox") { + return null; + } + const reusableLease = + input.target.reusableLeaseConfigured === true && + input.target.effectiveCapabilities?.reusableLeases === true; + if (input.lifecyclePolicy.mode === "warm" && !reusableLease) { + throw new Error("runner_warm_lifecycle_requires_reusable_provider_lease"); + } + return { + runnerProcess: input.lifecyclePolicy.mode, + sandboxResource: + input.lifecyclePolicy.mode === "warm" + ? "keep_running" + : reusableLease + ? "stop_and_reuse" + : "destroy_after_turn", + failoverBackup: "verified", + }; +} + +export function resolveNativeSandboxLifecycle(input: { + adapterType: string; + lifecyclePolicy: + | { mode: "per_turn"; idleTimeoutMs: null } + | { mode: "warm"; idleTimeoutMs: number }; + target: { + kind: "local" | "remote"; + transport?: string; + reusableLeaseConfigured?: boolean; + effectiveCapabilities?: { reusableLeases: boolean } | null; + } | null; +}): NativeSandboxLifecycle | null { + if ( + input.adapterType !== "paperclip_runner" || + input.target?.kind !== "remote" || + input.target.transport !== "sandbox" + ) + return null; + return resolveReusableSandboxLifecycle(input); +} + +export async function postNativeModelFallbackWarning(input: { + issuesSvc: Pick, "addComment">; + onEvent: (event: AdapterRuntimeEvent) => Promise; + issueId: string; + runId: string; + requestedModel: string | null; + effectiveModel: string | null; + codexCliVersion: string; +}): Promise { + const message = `Using ${input.effectiveModel} because the sandbox's Codex ${input.codexCliVersion} does not support ${input.requestedModel}. Work will continue with the compatible model. Update the sandbox's Codex CLI to use the requested model.`; + await input.onEvent({ + eventType: "runner.model_fallback", + stream: "system", + level: "warn", + message, + payload: { + requestedModel: input.requestedModel, + effectiveModel: input.effectiveModel, + codexCliVersion: input.codexCliVersion, + }, + }); + await input.issuesSvc.addComment(input.issueId, message, { runId: input.runId }, { + authorType: "system", + presentation: { + kind: "system_notice", + tone: "warning", + title: `Using ${input.effectiveModel}`, + density: "compact", + detailsDefaultOpen: false, + }, + }); +} + +function readNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.trim().length > 0 ? value : null; +} + +/** Prepare and persist the selected runtime before the executor dispatches provider work. */ +export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSelectionInput): Promise { + const { run, agent, nativeRuntimeResolution, issuesSvc, runtimeEnv, beforeNativeRuntimeSelection, onAdapterEvent, + onNativeLifecycleSelected, onProviderResourceDisposition, onNativeOwnershipHeld, stopControllerLease } = input; + const { issueRef, issueContext, context, executionContinuation, safeWakeComments, safeWakeCommentContext, + paperclipWakePayload, acceptedPlanContinuationWake } = input.task; + const { persistedNativeExecutionInput, persistedRunnerProfile, taskSessionCredentialCompatible, taskSessionDecodedParams, + taskSessionForRun, isFailedChatRunRetry, getFreshSessionHandoff } = input.session; + const { persistedExecutionWorkspace, executionWorkspace, executionTarget, activeEnvironmentLease, isDotRun, projectRepositoryPaths } = input.workspace; + const { runtimeConfig, runtimeSkillEntries, runScopedMentionedSkillKeys, instructionCopy, agentIdentity, issueAssigneeOverrides, resolvedConfig, managedAiRuntime } = input.config; + const runOptions = input.recovery; + const { providerTraceRequested, providerTraceCapture } = input.trace; + let nativeExecution: NativeExecutionInput | null = null; + let getNativeFreshSessionHandoff: (() => Promise) | undefined; + let nativeRunnerInstanceId: string | null = null; + let nativeWorkspaceSync: Awaited> = null; + let providerResourceDispositionForRun: ProviderResourceDisposition | undefined; + if (nativeRuntimeResolution.kind === "native") { + if (!issueRef) { + throw new Error("native_runtime_ineligible: issue is required"); + } + const nativeExecutionWorkspaceId = + persistedExecutionWorkspace?.id ?? run.id; + const nativeReviewContext = readNativeReviewAssignmentContext(context); + const nativeReview = nativeReviewContext ? await getNativeReviewAssignment(db, { + companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, + contextSnapshot: nativeReviewContext, + }) : null; + if (nativeReviewContext && !nativeReview) throw new Error("native_review_assignment_no_longer_available"); + const nativeReviewRequest = nativeReview + ? buildNativeReviewRequest({ + title: nativeReview.interaction.title, + summary: nativeReview.interaction.summary, + payload: nativeReview.interaction.payload, + }) + : null; + const persistedContract = run.completionContractId + ? await db + .select() + .from(completionContracts) + .where( + and( + eq(completionContracts.id, run.completionContractId), + eq(completionContracts.companyId, agent.companyId), + eq(completionContracts.issueId, issueRef.id), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null) + : null; + // Only a server-verified human resolution may supply a current answer + // reference. Tool/agent results and generated summaries stay evidence. + const currentHumanResponseId = !nativeReviewRequest + ? executionContinuation?.humanResponses?.find( + (response) => response.id === executionContinuation.trigger.interactionId, + )?.id + : undefined; + const immediateCompletion = (() => { + if (nativeReviewRequest) return { requests: [nativeReviewRequest], sources: [null] }; + const { requests, sources } = nativeCompletionRequestsWithSources( + safeWakeComments.length > 0 + ? safeWakeComments + : safeWakeCommentContext?.body + ? [safeWakeCommentContext] + : [], + { + requiredFullWakeCommentCount: + paperclipWakePayload?.fallbackFetchNeeded === true && + CHAT_PROVIDERS.some( + (provider) => + provider === + paperclipWakePayload.externalChatProvider, + ) && + Array.isArray(paperclipWakePayload.commentIds) + ? paperclipWakePayload.commentIds.length + : undefined, + }, + ); + // Preserve every admitted pending chat request while also + // retaining newer user direction materialized by recovery. + // A file-only wake must not inherit an old task objective. + const latestComment = + executionContinuation?.messages.findLast( + (message) => + message.authorType === "user" && + !message.createdByRunId && + !message.deleted && + message.body.trim().length > 0, + ); + const latestRequest = latestComment?.body; + if ( + latestRequest && + !requests.some( + (request) => request === latestRequest.trim(), + ) + ) { + requests.push(latestRequest.trim()); + sources.push(nativeCompletionSource("comment", latestComment!.id, latestRequest)); + } + return { requests: requests.length > 0 ? requests : undefined, sources }; + })(); + // Rebuilding a default contract is not a change in user direction. + // In particular, an upgraded checkpoint may have an intentionally + // authored contract and no continuation envelope yet. + const completionContract = + persistedContract && persistedNativeExecutionInput + ? { + row: persistedContract, + contract: persistedContract.contractJson as never, + } + : await ensureNativeCompletionContract({ + db, + companyId: agent.companyId, + issue: issueRef, + actorId: agent.id, + immediateRequest: + nativeReviewRequest ?? (currentHumanResponseId + ? null + : executionContinuation?.objective ?? safeWakeCommentContext?.body ?? null), + // The ordinary initial objective is selected by the server-owned + // continuation envelope. Carry its explicit description source + // through the singular-request compatibility path; do not infer + // provenance for review requests, answers, or wake fallbacks. + immediateRequestSource: nativeImmediateObjectiveSource({ + issueId: issueRef.id, + objectiveSource: executionContinuation?.objectiveSource, + excluded: Boolean(nativeReviewRequest || currentHumanResponseId), + }), + humanResponseId: currentHumanResponseId, + immediateRequests: immediateCompletion.requests, + immediateRequestSources: immediateCompletion.sources, + }); + const taskNativeSessionId = !taskSessionCredentialCompatible ? null : readNonEmptyString( + taskSessionDecodedParams?.sessionId, + ); + // Compatibility for native retry rows created before same-run restart + // recovery existed. Only an entirely unused replacement row may + // inherit its source checkpoint; any process/provider evidence on the + // replacement makes the ownership ambiguous and therefore ineligible. + const legacyRetrySource = + run.retryOfRunId && !isFailedChatRunRetry + ? await db + .select({ + id: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + agentId: heartbeatRuns.agentId, + runnerInstanceId: heartbeatRuns.runnerInstanceId, + nativeSessionId: heartbeatRuns.nativeSessionId, + runnerProfileJson: heartbeatRuns.runnerProfileJson, + runtimeMode: heartbeatRuns.runtimeMode, + status: heartbeatRuns.status, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, run.retryOfRunId), + eq(heartbeatRuns.companyId, agent.companyId), + eq(heartbeatRuns.agentId, agent.id), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null) + : null; + const nativeBootstrapHasProviderEvidence = + legacyRetrySource || + run.nativeSessionId || + persistedNativeExecutionInput + ? await db + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where( + and( + eq(heartbeatRunEvents.runId, run.id), + inArray(heartbeatRunEvents.eventType, [ + "harness.ready", + "session.started", + "session.resumed", + "session.updated", + "turn.started", + "provider.event", + "provider.rpc_result", + ]), + ), + ) + .limit(1) + .then((rows) => rows.length > 0) + : false; + const compatibleLegacyRetrySource = + !managedAiRuntime && !isConversation(issueContext) && context.forceFreshSession !== true && isUnusedLegacyNativeRetryReplacement({ + replacement: run, + source: legacyRetrySource, + hasProviderEvents: nativeBootstrapHasProviderEvidence, + }) + ? legacyRetrySource + : null; + const legacyRetrySessionId = + compatibleLegacyRetrySource?.nativeSessionId; + const taskResumeRunId = + taskSessionForRun?.lastRunId && + taskSessionForRun.lastRunId !== run.id && + isNativeSessionId(taskNativeSessionId) + ? taskSessionForRun.lastRunId + : null; + const resumableTaskSessionId = isDotRun ? null : taskResumeRunId + ? taskNativeSessionId + : (legacyRetrySessionId ?? null); + const requestedNativeSessionId = + run.nativeSessionId ?? resumableTaskSessionId; + // A task-session lastRunId can lag a failed turn or point at an older + // normalized session. Find the newest exact-session authority instead. + // Rows that already acquired provider authority are progress barriers, + // even when they do not contain a usable checkpoint. + const previousNativeRun = + requestedNativeSessionId && + isUnusedNativeSessionBootstrap( + run, + nativeBootstrapHasProviderEvidence, + ) + ? await findNativeSessionResumeRun(db, { + companyId: agent.companyId, + agentId: agent.id, + issueId: issueRef.id, + normalizedSessionId: requestedNativeSessionId, + currentRunId: run.id, + beforeCreatedAt: run.createdAt, + }) + : null; + nativeRunnerInstanceId = + previousNativeRun?.runnerInstanceId && + previousNativeRun.nativeSessionId === + (run.nativeSessionId ?? resumableTaskSessionId) + ? previousNativeRun.runnerInstanceId + : (run.runnerInstanceId ?? randomUUID()); + let nativeSessionId = + run.nativeSessionId ?? resumableTaskSessionId ?? randomUUID(); + let nativeResumeCheckpoint: ReturnType< + typeof rebindNativeSessionCheckpoint + > = null; + const agentLifecyclePolicy = + parseObject(agent.adapterConfig).lifecycleMode === "warm" + ? { + mode: "warm" as const, + idleTimeoutMs: resolvePaperclipRunnerIdleTimeoutMs( + parseObject(agent.adapterConfig).idleTimeoutMs, + ), + } + : { mode: "per_turn" as const, idleTimeoutMs: null }; + const environmentLifecyclePolicy = + executionTarget?.kind === "remote" && + executionTarget.transport === "sandbox" + ? (executionTarget.runnerLifecyclePolicy ?? null) + : null; + // Native Codex owns a durable, session-scoped home. It flushes refreshed + // auth into each invocation's private home before that home is removed. + // Other managed harnesses still require per-turn credential cleanup. + const supportsManagedWarmSession = agent.adapterType === "paperclip_runner" && + nativeRuntimeResolution.profile.backend === "codex_app_server"; + const effectiveLifecyclePolicy = persistedNativeExecutionInput?.session.lifecyclePolicy ?? + (nativeRuntimeResolution.profile.backend === "openai_dot_mcp" || managedAiRuntime && !supportsManagedWarmSession + ? { mode: "per_turn" as const, idleTimeoutMs: null } + : environmentLifecyclePolicy ?? agentLifecyclePolicy); + if ( + effectiveLifecyclePolicy.mode === "warm" && + executionTarget?.kind === "remote" && + executionTarget.transport === "sandbox" && + (executionTarget.reusableLeaseConfigured !== true || + executionTarget.effectiveCapabilities?.reusableLeases !== true) + ) { + throw new Error("runner_warm_environment_requires_reusable_lease"); + } + const persistedProfile = persistedRunnerProfile; + if (persistedNativeExecutionInput) { + nativeExecution = persistedNativeExecutionInput; + if ( + nativeExecution.binding.companyId !== agent.companyId || + nativeExecution.binding.runId !== run.id || + nativeExecution.binding.issueId !== issueRef.id || + nativeExecution.binding.agentId !== agent.id || + nativeExecution.binding.executionWorkspaceId !== + nativeExecutionWorkspaceId || + nativeExecution.completionContract.id !== + completionContract.row.id || + nativeExecution.completionContract.sha256 !== + completionContract.row.canonicalSha256 + ) + throw new Error( + "native_execution_input_persisted_binding_mismatch", + ); + // Recover only the originally admitted request. A stored idle + // checkpoint can precede an already-started provider turn, so even + // apparent idleness is not authority to rewrite its contract. + // Newer user direction retains its separate durable wakeup cause. + // A failed pre-bootstrap attempt may have persisted its immutable + // input before discovering that lastRunId no longer names this + // session. Restore only an exactly compatible prior checkpoint; + // never rewrite the admitted input or skip current provider work. + if ( + previousNativeRun && + isUnusedNativeSessionBootstrap( + run, + nativeBootstrapHasProviderEvidence, + ) + ) { + nativeResumeCheckpoint = rebindNativeSessionCheckpoint({ + previousRun: previousNativeRun, + currentExecution: nativeExecution, + executionTargetKind: executionTarget?.kind ?? "local", + }); + } + if (nativeExecution.provider.kind === "claude_managed") { + const recoveryProfile = await managedAgentProfileService( + db, + ).requireQualified( + agent.companyId, + nativeExecution.provider.managedProfile.profileId, + ); + assertManagedProfileRecoveryBinding({ + adapterConfig: agent.adapterConfig, + snapshot: nativeExecution.provider.managedProfile, + stored: recoveryProfile, + }); + } + if (nativeExecution.provider.kind === "aws_agentcore") { + const recoveryProfile = await remoteAgentProfileService( + db, + ).requireQualified( + agent.companyId, + nativeExecution.provider.agentCoreProfile.profileId, + "aws_bedrock_agentcore_harness", + ); + assertAgentCoreProfileRecoveryBinding({ + snapshot: nativeExecution.provider.agentCoreProfile, + stored: recoveryProfile, + }); + } + } else { + const interactionId = readNonEmptyString(context.interactionId); + const interactionResponses = context[ + EXTERNAL_CHAT_QUESTION_RESPONSE_KEY + ] + ? await materializeExternalChatQuestionResponseInput({ + db, + binding: { + companyId: agent.companyId, + issueId: issueRef.id, + runId: run.id, + agentId: agent.id, + }, + contextSnapshot: context, + }) + : await materializeNativeInteractionResponses({ + db, + companyId: agent.companyId, + issueId: issueRef.id, + runId: run.id, + agentId: agent.id, + interactionIds: Array.isArray(context.interactionIds) + ? [ + ...new Set([ + ...(interactionId ? [interactionId] : []), + ...context.interactionIds.filter( + (id): id is string => typeof id === "string", + ), + ]), + ] + : interactionId + ? [interactionId] + : [], + }); + const runnerAdapterConfig = parseObject(agent.adapterConfig); + const managedProfile = + nativeRuntimeResolution.profile.backend === + "claude_managed_agents_api" + ? await managedAgentProfileService(db).requireQualified( + agent.companyId, + readNonEmptyString(runnerAdapterConfig.managedProfileId) ?? + "", + ) + : null; + const agentCoreProfile = + nativeRuntimeResolution.profile.backend === + "aws_agentcore_harness_api" + ? await remoteAgentProfileService(db).requireQualified( + agent.companyId, + readNonEmptyString( + runnerAdapterConfig.agentCoreProfileId, + ) ?? "", + "aws_bedrock_agentcore_harness", + ) + : null; + if (managedProfile) { + const rawApiKeyBinding = parseObject( + runnerAdapterConfig.env, + ).ANTHROPIC_API_KEY; + const boundSecretId = + typeof rawApiKeyBinding === "object" && + rawApiKeyBinding !== null + ? readNonEmptyString( + (rawApiKeyBinding as Record).secretId, + ) + : null; + if (boundSecretId !== managedProfile.apiKeySecretId) { + throw new ConfigurationIncompleteFailure( + "configuration incomplete: Claude Managed profile API key is not bound at env.ANTHROPIC_API_KEY", + { + configurationIncomplete: { + reason: "managed_agent_profile_secret_binding_mismatch", + companyId: agent.companyId, + agentId: agent.id, + profileId: managedProfile.id, + requiredEnvKeys: ["ANTHROPIC_API_KEY"], + }, + }, + ); + } + } + const executionMode = + issueRef.workMode === "planning" && !isConversation(issueContext) && !acceptedPlanContinuationWake + ? ("plan" as const) + : ("default" as const); + const pinnedPlan = + executionMode === "plan" + ? await documentService(db).getIssueDocumentByKey( + issueRef.id, + "plan", + ) + : null; + const pinnedReviewContext = + executionMode === "plan" + ? await buildPlanReviewContext({ + db, + companyId: agent.companyId, + issueId: issueRef.id, + issueWorkMode: issueRef.workMode, + interactionId: readNonEmptyString(context.interactionId), + }) + : null; + const pinnedPlanMarkdown = pinnedPlan?.body ?? ""; + const dotBinding = nativeRuntimeResolution.profile.backend === "openai_dot_mcp" + ? await dotRunnerBroker(db).snapshot(agent.companyId, agent.id, String(parseObject(agent.adapterConfig).dotBindingId ?? "")) : undefined; + const nativeRuntimeContext = await buildNativeRuntimeContext({ + db, + agent, + runId: run.id, + runtimeConfig, + runtimeSkillEntries, + instructionWorkingCopy: instructionCopy ? { rootPath: instructionCopy.executionRoot, entryPath: instructionCopy.entryFile, ...(isAgentDirectoryCopy(instructionCopy) ? { kind: "agent_files" as const } : {}) } : undefined, + }); + getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff; + const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "codex_app_server" + || nativeRuntimeResolution.profile.backend === "opencode_server" + ? projectPaperclipRunnerTaskConfig( + nativeRuntimeResolution.profile.backend, + agent.adapterConfig, + issueAssigneeOverrides?.adapterConfig, + managedAiRuntime ? readNonEmptyString(resolvedConfig.model) ?? undefined : undefined, + ) + : agent.adapterConfig; + const requestedNativeProvider = resolvePaperclipRunnerNativeProviderInput({ + backend: nativeRuntimeResolution.profile.backend, + adapterConfig: nativeProviderConfig, managedProfile, agentCoreProfile, dotBinding, + }); + const codexCliVersion = agent.adapterType === "paperclip_runner" && requestedNativeProvider.provider === "codex" + ? await readRemoteCodexModelCliVersion({ + model: requestedNativeProvider.model, + target: executionTarget, + remoteCodexPath: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH, + remoteCodexNpmSpec: runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC, + }) : null; + const nativeProvider = codexCliVersion + ? resolvePaperclipRunnerNativeProviderInput({ + backend: nativeRuntimeResolution.profile.backend, + adapterConfig: nativeProviderConfig, codexCliVersion, managedProfile, agentCoreProfile, dotBinding, + }) : requestedNativeProvider; + if (nativeProvider.model !== requestedNativeProvider.model) { + await postNativeModelFallbackWarning({ + issuesSvc, onEvent: onAdapterEvent, issueId: issueRef.id, runId: run.id, + requestedModel: requestedNativeProvider.model, effectiveModel: nativeProvider.model, + codexCliVersion: codexCliVersion!, + }); + } + const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) => + buildNativeExecutionInput({ + agentKeyId: agentIdentity?.keyId, + companyId: agent.companyId, + runId: run.id, + issue: nativeReviewRequest ? { ...issueRef, title: `Review: ${issueRef.title}`, description: nativeReviewRequest } : issueRef, + taskPrompt: [ + nativeReviewRequest ?? readNonEmptyString( + selectPaperclipTaskMarkdown(context, { + resumedSession: false, + includeCommunicationGuidance: false, + }), + ) ?? + `# ${issueRef.identifier ?? issueRef.id}: ${issueRef.title}`, + nativeRuntimeResolution.profile.backend !== "openai_dot_mcp" && projectRepositoryPaths.length > 0 + ? `## Project repositories\nThe task workspace also contains these editable Git repositories:\n${projectRepositoryPaths.map((repo) => `- ${repo}`).join("\n")}` + : null, + ].filter(Boolean).join("\n\n"), + initialCommunicationGuidance: nativeReviewRequest ? null : readNonEmptyString(context.paperclipTaskCommunicationGuidance), + wakePayload: context.paperclipWake, + turnContext: context.paperclipTurnContext, + githubInstructionSkillKeys: runScopedMentionedSkillKeys, + resumedSession, + previousTurn: (() => { + if (!previousNativeRun || nativeReviewRequest) return null; + try { + const previousTask = parseNativeExecutionInput(parseObject(previousNativeRun.runnerProfileJson).nativeExecutionInput).task; + if (paperclipWakePayload?.externalChatProvider) { + // External native inputs use a neutral task title. Compare + // the saved canonical brief so old provider text is not + // repeated as a change, while genuine edits still arrive. + const savedIssue = parseObject(parseObject(previousNativeRun.contextSnapshot).paperclipIssue); + if (savedIssue.id !== issueRef.id || typeof savedIssue.title !== "string" || + (savedIssue.description !== null && typeof savedIssue.description !== "string")) return null; + return { runId: previousNativeRun.id, task: { title: savedIssue.title, description: savedIssue.description } }; + } + return { runId: previousNativeRun.id, task: previousTask }; + } catch { + // An invalid prior snapshot must use the fresh bootstrap. + return null; + } + })(), + conversationMode: context.conversationMode === true, + agentId: agent.id, + workspace: { + // Projectless paperclip_runner tasks still have a resolved local cwd. Bind that + // transient workspace to the run id so the native input remains durable and replayable + // without fabricating a project-scoped execution_workspaces row. + id: nativeExecutionWorkspaceId, + cwd: executionWorkspace.cwd, + repoUrl: executionWorkspace.repoUrl, + repoRef: executionWorkspace.repoRef, + branchName: executionWorkspace.branchName, + }, + normalizedSessionId, + executionMode, + planningContext: + executionMode === "plan" + ? { + documentId: pinnedPlan?.id ?? null, + baseRevisionId: + pinnedPlan?.latestRevisionId ?? null, + baseRevisionNumber: + pinnedPlan?.latestRevisionNumber ?? 0, + markdown: pinnedPlanMarkdown, + sha256: createHash("sha256") + .update(pinnedPlanMarkdown) + .digest("hex"), + reviewContext: pinnedReviewContext + ? (structuredClone( + pinnedReviewContext, + ) as unknown as Record) + : {}, + } + : null, + ...nativeProvider, + lifecyclePolicy: effectiveLifecyclePolicy, + interactionResponses, + completionContract: { + id: completionContract.row.id, + sha256: completionContract.row.canonicalSha256, + schemaVersion: completionContract.row.schemaVersion, + contract: completionContract.contract, + sources: "sources" in completionContract ? completionContract.sources : undefined, + }, + runtimeContext: nativeRuntimeContext, + }); + const backendDescriptor = await describeRunnerdNativeSessionBackend(buildExecution({ + normalizedSessionId: nativeSessionId, resumedSession: previousNativeRun !== null, + })); + const nativeExecutionWithCheckpoint = buildNativeExecutionWithCheckpoint({ + previousRun: previousNativeRun, + normalizedSessionId: nativeSessionId, + executionTargetKind: executionTarget?.kind ?? "local", + toolRefreshOnResume: backendDescriptor.capabilities.toolRefreshOnResume === true, + refreshTools: context.refreshTools === true, + buildExecution, + }); + nativeExecution = nativeExecutionWithCheckpoint.execution; + nativeResumeCheckpoint = nativeExecutionWithCheckpoint.checkpoint; + if ( + nativeSessionId !== + nativeExecutionWithCheckpoint.normalizedSessionId + ) { + nativeRunnerInstanceId = randomUUID(); + } + nativeSessionId = nativeExecutionWithCheckpoint.normalizedSessionId; + } + const nativeSandboxLifecycle = resolveNativeSandboxLifecycle({ + adapterType: agent.adapterType, + lifecyclePolicy: nativeExecution.session.lifecyclePolicy, + target: executionTarget, + }); + if (nativeSandboxLifecycle) { + onNativeLifecycleSelected({ + provider: nativeExecution.provider.kind, + harness: nativeExecution.session.driverKind, + lifecycleMode: nativeExecution.session.lifecyclePolicy.mode, + sandboxResource: nativeSandboxLifecycle.sandboxResource, + }); + const selectedLifecycleSpan = getStartupTracer( + "paperclip.environment-lifecycle", + ).startSpan("sandbox.lifecycle.selected", { + attributes: { + "paperclip.native.span.provider": nativeExecution.provider.kind, + "paperclip.native.span.harness": + nativeExecution.session.driverKind, + "paperclip.native.span.lifecycle_mode": + nativeExecution.session.lifecyclePolicy.mode, + "paperclip.native.span.sandbox_resource": + nativeSandboxLifecycle.sandboxResource, + "paperclip.native.span.outcome": "selected", + "paperclip.native.span.bytes_transferred": 0, + }, + }); + selectedLifecycleSpan.end(); + } + providerResourceDispositionForRun = + nativeSandboxLifecycle?.sandboxResource === "keep_running" + ? "keep_running" + : nativeSandboxLifecycle?.sandboxResource === "stop_and_reuse" + ? "stop_and_retain" + : nativeSandboxLifecycle?.sandboxResource === + "destroy_after_turn" + ? "destroy" + : undefined; + onProviderResourceDisposition(providerResourceDispositionForRun); + await beforeNativeRuntimeSelection?.(run.id); + const nativeSelected = await db.transaction(async (tx) => { + const lockedRun = await tx + .select() + .from(heartbeatRuns) + .where(eq(heartbeatRuns.id, run.id)) + .for("update") + .limit(1) + .then((rows) => rows[0] ?? null); + if (!lockedRun) throw new Error("native_runtime_run_missing"); + // Cancellation and runtime selection serialize on this row. A + // stopped preparation must never create a new native coordinator. + if (lockedRun.status !== "running" || lockedRun.resultJson?.startupCancellation) return false; + if (lockedRun.runtimeMode === "legacy" && lockedRun.controllerBootId && + !(await renewLegacyControllerLease(tx as unknown as Db, lockedRun))) { + onNativeOwnershipHeld(); + return false; + } + if ( + lockedRun.runtimeModeResolvedAt && + lockedRun.runtimeMode !== "native" + ) { + throw new Error("native_runtime_mode_conflict"); + } + const lockedProfile = parseObject(lockedRun.runnerProfileJson); + const persistedNativeSessionId = + await prepareNativeSessionBootstrapPersistence(tx, { + run: lockedRun, + selectedSessionId: nativeSessionId, + execution: nativeExecution!, + restoringCheckpoint: nativeResumeCheckpoint !== null, + }); + await tx + .update(heartbeatRuns) + .set({ + runtimeMode: "native", + runtimeModeResolverVersion: + lockedRun.runtimeModeResolverVersion ?? + nativeRuntimeResolution.resolverVersion, + runtimeModeReason: + lockedRun.runtimeModeReason ?? nativeRuntimeResolution.reason, + runtimeModeResolvedAt: + lockedRun.runtimeModeResolvedAt ?? new Date(), + runnerProfileJson: { + ...nativeRuntimeResolution.profile, + ...lockedProfile, + ...(lockedProfile.nativeExecutionInput + ? {} + : { recoveryEventInventoryVersion: 1 }), + ...(providerTraceRequested + ? { + providerTrace: { + mode: "raw", + traceId: providerTraceCapture?.metadata.id ?? null, + maxBytes: PROVIDER_TRACE_MAX_BYTES, + }, + } + : {}), + nativeExecutionInput: + lockedProfile.nativeExecutionInput ?? nativeExecution, + nativeToolContractFingerprint: + nativeToolContractFingerprintForTarget( + executionTarget?.kind ?? "local", + ), + ...(lockedProfile.sessionCheckpoint != null + ? { sessionCheckpoint: lockedProfile.sessionCheckpoint } + : nativeResumeCheckpoint + ? { + sessionCheckpoint: + nativeResumeCheckpoint as unknown as Record< + string, + unknown + >, + } + : {}), + }, + runnerInstanceId: + previousNativeRun?.runnerInstanceId && + persistedNativeSessionId === previousNativeRun.nativeSessionId + ? previousNativeRun.runnerInstanceId + : lockedRun.nativeSessionId !== persistedNativeSessionId + ? nativeRunnerInstanceId + : (lockedRun.runnerInstanceId ?? nativeRunnerInstanceId), + nativeSessionId: persistedNativeSessionId, + processPid: + lockedRun.processPid ?? + (previousNativeRun?.nativeSessionId === nativeSessionId + ? previousNativeRun.processPid + : null), + processGroupId: + lockedRun.processGroupId ?? + (previousNativeRun?.nativeSessionId === nativeSessionId + ? previousNativeRun.processGroupId + : null), + processStartedAt: + lockedRun.processStartedAt ?? + (previousNativeRun?.nativeSessionId === nativeSessionId + ? previousNativeRun.processStartedAt + : null), + nativeIssueId: lockedRun.nativeIssueId ?? issueRef.id, + driverKind: + lockedRun.driverKind ?? + nativeExecution?.session.driverKind ?? + "codex_app_server", + driverVersion: lockedRun.driverVersion ?? "phase6-v1", + completionContractId: + lockedRun.completionContractId ?? completionContract.row.id, + completionContractSha256: + lockedRun.completionContractSha256 ?? + completionContract.row.canonicalSha256, + nativePhase: lockedRun.nativePhase ?? "observed", + nativePhaseUpdatedAt: + lockedRun.nativePhaseUpdatedAt ?? new Date(), + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, run.id)); + await tx + .insert(nativeRunFinalizations) + .values({ + runId: run.id, + companyId: agent.companyId, + issueId: issueRef.id, + phase: "observed", + }) + .onConflictDoNothing(); + return true; + }); + if (!nativeSelected) return { selected: false }; + stopControllerLease(); + nativeWorkspaceSync = await prepareNativeWorkspaceSync({ + db, + runId: run.id, + companyId: agent.companyId, + workspaceId: nativeExecutionWorkspaceId, + workspaceLocalDir: executionWorkspace.cwd, + target: executionTarget, + lease: activeEnvironmentLease.lease, + restartRecovery: runOptions.nativeRestartRecovery, + sameRunRecovery: Boolean(runOptions.nativeLeaseOwner), + resourceDisposition: providerResourceDispositionForRun, + }); + } else { + const legacyWarmLifecycle = + executionTarget?.kind === "remote" && + executionTarget.transport === "sandbox" && + executionTarget.runnerLifecyclePolicy?.mode === "warm" + ? resolveReusableSandboxLifecycle({ + lifecyclePolicy: executionTarget.runnerLifecyclePolicy, + target: executionTarget, + }) + : null; + if (legacyWarmLifecycle?.sandboxResource === "keep_running") { + providerResourceDispositionForRun = "keep_running"; + onProviderResourceDisposition(providerResourceDispositionForRun); + } + await db + .update(heartbeatRuns) + .set({ + runtimeMode: "legacy", + runtimeModeResolverVersion: + nativeRuntimeResolution.resolverVersion, + runtimeModeReason: nativeRuntimeResolution.reason, + runtimeModeResolvedAt: run.runtimeModeResolvedAt ?? new Date(), + // Preserve server-owned admission and dispatch evidence on this + // row; never copy another run's execution profile. + runnerProfileJson: sql`(case when ${heartbeatRuns.runnerProfileJson} ? ${CHAT_CONTROL_RECOVERY_ADMISSION_KEY} + then jsonb_build_object(${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}::text, ${heartbeatRuns.runnerProfileJson}->${CHAT_CONTROL_RECOVERY_ADMISSION_KEY}) + else '{}'::jsonb end) + || (case when ${heartbeatRuns.runnerProfileJson} ? 'adapterDispatch' + then jsonb_build_object('adapterDispatch', ${heartbeatRuns.runnerProfileJson}->'adapterDispatch') + else '{}'::jsonb end) || ${JSON.stringify(providerTraceRequested ? { providerTrace: { mode: "raw", traceId: providerTraceCapture?.metadata.id ?? null, maxBytes: PROVIDER_TRACE_MAX_BYTES } } : {})}::jsonb`, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, run.id)); + } + return { selected: true, nativeExecution, nativeRunnerInstanceId, getNativeFreshSessionHandoff, nativeWorkspaceSync }; +} diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index fedc1628476..d486db139f4 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -461,7 +461,7 @@ describe("capability-gated connection tool refresh", () => { it("wires exact-session recovery and guarded selected identity into heartbeat persistence", () => { const source = readFileSync( - new URL("../heartbeat.ts", import.meta.url), + new URL("../heartbeat/runtime-selection.ts", import.meta.url), "utf8", ); expect(source).toContain("await findNativeSessionResumeRun(db,");