diff --git a/DESIGN.md b/DESIGN.md index a4804cb0e27..bda54f4da2b 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -60,8 +60,12 @@ title sizes remain larger. The bottom navigation responds to accumulated scrolling, ignoring small reversals and Safari's edge bounce. It glides out and eases back in with shared -motion tokens; the task composer follows the same motion. Keep page padding -stable while the navigation moves, and honor reduced-motion preferences. +motion tokens. Keep page padding stable while the navigation moves, and honor +reduced-motion preferences. + +Mobile task conversations size their shell to the visible viewport, with one +message scroller and a normal flex footer for the composer. Navigation stays at +the layout viewport bottom, behind the keyboard. Cold task opens reveal the instruction and recent conversation independently of plans, related metadata, and historical run activity. Show the latest human diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index f8d408cdff9..cd78289ac2e 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1983,6 +1983,14 @@ runs; immutable run descriptors remain readable. The **Paperclip Runner** experimental setting applies to other native providers. Native Codex, OpenCode, ACPX Claude, and ACPX Pi do not require it. Company access checks still apply. +Native Codex preserves an explicit `adapterConfig.search: false` when converting +or saving an agent. The disabled hosted-search policy is recorded in the native +execution input and launch requirements, so prepared launches and recovery keep +it. An absent setting retains the provider default; this does not add support +for converting a Legacy agent with `search: true`. Authentication hello probes +always disable hosted web search because they only verify the selected account +and model. + OpenCode requires an explicit `provider/model` before testing or finishing setup. An inactive setup draft can be saved before choosing a model. Automatic native selection uses `allow` permissions and a `per_turn` lifecycle. Native `ask` and diff --git a/doc/architecture/runner-copilot-capabilities.md b/doc/architecture/runner-copilot-capabilities.md index 4226846d5af..36e2f8ea81c 100644 --- a/doc/architecture/runner-copilot-capabilities.md +++ b/doc/architecture/runner-copilot-capabilities.md @@ -1,6 +1,24 @@ # Copilot 1.0.88 rich ACP capability audit -Current integration (2026-10-17): **Copilot profile v21 remains pending** with +Current source declaration (2026-10-26): **Copilot profile v61** binds controller +journal retirement and the historical profile decoder. Its digest is +`sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3`. +The v60 declaration remains unchanged and readable as history. This source +identity update changes no Copilot executable, arguments, permission policy, +tool receipts, or qualification flags. Offline compatibility checks supply no +new live qualification; prior live outcomes retain their original identities. + +Historical source declaration (2026-10-26): **Copilot profile v60** binds the shared +sources that carry Codex's explicit hosted-search restriction. Its digest is +`sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa`. +The v59 declaration remains unchanged and readable as history. Exact-current +launch admission still applies. The Copilot executable, arguments, permissions, +receipts, and qualification flags are unchanged. Copilot authentication probes +do not carry the Codex-only setting. Offline compatibility checks do not supply +new live qualification; prior live outcomes remain attached to their original +source identities. + +Historical integration (2026-10-17): **Copilot profile v21 remained pending** with digest `sha256:1dad3d26d91e66c6f911dbfa43e7963f36c0bbf0d7709f4bbb568029a8b4b46a`. It binds the shared provider completion contract, direct driver, production sidecar, and controller continuation resolver. Installation verification requires this generated v21 diff --git a/doc/connections/AI-CONNECTIONS.md b/doc/connections/AI-CONNECTIONS.md index 1bd9d7a81da..6a12c8930e3 100644 --- a/doc/connections/AI-CONNECTIONS.md +++ b/doc/connections/AI-CONNECTIONS.md @@ -59,6 +59,12 @@ the selected connection owns its provider routing. Changing those separately may make a binding incompatible; saving then requires a compatible choice. Agent configuration cannot grant access to another account. +Agent account choices and legacy-adoption confirmations show the sign-in method, +personal or company-shared scope, and available account or owner identity beside +the connection name. Accounts with the same provider name can be distinguished +before selection. These labels use existing authorized account metadata, never +credentials or authentication paths. + Personal defaults are unique per company, user, and provider. A Claude bot can use one user’s subscription and another user’s API key without changing its harness or model. Explicit shared selections remain pinned to the selected account and method. The first successful personal connection sets a default only when none exists. The additive `ai_provider_defaults` table preserves the legacy per-method preferences. Migration selects each user’s most recently updated provider preference (including unavailable accounts), and rerunning it never overwrites a provider default. New writes maintain the legacy table for older servers. A database trigger propagates older servers’ explicit default updates to the provider default. Inserting an additional method default does not replace an existing provider default. diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 52dbcac821e..2c6fc2218d0 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -730,7 +730,16 @@ recovery path with the specific scan cause and an operator action. Generic stranded-work recovery must not grant another budget for these errors. This does not automatically replay historical generic `setup_failed` runs. -### ACP startup handshake bound +### Runner authentication and ACP startup handshake bounds + +The native Runner waits up to the ten-second connection setup budget for the +authenticated welcome. The current execution or reconnect deadline can shorten +that wait. Controller admission and durable lease persistence happen before the +welcome, so a short internal response limit must not discard an otherwise valid +bootstrap ticket during that work. A failed welcome still makes the one-use +ticket unsafe to replay. Authentication proofs, credential expiry, and identity +checks remain required. Steady-state reads return to their 250 ms polling bound +after the welcome. An adapter-backed live path also requires that the ACP startup handshake itself cannot hang forever. The engine bounds the handshake with a fixed startup deadline and a poll of the duplex control-channel disposition. Either condition ends the handshake and reports a closed, typed code, so the issue can reach a settled disposition instead of staying `in_progress` with no observable next action. @@ -1327,6 +1336,13 @@ does not reset the provider-attempt budget. Provider execution and control-plane finalization have different clocks. A healthy provider can think or execute a long tool without output. Once execution settles, recovery and finalization control steps have a 60-second deadline, checked on startup and every 15 seconds. With a healthy database and scheduler, an abandoned transition must be repaired or surfaced within 90 seconds. Terminal persistence must not wait on provider cleanup or publication; a late finalizer cannot change a reassigned or closed task or release another run's locks. Historical ambiguous runs are never automatically replayed after an upgrade. +When that deadline fails a native run with a prepared workspace, the same +transaction records the exact source's file-preservation obligation. Existing +stop-and-retain recovery contains the provider and saves the files before +ordinary Retry can continue. Finalized workspaces, mismatched sources and +accepted results keep their existing settlement paths. Reassignment does not +discard the stopped run's files or grant authority over the new assignee. + Every continuation carries the triggering request, ordered user direction, interaction outcomes, completed work, and explicit history coverage. A delivered message remains part of the task's request after its connection or approval resolves. The original title is background; a completed Notion read does not satisfy a later Gmail request. Author and source-trust boundaries survive rendering into both native and legacy prompts. Missing required history must be fetched before dispatch rather than described as complete. ### Interrupted conversation continuation @@ -1376,14 +1392,25 @@ accepted native result. A successful `workspace_recovery` operation and matching recovery record prove persistence; adapter output or a user repair note cannot. The failed run and unknown external effects remain historical facts. -Startup and periodic recovery discover these obligations through the retained -workspace index. Two background exports per server keep unrelated recovery -responsive. Automatic file recovery has three total attempts with a 30-second -delay. An idempotent board request through +Startup and the retained-workspace sweep reconstruct these obligations through +the retained workspace index. Committed Retry, recovery admission, and recovery +progress writes wake a separate preservation worker. It follows the persisted +retry deadline and becomes idle when no eligible recovery remains. Two background +exports per server keep unrelated execution finalization responsive. A resolved +exact-source files-only obligation remains eligible; unrelated resolved or +cancelled recovery history does not keep this worker active. The persisted +attempt budget applies across restarts. An idempotent board request through `POST /api/issues/:id/recovery-actions/recover-workspace` admits one additional attempt after a transient failure. It requires task mutation access, accepts no infrastructure certification, and grants no agent or provider execution authority. -Restarts do not reset the attempt limit. Missing stop or descriptor proof keeps +Each completed export frees its own slot and wakes the next scan. A slow export +does not hold a free slot. Completed cleanup and saved-input obligations receive +one bounded pass through unvisited sources. The pass stops at its tail so paused +input does not keep the worker active; a committed change starts a fresh pass. +Shutdown joins this worker's exports before closing the database. Repeated Retry +during physical preservation binds the canonical intent +without changing the attempt's generation or revision. Progress writes preserve +that newer intent so its fenced completion can still commit. Missing stop or descriptor proof keeps the source allocated and exposes the specific service blocker. File recovery alone never requests another agent turn. Existing saved human @@ -1995,6 +2022,25 @@ not grant execution authority: calls still use the existing gateway policy, approval, revocation, and audit paths. +Native OpenCode sessions use the server-owned assigned-tool relay for local and +remote execution. The provider receives only bounded search and call schemas, +even when its protocol can fit the full catalog. This bounds provider grammar +compilation independently of the transport's tool-count and byte limits. Search +returns pinned tool aliases and schemas, intersects fresh grants, and respects +planning/ask restrictions. OpenCode discovery pages use a 32 KiB budget; larger +schemas use 16K-character chunks so JSON escaping and the result envelope fit +its HTTP MCP bridge. Other harnesses keep the existing PRP discovery budgets. +Calls retain the original gateway binding and its +argument validation, approvals, revocation, and audit rules; prompt arguments +cannot choose a gateway or credential. Assigned gateway launch credentials stay +on the controller. The MCP bridge preserves exact declared operation names, +including names that start with `paperclip_`; only a catalog-bound transport +prefix may resolve to that declared name. Shortened or unknown names do not +gain authority. OpenCode sessions with the former direct catalog rotate their +provider checkpoint before reuse; other providers retain their existing tool +contract fingerprints. + + ### Use real connection requests (2026-09-14) When a user asks to connect a known service, the agent searches for that service diff --git a/doc/task-recovery.md b/doc/task-recovery.md index 870b8299a34..714d921a97b 100644 --- a/doc/task-recovery.md +++ b/doc/task-recovery.md @@ -15,8 +15,26 @@ new generation; a repeated HTTP request cannot. The existing exact-run retry endpoint retains its source-bound contract. Connected conversations use their existing inbound authorization owner. A board -Retry changes the workspace strategy, not the external principal, source batch, -conversation access or publication authority. +Retry preserves external principal, conversation access and publication checks. +A GitHub Retry selects already-saved, undelivered input before replaying the +failed run's older request. The selected receipts transfer to the retry owner +under the dispatch transaction. They link to its one successor, including after +a restart or a lost dispatch response. Rejected, unadmitted retries release the +saved input for another deliberate Retry. + +A deliberate GitHub Retry can admit saved input after the same App reconnects. +It verifies the App binding in the accepted delivery or its historical working +comment receipt, then checks current sender, repository, conversation and task +access. The retry records the current runtime generation and credential-reference +fingerprint. Another reconnect invalidates that admission. Historical deliveries, +stale webhook callbacks and the exact-run retry contract retain their fences. + +Task Retry keeps the complete saved-input batch, including batches larger than +the exact-run endpoint's limit. Source rows, receipts and invalidations are read +in batches. Within one transaction, messages with the same principal and automatic +policy context share a current-access check; every message still proves its App, +destination, comment link and immutable source. Different automatic policy contexts +receive separate access checks. Input in another conversation remains queued. A connected-conversation Retry can save its exact request while the matching system recovery generation verifies the old executor. Saving that wait grants diff --git a/maps/agent-access-map/recovery.md b/maps/agent-access-map/recovery.md index e0c6003c9f2..7f212f4a3d2 100644 --- a/maps/agent-access-map/recovery.md +++ b/maps/agent-access-map/recovery.md @@ -4,6 +4,17 @@ Product map: [Recovery](../feature-map/recovery.md). Feature ID: `recovery`. +Native finalization deadline reconciliation records unfinished workspace +preservation through the existing server-owned recovery path. It grants no +additional agent API, provider-stop authority or permission to replay work. + +Ordinary board Retry can reauthorize saved GitHub input for the same App after +reconnect. It pins current runtime authority and preserves sender, repository, +conversation and task checks. Pending input takes precedence over the older +failed request and transfers once to the retry owner. This grants no new agent +Retry API or stale webhook authority. Local PostgreSQL and HTTP-route regressions +cover these transitions; hosted execution and delivery remain separate acceptance. + Source revision: `3bbc2104b65caf9f7763938a01260a5673e12f7a`. Review depth and executed proof are separate. The operation records enumerate the named behavior and access boundaries. Their evidence states describe the reviewed sources; run results and environment limits belong in the PR or verification artifacts. A listed API or source test does not establish a live authorized journey. Follow the [map contract](../README.md). @@ -225,6 +236,13 @@ Gaps: An acknowledged retry can remain pending or fail; report actual useful res Capability ID: `recovery/workspace`. +Committed authorized Retry and workspace-recovery requests notify the internal +preservation worker. Startup reconstructs its source-bound obligations and saved +retry deadlines. A duplicate request during physical preservation does not +replace the owner's generation or revision. Resolved files-only recovery retains +its no-replay restriction. The local PostgreSQL regression checks the producer, +worker, and physical source receipt; it is not deployed provider acceptance. + The internal preservation resume path grants no board or agent execution API. It requires the exact source action, company, run, lease and environment, a held physical workspace owner, and its current stop-only cleanup claim. The diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index a873db39633..4c732929e7e 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -162,6 +162,18 @@ No grant, token, authority cache or provider dispatch bypass is added. The [executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts) remain separate from live latency and permitted/denied agent acceptance. +Scoped OpenCode assigned-tool source evidence: local and remote native sessions +use the server-owned authenticated relay. The provider sees bounded +`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool` schemas, +then uses pinned aliases. Search intersects fresh grants and work modes; calls +retain the run's original gateway policy, validation, approvals, revocation and +audit path. Prompt arguments cannot select gateway authority. Assigned gateway +launch credentials stay on the controller. Former direct-catalog OpenCode +checkpoints refresh; other provider fingerprints stay unchanged. The +[projection controls](../../server/src/services/native-runtime/assigned-mcp-tools.test.ts) +and [wiring controls](../../server/src/services/native-runtime/native-session-executor.test.ts) +are source evidence, not live permitted/denied sandbox acceptance. + For a linked Codex subscription with server-resolved managed credential provenance, setup checks the resolved owner and stages a private probe copy before local or remote isolation. Without a selected API key, an unbound linked `CODEX_HOME` is diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index bb3b705cd1d..e19eb2205b6 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -128,6 +128,14 @@ from the hiring picker; API fixture creation does not prove this UI journey. Automated: [instruction service](../../server/src/__tests__/agent-instructions-service.test.ts) and [instruction revisions](../../server/src/__tests__/agent-instruction-revisions.test.ts) cover file/revision contracts. +The [AI account field regression](../../ui/src/components/ai-connections/AiConnectionField.test.tsx) +checks duplicate provider names, account identity in choices and adoption confirmation, +the exact selected grant, and disabled unavailable accounts with authorized metadata fixtures. +Manually compare a personal API key and shared subscription with the same name in +Harness / Runtime. Confirm their method, scope and available identity before saving. +Reload the saved choice and verify the next eligible task uses it; labels alone +do not establish credential validity or change account access. + Manual: Save a harmless instruction edit, reload, and verify its revision and effect in the next eligible run. Pause and resume the disposable agent; confirm pause prevents new work and does not erase previous runs. For failed setup, use an unavailable model or invalid command on a disposable agent. diff --git a/maps/feature-map/recovery.md b/maps/feature-map/recovery.md index a6f49522fef..8514cd446be 100644 --- a/maps/feature-map/recovery.md +++ b/maps/feature-map/recovery.md @@ -33,6 +33,8 @@ Agent access: [companion](../agent-access-map/recovery.md). workspace link when the destination is disabled. Local PostgreSQL and real filesystem regressions cover native and legacy recovery and one-time delivery; hosted provider recovery remains separate acceptance. + Native finalization deadline failures also retain the exact prepared source + for the same stop-and-save workflow before Retry admits another executor. - `unknown-outcome`: `recovery/unknown-outcome` — uncertain external side effects require inspection and an explicit decision, not automatic replay. - `user-continuation`: `recovery/user-continuation` — saved user messages after a stop are reconsidered once @@ -89,6 +91,14 @@ The connected-chat database regression in exercises admission and queue continuation. It does not establish hosted provider recovery or successful delivery. +For GitHub, save a follow-up behind a failed run, then reconnect the same App. +Click ordinary Retry and verify that one successor receives the saved follow-up +instead of replaying the older request. Reload and repeat the same request key; +there must be one successor and no duplicate publication. A second reconnect +after admission, changed App, revoked sender or changed repository must prevent +dispatch. An unadmitted rejection keeps the follow-up available for another +deliberate Retry. Historical deliveries must remain unchanged. + ### `post-stop-message` After a stop, send a new instruction on the same task or inspect one already @@ -175,6 +185,13 @@ provider restart. For that journey, interrupt a disposable deployment after workspace preparation and before its first executor claim, then confirm one provider turn starts on the original run through the normal admission gates. +The [Runner transport regressions](../../packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs) +exercise a real WebSocket welcome delayed beyond two seconds. Authentication +uses the ten-second connection setup budget, clipped by the current lifecycle +deadline. The cases also retain failed-welcome ticket fencing, identity checks, +and reconnect replay without repeated provider effects. These transport cases +do not establish hosted provider acceptance. + Active remote process supervision distinguishes a failed health read from a confirmed exit. The [executor tests](../../server/src/services/native-runtime/native-session-executor.test.ts) exercise timeout recovery with a real listening process, unchanged ownership, @@ -242,6 +259,17 @@ gap. ### `operator-retry` +The [workspace recovery regression](../../server/src/services/workspace-recovery.test.ts) +uses the real Retry producer and a quiet delivery coordinator. It checks that a +committed request starts one physical source preservation, duplicate Retry keeps +the active owner's receipt valid, and startup reconstructs a resolved files-only +obligation without provider replay. The [execution-control lifecycle suite](../../server/src/__tests__/execution-control-work.test.ts) +checks independent finalization, stored deadlines, and a bounded pass through +completed obligations followed by quiet paused input. +The physical export regression holds one export while another completes, then +verifies a third Retry can use the free slot and shutdown joins the held work. +These local checks do not prove recovery of a deployed allocation. + Automated: [disposition notices](../../ui/src/components/DispositionRecoveryNotice.test.tsx) tests pending/error/gated states and waiting for the real retry acknowledgement. The recovery-card suite covers workspace divergence, reissue, reconcile, and diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index c44576823c7..32c7e51ae6a 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -145,6 +145,16 @@ is released before failure returns. Repeat a warm follow-up and retain its actua reply time. The [executor controls](../../server/src/services/native-runtime/native-session-executor.test.ts) cover ordering, failure drain and credential projection; they do not qualify a live response-time target. +For native OpenCode with assigned connections, inspect the provider tool surface +on both local and sandbox execution. It should contain bounded assigned-tool +search/call schemas. Discover an allowed tool, call its returned alias, revoke +its grant, and confirm discovery and execution enforce that revocation. Repeat +in planning/ask mode with a write tool and confirm denial. The controller must +retain the assigned gateway URL and token. An older direct-catalog OpenCode +checkpoint must refresh without changing another provider's fingerprint. The +[assigned-tool](../../server/src/services/native-runtime/assigned-mcp-tools.test.ts) +and executor controls cover source behavior; live sandbox discovery and a real +call remain separate acceptance evidence. Missing dependencies, failed authentication, invalid models and incompatible custom settings must remain actionable errors without silent runner switching. Saved unchanged execution can still be tested without applying new diff --git a/maps/feature-map/tasks.md b/maps/feature-map/tasks.md index 1108526dfb0..35599105a4b 100644 --- a/maps/feature-map/tasks.md +++ b/maps/feature-map/tasks.md @@ -39,7 +39,7 @@ Open `/issues` or a project’s Issues view; legacy `/tasks` links redirect. ### `task-detail` -Open `/issues/:issueId` from a row, mention, or direct link. Cold opens show the instruction and recent exchange first. **More messages** loads earlier messages and run activity; incoming messages remain visible while the task stays open. +Open `/issues/:issueId` from a row, mention, or direct link. Cold opens show the instruction and recent exchange first. **More messages** appears only when earlier comments or runs are omitted and loads that history. Fresh tasks with no omitted history have no disclosure. A single legacy run with a truncated log offers **More run output** instead. **More task actions → Show task activity** loads full activity even for a single native run; incoming messages remain visible while the task stays open. ### `task-sharing` diff --git a/packages/adapters/codex-local/src/ui/build-config.test.ts b/packages/adapters/codex-local/src/ui/build-config.test.ts index 0557e3c6a57..10c62130ef8 100644 --- a/packages/adapters/codex-local/src/ui/build-config.test.ts +++ b/packages/adapters/codex-local/src/ui/build-config.test.ts @@ -36,6 +36,11 @@ function makeValues(overrides: Partial = {}): CreateConfigVa } describe("buildCodexLocalConfig", () => { + it("preserves explicitly disabled hosted search in native Codex configuration", () => { + expect(buildPaperclipRunnerConfig(makeValues({ search: false }))).toHaveProperty("search", false); + expect(buildPaperclipRunnerConfig(makeValues({ search: true }))).not.toHaveProperty("search"); + }); + describe("qualified Copilot config", () => { it("round-trips Copilot with the exact selected model", () => { const first = buildPaperclipRunnerConfig(makeValues({ model: "gpt-5.6-luna", adapterSchemaValues: { provider: "acpx", acpxAgent: "copilot" } })); @@ -171,7 +176,6 @@ describe("buildPaperclipRunnerConfig", () => { "engine", "agentCommand", "stateDir", - "search", "fastMode", "dangerouslyBypassApprovalsAndSandbox", "command", diff --git a/packages/adapters/codex-local/src/ui/build-config.ts b/packages/adapters/codex-local/src/ui/build-config.ts index b23172938e9..73039f2cc10 100644 --- a/packages/adapters/codex-local/src/ui/build-config.ts +++ b/packages/adapters/codex-local/src/ui/build-config.ts @@ -249,6 +249,7 @@ export function buildPaperclipRunnerConfig(v: CreateConfigValues): Record error, + Ok(_) => panic!("welcome authenticated after the lifecycle deadline"), + }; + assert!(failure.bootstrap_maybe_consumed); + assert!(started.elapsed() < Duration::from_secs(2)); + assert!(failure.error.to_string().contains("deadline elapsed")); + } + #[test] fn authenticates_bootstrap_and_receives_bound_lease() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index eb6c82cefcb..64dc17a0c88 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", + "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs index 9b2fbe2b089..0a23b4d036a 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs @@ -8,6 +8,7 @@ import { writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; +import { createRequire } from 'node:module'; const runtime = process.env.PAPERCLIP_CONTROLLER_TEST_RUNTIME ? pathToFileURL(resolve(process.env.PAPERCLIP_CONTROLLER_TEST_RUNTIME) + '/') : import.meta.url; const { canonical, costRuntime, observabilityBoundConfiguration, requestBindingSha256, sha256, validateManifest } = await import(new URL('./manifest.mjs', runtime)); @@ -18,6 +19,8 @@ const { readRegularFile, replaceEvidenceFile } = await import(new URL('./evidenc import { aggregateProtocolEvalCampaign, sanitizeProtocolEvalRuns } from '../runner-protocol-eval-campaign.mjs'; +const { build } = createRequire(new URL('../../../../package.json', import.meta.url))('esbuild'); + const childScript = options => ` import assert from 'node:assert/strict'; import {createHash} from 'node:crypto'; @@ -54,10 +57,7 @@ if(args.has('--observability-config')) { } nativeObservability={mode:'controller_otlp',configured:true,verification:'unverified',projectId:config.projectId,traceId:config.traceId,spanId:config.spanId,descriptorSha256:createHash('sha256').update(raw).digest('hex')}; if(${Boolean(options.realObservabilityValidator)}) { - const {createRequire}=await import('node:module'); - const require=createRequire(${JSON.stringify(fileURLToPath(new URL('../../../../server/package.json', import.meta.url)))}); - const {tsImport}=await import(require.resolve('tsx/esm/api')); - const {prepareEvalObservabilityRuntime}=await tsImport(${JSON.stringify(fileURLToPath(new URL('../../src/cli/eval-observability-runtime.ts', import.meta.url)))},import.meta.url); + const {prepareEvalObservabilityRuntime}=await import(${JSON.stringify(options.observabilityValidatorModule ?? '')}); await mkdir(process.env.HOME+'/codex',{recursive:true}); const prepared=await prepareEvalObservabilityRuntime(args.get('--observability-config'),request, {evidence:{campaignId:descriptor.campaignId,executionId:descriptor.executionId},transportOptions:{environment:{CODEX_HOME:process.env.HOME+'/codex'}}},process.env); @@ -90,6 +90,11 @@ async function fixture(t, { count = 1, allocation = '0.01', providerProfile = 'c const root = await mkdtemp(join(tmpdir(), 'runner-controller-test-')); t.after(() => rm(root, { recursive: true, force: true })); const cli = join(root, 'fake-cli.mjs'), binary = join(root, 'runnerd'), archive = join(root, 'runner.tgz'); + if (options.realObservabilityValidator) { + options.observabilityValidatorModule = join(root, 'eval-observability-runtime.mjs'); + await build({ entryPoints: [fileURLToPath(new URL('../../src/cli/eval-observability-runtime.ts', import.meta.url))], + outfile: options.observabilityValidatorModule, bundle: true, platform: 'node', format: 'esm', target: 'node24', logLevel: 'silent' }); + } await writeFile(cli, childScript(options)); await writeFile(binary, 'synthetic daemon'); await writeFile(archive, 'synthetic package'); const provenance = async path => ({ path, sha256: sha256(await readFile(path)) }); const now = Date.now(), verifiedAt = new Date(now - 1000).toISOString(), expiresAt = new Date(now + 86_400_000).toISOString(); diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index 7f43d83e8af..354f06c837a 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v59-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v61-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 59 }; +const declaration = { ...historical.declaration, agentProfileVersion: 61 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/backends/codex-native-backend.ts b/packages/paperclip-runner/src/backends/codex-native-backend.ts index 2c223c9d9db..0bb809e49c6 100644 --- a/packages/paperclip-runner/src/backends/codex-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-native-backend.ts @@ -154,6 +154,8 @@ function createTransportBackedNativeSessionBackend( return new HarnessDriverBackend( new CodexAppServerDriver({ ...(input.provider.model ? { model: input.provider.model } : {}), + ...(input.provider.kind === "codex" && "webSearch" in input.provider && input.provider.webSearch === "disabled" + ? { disableWebSearch: true } : {}), ...(input.provider.kind === "codex" && "reasoningEffort" in input.provider && input.provider.reasoningEffort ? { reasoningEffort: input.provider.reasoningEffort } : {}), diff --git a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts index 6838f56cdfb..d6366addb8b 100644 --- a/packages/paperclip-runner/src/backends/native-backend-factory.test.ts +++ b/packages/paperclip-runner/src/backends/native-backend-factory.test.ts @@ -2,7 +2,8 @@ import { QUALIFIED_ACPX_PROFILES, resolveQualifiedAcpxProfile } from "../drivers import { mkdtempSync, readFileSync, writeFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import * as codexSecurity from "../drivers/codex/codex-security-config.js"; import { buildNativeModelEnvelope, parseNativeExecutionInput, type NativeExecutionInput } from "../contracts/native-execution.js"; import { NATIVE_RUNTIME_ASSET_SCHEMA, PAPERCLIP_EXECUTION_PROMPT, PAPERCLIP_EXECUTION_PROMPT_REVISION, nativeRuntimePromptDigest, canonicalNativeRuntimeContextDigest, composeNativeSystemInstructions } from "../contracts/runtime-context.js"; @@ -210,6 +211,26 @@ function managedExecution( } describe("native backend factory", () => { + it.each([undefined, "disabled"] as const)("honors saved webSearch=%s with the default process transport", async webSearch => { + const input = preparedExecution(execution()); + if (input.provider.kind !== "codex") throw new Error("Codex fixture required"); + if (webSearch) Object.assign(input.provider, { webSearch }); + const buildArgs = codexSecurity.createIsolatedCodexAppServerArgs; + let generated: string[] | undefined; + const spy = vi.spyOn(codexSecurity, "createIsolatedCodexAppServerArgs").mockImplementation((...args) => { + generated = buildArgs(...args); + throw new Error("offline-stop-before-process-spawn"); + }); + try { + const backend = createNativeSessionBackend(input, { environment: {} }); + await expect(backend.openSession({ identity: { runId: "run", sessionId: "session", companyId: "company", agentId: "agent" }, + workingDirectory: WORKSPACE })).rejects.toThrow("offline-stop-before-process-spawn"); + expect(generated).toBeDefined(); + if (webSearch) expect(generated?.slice(-3)).toEqual(["-c", 'web_search="disabled"', "app-server"]); + else expect(generated).not.toContain('web_search="disabled"'); + } finally { spy.mockRestore(); } + }); + it("constructs the Codex backend without starting its transport", async () => { const backend = createNativeSessionBackend(execution(), { codexTransportFactory: () => { @@ -637,6 +658,7 @@ describe("native backend factory", () => { ["cursor", "../../test/fixtures/cursor-acp/profile-v7-identity.json"], ["cursor", "../../test/fixtures/cursor-acp/profile-v10-identity.json"], ["copilot", "../../test/fixtures/copilot-profile-v14-identity.json"], + ["copilot", "../../test/fixtures/copilot-profile-v60-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v13-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v14-identity.json"], ["pi", "../../test-fixtures/pi-acp/profile-v18-identity.json"], diff --git a/packages/paperclip-runner/src/contracts/native-execution.test.ts b/packages/paperclip-runner/src/contracts/native-execution.test.ts index b161675f19f..9a3edcc531d 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.test.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.test.ts @@ -194,6 +194,13 @@ describe("NativeExecutionInputV1", () => { }); expect(withEffort.provider).toMatchObject({ kind: "codex", reasoningEffort: "ultra" }); expect(parseNativeExecutionInput(withEffort)).toEqual(withEffort); + const disabledSearch = parseNativeExecutionInput({ ...withEffort, provider: { ...withEffort.provider, webSearch: "disabled" } }); + expect(disabledSearch.provider).toHaveProperty("webSearch", "disabled"); + expect(parseNativeExecutionInput(JSON.parse(JSON.stringify(disabledSearch)))).toEqual(disabledSearch); + for (const webSearch of [true, false, "live", "cached", null]) { + expect(() => parseNativeExecutionInput({ ...withEffort, provider: { ...withEffort.provider, webSearch } })).toThrow("webSearch"); + } + expect(() => parseNativeExecutionInput({ ...withEffort, provider: { kind: "codex", model: "gpt-6-astra", approvalPolicy: "on-request", reasoningEffort: "impossible" }, diff --git a/packages/paperclip-runner/src/contracts/native-execution.ts b/packages/paperclip-runner/src/contracts/native-execution.ts index 295e44c89b6..d6d5993e528 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.ts @@ -160,7 +160,7 @@ export type NativeProviderConfigV4 = export type NativeProviderConfigV5 = | Exclude - | (Extract & { reasoningEffort?: string }); + | (Extract & { reasoningEffort?: string; webSearch?: "disabled" }); export interface NativeExecutionInputV1 { schema: typeof NATIVE_EXECUTION_INPUT_SCHEMA_V1; @@ -571,7 +571,7 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput : provider.kind === "acpx" ? ["kind", "agent", "model", isV4 ? "permissionMode" : "permissionPolicy", "profile", "piThinkingLevel", ...(isV4 ? ["mode"] : []), ...(isV9 ? ["claudeEffort"] : [])] : provider.kind === "codex" && isV4 - ? ["kind", "model", "approvalPolicy", ...(isV5 ? ["reasoningEffort"] : [])] + ? ["kind", "model", "approvalPolicy", ...(isV5 ? ["reasoningEffort", "webSearch"] : [])] : provider.kind === "opencode" && isV4 ? ["kind", "model", "permissionMode"] : ["kind", "model"], @@ -780,6 +780,9 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput if (isV5 && provider.reasoningEffort !== undefined && !["minimal", "low", "medium", "high", "xhigh", "max", "ultra"].includes(String(provider.reasoningEffort))) { throw new NativeExecutionInputError("input.provider.reasoningEffort is unsupported"); } + if (isV5 && provider.webSearch !== undefined && provider.webSearch !== "disabled") { + throw new NativeExecutionInputError("input.provider.webSearch must be disabled when specified"); + } parsedProvider = { kind: "codex", model: providerModel, @@ -789,6 +792,7 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput ...(isV5 && provider.reasoningEffort !== undefined ? { reasoningEffort: text(provider.reasoningEffort, "input.provider.reasoningEffort") } : {}), + ...(isV5 && provider.webSearch === "disabled" ? { webSearch: "disabled" as const } : {}), }; } if (!Array.isArray(contract.criteria) || contract.criteria.length === 0) { diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index ff4812fcb94..bcbc3e6f243 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -13,6 +13,8 @@ import { } from "node:fs"; import { connect, type Socket } from "node:net"; import nodeFs from "node:fs"; +import fsPromises from "node:fs/promises"; +import { createServer } from "node:http"; import { syncBuiltinESMExports } from "node:module"; import { tmpdir } from "node:os"; import { resolve } from "node:path"; @@ -47,6 +49,117 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it.each(["event", "newer_command", "replacement", "write_failure", "drain", "semantic_replacement"] as const)( + "keeps HTTP responsive during a durable event write and preserves %s", async (mode) => { + const root = mkdtempSync(resolve(tmpdir(), "runner-event-io-")); + const dispatch = vi.fn(async () => { + expect(JSON.parse(readFileSync(resolve(root, "control-plane-state.json"), "utf8")).ackedSourceSeq).toBe(1); + return { result: { ok: true } }; + }); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, + expectedRunnerVersion, expectedRunnerDigest, onCommittedEvent: async () => {}, onSemanticToolInput: dispatch }); + const http = createServer((_req, res) => res.end("responsive")); + let release!: () => void; + const barrier = new Promise(resolveWrite => { release = resolveWrite; }); + const originalOpen = fsPromises.open; + let staged = false; + let holdFirst = true; + let client: AuthenticatedClient | null = null; + let replacement: AuthenticatedClient | null = null; + const open = vi.spyOn(fsPromises, "open").mockImplementation(async (...args) => { + const file = await originalOpen(...args); + if (holdFirst && String(args[0]).includes(".control-plane-state.json.")) { + holdFirst = false; + const sync = file.sync.bind(file); + file.sync = async () => { + staged = true; + await barrier; + if (mode === "write_failure") throw new Error("controlled disk failure"); + await sync(); + }; + } + return file; + }); + try { + await core.start(); + await new Promise(resolveListen => http.listen(0, "127.0.0.1", resolveListen)); + client = (await authenticate(core, core.issueBootstrapTicket()))!; + const event = semanticInputEvent(); + const payload = event.payload as Record; + if (mode !== "semantic_replacement") { + payload.eventType = "item.delta"; + payload.priority = 2; + payload.payload = { delta: "output while the volume is slow" }; + } + expect(validatePrpEvent(payload)).toMatchObject({ ok: true }); + sendSecure(client, event); + await vi.waitFor(() => expect(staged || dispatch.mock.calls.length > 0).toBe(true)); + if (mode === "semantic_replacement") { + replacement = (await authenticate(core, client.leaseToken!))!; + release(); + await vi.waitFor(() => expect(dispatch).toHaveBeenCalledTimes(1)); + await vi.waitFor(() => expect(core.store.state.commands.some(command => command.type === "semantic_tool.result")).toBe(true)); + return; + } + // The actual socket request completes while the real file's fsync is held. + const address = http.address() as { port: number }; + expect(await (await fetch(`http://127.0.0.1:${address.port}`)).text()).toBe("responsive"); + expect(JSON.parse(readFileSync(core.store.path, "utf8")).ackedSourceSeq).toBe(0); + if (mode === "newer_command") core.queueCommand("session.snapshot", {}, "command_during_event"); + let drained = false; + const retirement = mode === "drain" ? (async () => { + await core.stop(); + await core.drainPendingConnectionProcessing(); + drained = true; + })() : null; + if (retirement) { + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(drained).toBe(false); + } + if (mode === "replacement") { + replacement = (await authenticate(core, client.leaseToken!))!; + const second = structuredClone(event); + const next = second.payload as Record; + next.sourceSeq = 2; + next.sourceEventId = "output-event-2"; + sendSecure(replacement, second); + expect(await receiveSecure(replacement)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 2 } }); + } + release(); + if (retirement) { + expect(await receiveSecure(client)).toBeNull(); + await retirement; + expect(drained).toBe(true); + } else if (mode === "write_failure") { + expect(await receiveSecure(client)).toBeNull(); + expect(() => core.queueCommand("session.snapshot")).toThrow("indeterminate"); + } else if (mode !== "replacement") { + expect(await receiveSecure(client)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 1 } }); + } + await core.stop(); + await core.drainPendingConnectionProcessing(); + const restored = new DurablePrpControlPlane({ stateDirectory: root, identity, + expectedRunnerVersion, expectedRunnerDigest }); + const expectedEvents = mode === "replacement" ? 2 : mode === "write_failure" ? 0 : 1; + expect(restored.store.state.ackedSourceSeq).toBe(expectedEvents); + expect(restored.store.state.committedEvents).toHaveLength(expectedEvents); + if (mode === "newer_command") { + expect(restored.store.state.commands.some(command => command.commandId === "command_during_event")).toBe(true); + } + expect(nodeFs.readdirSync(root).filter(name => name.endsWith(".tmp"))).toEqual([]); + } finally { + release(); + open.mockRestore(); + client?.socket.destroy(); + replacement?.socket.destroy(); + await core.stop(); + await core.drainPendingConnectionProcessing(); + await new Promise(resolveClose => http.close(() => resolveClose())); + rmSync(root, { recursive: true, force: true }); + } + }, +); + it("commits credential updates before ACK and preserves replay identity", async () => { const root = mkdtempSync(resolve(tmpdir(), "runner-credential-update-")); let release!: () => void; diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 2556c47f48c..2ce1af85df8 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -23,12 +23,13 @@ import { renameSync, unlinkSync, writeFileSync, - type Stats, } from "node:fs"; import { createServer, type IncomingMessage, type Server } from "node:http"; import { dirname, resolve } from "node:path"; import type { Duplex } from "node:stream"; import { fileURLToPath } from "node:url"; +import { withStagedPrivateWrite } from "./staged-private-write.js"; +import { verifyPrivateRegularFile } from "./private-state-file.js"; import { NativeSessionProtocolIntegrityError } from "../contracts/native-session-backend.js"; import { ACPX_CREDENTIAL_BINDING_ENV, ACPX_CREDENTIAL_NAMES, CLAUDE_ROUTING_ENV_KEYS, createAcpxSidecarHostEnvironment } from "../drivers/acpx/environment.js"; @@ -1122,22 +1123,6 @@ function verifyPrivateDirectory(path: string): void { } } -function verifyPrivateRegularFile(file: Stats, path: string): void { - if (!file.isFile()) { - throw new Error(`Private state path is not a regular file: ${path}`); - } - if (process.platform !== "win32") { - if ((file.mode & 0o777) !== 0o600) { - throw new Error(`Private state file does not use mode 0600: ${path}`); - } - if (process.geteuid !== undefined && file.uid !== process.geteuid()) { - throw new Error( - `Private state file is not owned by the daemon user: ${path}`, - ); - } - } -} - export function readPrivateFile(path: string): string | null { let descriptor: number; try { @@ -1218,6 +1203,8 @@ class DurableCoreStore { readonly path: string; #state: StoredCoreState; #writeIndeterminate = false; + #revision = 0; + #savedRevision = 0; constructor(directory: string, identity: DurableRecoveryIdentity) { try { @@ -1255,7 +1242,33 @@ class DurableCoreStore { save(): void { this.assertWritable(); - atomicPrivateWrite(this.path, `${JSON.stringify(this.#state, null, 2)}\n`); + const revision = ++this.#revision; + try { + atomicPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`); + this.#savedRevision = revision; + } catch (error) { + this.#writeIndeterminate = true; + throw error; + } + } + + async saveEvent(): Promise { + this.assertWritable(); + const revision = ++this.#revision; + try { + await withStagedPrivateWrite(this.path, `${JSON.stringify(this.#state)}\n`, (temporary) => { + this.assertWritable(); + // A command or replacement connection can persist this event and newer + // authority while its file is staged. Never roll that state back. + if (this.#savedRevision >= revision) return; + renameSync(temporary, this.path); + syncParentDirectory(this.path); + this.#savedRevision = revision; + }); + } catch (error) { + this.#writeIndeterminate = true; + throw error; + } } assertWritable(): void { @@ -1268,9 +1281,11 @@ class DurableCoreStore { /** Persist a complete candidate before publishing any new authority in memory. */ commit(candidate: StoredCoreState): void { this.assertWritable(); + const revision = ++this.#revision; try { - atomicPrivateWrite(this.path, `${JSON.stringify(candidate, null, 2)}\n`); + atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); this.#state = candidate; + this.#savedRevision = revision; } catch (error) { // Rename may already have succeeded before directory fsync failed. // Never overwrite that possibly durable receipt using stale memory. @@ -3292,7 +3307,14 @@ export class DurablePrpControlPlane { }); this.#store.state.ackedSourceSeq = sourceSeq; } - this.#store.save(); + // Output deltas have no lifecycle or control effects. Other events must + // become durable before transport readers can observe them in memory. + if (eventType === "item.delta") { + await this.#store.saveEvent(); + if (this.#eventOwner !== connection || this.#identity !== identity || this.#protocolIntegrityError !== null) return; + } else { + this.#store.save(); + } if ( isSemanticInput && diff --git a/packages/paperclip-runner/src/control-plane/private-state-file.ts b/packages/paperclip-runner/src/control-plane/private-state-file.ts new file mode 100644 index 00000000000..788cce5cbfd --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/private-state-file.ts @@ -0,0 +1,15 @@ +import type { Stats } from "node:fs"; + +export function verifyPrivateRegularFile(file: Stats, path: string): void { + if (!file.isFile()) { + throw new Error(`Private state path is not a regular file: ${path}`); + } + if (process.platform !== "win32") { + if ((file.mode & 0o777) !== 0o600) { + throw new Error(`Private state file does not use mode 0600: ${path}`); + } + if (process.geteuid !== undefined && file.uid !== process.geteuid()) { + throw new Error(`Private state file is not owned by the daemon user: ${path}`); + } + } +} diff --git a/packages/paperclip-runner/src/control-plane/staged-private-write.ts b/packages/paperclip-runner/src/control-plane/staged-private-write.ts new file mode 100644 index 00000000000..fed62c54ba4 --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/staged-private-write.ts @@ -0,0 +1,33 @@ +import { randomUUID } from "node:crypto"; +import { constants } from "node:fs"; +import fs from "node:fs/promises"; +import { basename, dirname, resolve } from "node:path"; +import { verifyPrivateRegularFile } from "./private-state-file.js"; + +/** Stage durable bytes off the event loop; the owner publishes them without yielding. */ +export async function withStagedPrivateWrite( + path: string, + contents: string, + publish: (temporary: string) => void, +): Promise { + const temporary = resolve(dirname(path), `.${basename(path)}.${randomUUID()}.tmp`); + const file = await fs.open(temporary, constants.O_WRONLY | constants.O_CREAT | + constants.O_EXCL | (constants.O_NOFOLLOW ?? 0), 0o600); + try { + try { + if (process.platform !== "win32") await file.chmod(0o600); + const metadata = await file.stat(); + verifyPrivateRegularFile(metadata, temporary); + await file.writeFile(contents, "utf8"); + await file.sync(); + } finally { + await file.close(); + } + publish(temporary); + } finally { + // Publication can either rename this file or retain a newer durable write. + await fs.unlink(temporary).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + }); + } +} diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 8bdf9b3950d..c1c175f2994 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,39 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(59); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(61); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3"); + }); + + it("preserves v60 while binding controller journal retirement in v61", async () => { + const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v60-identity.json", import.meta.url)); + expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("e3c45b0311517d05f06ed10f357db713ae157cfa87e4a8becf087c5b6dfa7003"); + const prior = JSON.parse(priorBytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v61-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 60)).toBe(true); + expect(current.commandDigest).not.toBe(prior.commandDigest); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 60, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + }); + + it("preserves v59 while binding the Codex-only hosted search restriction in v60", async () => { + const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v59-identity.json", import.meta.url)); + expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("a9116d0981cb0feebec31f61cd23d16b3a263fe6a797b1ba1436c6e41198538c"); + const prior = JSON.parse(priorBytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v60-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "nativeBackendSourceSha256", "nativeExecutionSourceSha256", "profileCompatibilitySourceSha256", "providerLaunchSourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 59)).toBe(true); + expect(current.commandDigest).not.toBe(prior.commandDigest); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 59, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("preserves v58 while binding the separate remote process reconnect policy in v59", () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 36f2bdf29e1..f52273f437c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 59, + "agentProfileVersion": 61, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:2f39ffe54a6af832d044b5098c31df9e08d4d27aee0ac675ba4b0c9da7445c40", + "commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 56a5878aa39..6a1888d273c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts index 3cb3164b5b7..7103c137c48 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts @@ -710,7 +710,7 @@ export class CodexAppServerDriver implements HarnessDriver { this.#options.transportFactory?.(context) ?? new ProcessCodexAppServerTransport({ workingDirectory, - args: createIsolatedCodexAppServerArgs(this.#options.environment, codexExecutableReadOnlyRoots(this.#options.environment ?? process.env), this.#options.instructionWorkingCopyRoot), + args: createIsolatedCodexAppServerArgs(this.#options.environment, codexExecutableReadOnlyRoots(this.#options.environment ?? process.env), this.#options.instructionWorkingCopyRoot, [], this.#options.disableWebSearch), environment: createSanitizedCodexEnvironment(this.#options.environment), onDiagnostic: message => this.#options.onDiagnostic?.(redactObservabilityDiagnostic(message, this.#options.environment, this.#options.observability?.otelConfig)), processGroup: true, diff --git a/packages/paperclip-runner/src/drivers/codex/codex-driver-types.ts b/packages/paperclip-runner/src/drivers/codex/codex-driver-types.ts index a9256351aa5..a9ba79f2162 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-driver-types.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-driver-types.ts @@ -19,6 +19,8 @@ export interface CodexAppServerDriverOptions { model?: string; /** Per-run reasoning effort sent with each Codex turn, including resumed turns. */ reasoningEffort?: string; + /** Saved hosted-search restriction for the default process transport. */ + disableWebSearch?: boolean; approvalPolicy?: "never" | "on-request" | "untrusted"; /** Paperclip runtime instructions. Codex receives these as additive developer * instructions; the historical option name remains compatible with callers. */ diff --git a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts index c5f15d2c5df..78d7d4d2f5d 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts @@ -254,6 +254,7 @@ export function createIsolatedCodexAppServerArgs( /** Server-registered run copy, never an environment/config-supplied root. */ instructionWorkingCopyRoot?: string, protectedPaths: string[] = [], + disableWebSearch = false, ): string[] { const explicitSource = source; source ??= process.env; @@ -356,6 +357,7 @@ export function createIsolatedCodexAppServerArgs( ...(externalRunnerSandbox ? ["--dangerously-bypass-approvals-and-sandbox"] : []), + ...(disableWebSearch ? ["-c", 'web_search="disabled"'] : []), "app-server", ]; } diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts index 07681400ddd..f281fd2a55a 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.test.ts @@ -94,6 +94,33 @@ describe("OpenCode MCP bridge", () => { expect(await conflict.json()).toMatchObject({ result: { isError: true, content: [{ text: "Duplicate call identity conflict." }] } }); }); + it("preserves declared prefixed operation IDs through HTTP MCP and rejects undeclared aliases", async () => { + const names = ["paperclip_search_assigned_tools", "paperclip_call_assigned_tool", "search_assigned_tools"]; + const handler = vi.fn(async call => ({ operation: call.tool })); + const bridge = await startOpenCodeMcpBridge({ tools: names.map(name => ({ name, inputSchema: { type: "object" } })), handler }); + bridges.push(bridge); + const listed = await (await rpc(bridge, { id: "list", method: "tools/list" })).json() as { result: { tools: Array<{ name: string }> } }; + expect(listed.result.tools.map(tool => tool.name)).toEqual([...names, "paperclip_finish", "paperclip_block"]); + for (const [i, name] of names.entries()) { + for (const [j, prefix] of ["", "paperclip_", "paperclip__", "paperclip."].entries()) { + const called = prefix === "paperclip_" && name === "search_assigned_tools" ? name : prefix + name; + const result = await (await rpc(bridge, { id: `call-${i}-${j}`, method: "tools/call", params: { name: called, arguments: {} } })).json(); + expect(result).toMatchObject({ result: { content: [{ text: JSON.stringify({ operation: name }) }] } }); + } + } + const before = handler.mock.calls.length; + expect(await (await rpc(bridge, { id: "unknown", method: "tools/call", params: { name: "paperclip_unassigned_tool", arguments: {} } })).json()).toMatchObject({ result: { isError: true } }); + expect(handler).toHaveBeenCalledTimes(before); + }); + + it("does not authorize a shortened operation when only its prefixed name is declared", async () => { + const handler = vi.fn(async () => ({ ok: true })); + const bridge = await startOpenCodeMcpBridge({ tools: [{ name: "paperclip_search_assigned_tools", inputSchema: { type: "object" } }], handler }); + bridges.push(bridge); + expect(await (await rpc(bridge, { id: "short", method: "tools/call", params: { name: "search_assigned_tools", arguments: {} } })).json()).toMatchObject({ result: { isError: true } }); + expect(handler).not.toHaveBeenCalled(); + }); + it("normalizes all supported OpenCode MCP prefixes", () => { expect(canonicalOpenCodeMcpToolName("paperclip_documents.read")).toBe("documents.read"); expect(canonicalOpenCodeMcpToolName("paperclip__documents.read")).toBe("documents.read"); diff --git a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts index 1bf1555dd16..cb1efd91922 100644 --- a/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts +++ b/packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts @@ -108,7 +108,7 @@ function normalizeTools( const tools: OpenCodeMcpToolDefinition[] = []; const seen = new Set(); for (const raw of input) { - const name = typeof raw.name === "string" ? canonicalOpenCodeMcpToolName(raw.name) : ""; + const name = typeof raw.name === "string" ? raw.name.trim() : ""; if (!name || seen.has(name)) continue; const inputSchema = isRecord(raw.inputSchema) ? structuredClone(raw.inputSchema) : { type: "object" }; tools.push({ @@ -218,7 +218,12 @@ async function handleRequest( } const params = isRecord(message.params) ? message.params : {}; const rawName = typeof params.name === "string" ? params.name : ""; - const tool = canonicalOpenCodeMcpToolName(rawName); + // A declared operation may itself start with paperclip_. Preserve that + // authority identity before accepting an OpenCode transport prefix alias. + const exactName = rawName.trim(); + const tool = context.admittedTools.some(entry => entry.name === exactName) + ? exactName + : canonicalOpenCodeMcpToolName(exactName); if (!context.admittedTools.some((entry) => entry.name === tool)) { writeRpc(response, id, { isError: true, content: [{ type: "text", text: "Unsupported tool." }] }); return; diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index 5d568eddf86..a44136bd364 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -2219,6 +2219,27 @@ describe("OpenCodeServerDriver", () => { } finally { await session.close({ reason: "monitor default test complete" }); } }); + it("keeps assigned discovery operation IDs at the authenticated MCP and controller boundary", async () => { + await chmod(fixture, 0o755); + const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-assigned-")); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-opencode-workspace-")); + roots.push(root, workspace); + const calls: string[] = []; + const operation = "paperclip_search_assigned_tools"; + const driver = new OpenCodeServerDriver({ model: "openrouter/deepseek/deepseek-v4.1-flash-1017", runtimeDirectory: root, command: fixture, + environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "fixture-key" }, + dynamicTools: [{ name: operation, inputSchema: { type: "object" } }], + dynamicToolHandler: async call => { calls.push(call.tool); if (call.tool !== operation) throw new Error("unauthorized operation"); return { tools: [] }; }, + }); + const session = await driver.openSession({ runId: "assigned-name", normalizedSessionId: "assigned-name", workingDirectory: workspace }); + try { + await session.startTurn({ message: { role: "user", text: "Discover assigned tools." } }); + const events = await collectTurnEvents(session.events()); + expect(calls).toEqual([operation]); + expect(events.some(event => event.eventType === "item.completed" && (event.payload as { item?: { is_error?: boolean } }).item?.is_error)).toBe(false); + } finally { await session.close({ reason: "assigned operation identity test" }); } + }); + it("normalizes a structured block result", async () => { await chmod(fixture, 0o755); const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-driver-")); diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index 4ab25b38b83..bf2a8bed9d5 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -1026,7 +1026,9 @@ class OpenCodeHarnessSession implements HarnessSession { callId: string; arguments: unknown; }): Promise { - const tool = canonicalOpenCodeMcpToolName(call.tool); + // The authenticated MCP bridge has resolved this against its closed + // declared catalog. A second prefix strip changes legitimate operation IDs. + const tool = call.tool; const turnId = this.#activeTurnId; if (turnId === null) throw new Error("OpenCode tool call is not bound to an active turn"); diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts index 5dd8b66c157..57f237bd57d 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import { createRunnerProviderLaunch, preparedRunnerLaunchWorkingDirectory, runnerLaunchRequirementsDigest, validatePreparedRunnerLaunch, - type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; + runnerProviderLaunchRequirements, type RunnerProviderLaunchEnvironment, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; const requirements: RunnerProviderLaunchRequirements = { schema: "paperclip.runner.launch-requirements.v1", @@ -19,6 +19,42 @@ const environment: RunnerProviderLaunchEnvironment = { }; describe("built-in provider launch preparation", () => { + it("binds disabled hosted search to the prepared launch and overrides execution-host arguments", () => { + const disabled = runnerProviderLaunchRequirements({ provider: "codex", disableCodexWebSearch: true }, + requirements.identity, { model: requirements.model }, null, ""); + expect(disabled.settings).toHaveProperty("disableCodexWebSearch", true); + expect(runnerLaunchRequirementsDigest(disabled)).not.toBe(runnerLaunchRequirementsDigest({ ...disabled, settings: {} })); + const launch = createRunnerProviderLaunch(disabled, { ...environment, + options: { codexArgs: ["-c", 'web_search="live"', "app-server"] } }); + expect(launch.args).toEqual(["-c", 'web_search="live"', "-c", 'web_search="disabled"', "app-server"]); + const prepared = { configurationId: "search-policy", requirementsDigest: runnerLaunchRequirementsDigest(disabled), provider: launch }; + expect(() => validatePreparedRunnerLaunch(disabled, prepared)).not.toThrow(); + expect(() => validatePreparedRunnerLaunch(disabled, { ...prepared, + provider: { ...launch, args: ["app-server"] } })).toThrow("runner_prepared_launch_requirements_mismatch"); + for (const leading of ["--", "--search", "--search=true"]) { + expect(() => createRunnerProviderLaunch(disabled, { ...environment, + options: { codexArgs: [leading, "app-server"] } })).toThrow("runner_codex_web_search_policy_override"); + expect(() => validatePreparedRunnerLaunch(disabled, { ...prepared, + provider: { ...launch, args: [leading, ...launch.args as string[]] } })).toThrow("runner_prepared_launch_requirements_mismatch"); + } + expect(() => validatePreparedRunnerLaunch(disabled, { ...prepared, + provider: { ...launch, args: ["--", ...launch.args as string[]] } })).toThrow("runner_prepared_launch_requirements_mismatch"); + for (const tail of [["-c", 'web_search="live"'], ['-cweb_search="live"'], ["--search"], ['--config=web_search="live"'], ["--"]]) { + expect(() => createRunnerProviderLaunch(disabled, { ...environment, + options: { codexArgs: ["app-server", ...tail] } })).toThrow("runner_codex_web_search_policy_override"); + expect(() => validatePreparedRunnerLaunch(disabled, { ...prepared, + provider: { ...launch, args: [...launch.args as string[], ...tail] } })).toThrow("runner_prepared_launch_requirements_mismatch"); + } + + }); + + it.each([undefined, false])("preserves default hosted search when disable is %s", disableCodexWebSearch => { + const request = runnerProviderLaunchRequirements({ provider: "codex", disableCodexWebSearch }, + requirements.identity, { model: requirements.model }, null, ""); + const launch = createRunnerProviderLaunch(request, environment); + expect(launch.args).toEqual(["app-server"]); + }); + it("keeps Core's selection and policy when the execution environment changes", () => { const before = structuredClone(requirements); const local = createRunnerProviderLaunch(requirements, environment); @@ -30,6 +66,17 @@ describe("built-in provider launch preparation", () => { expect(requirements).toEqual(before); }); + it("preserves exact Copilot requirements and launch descriptors when Codex search disabling is absent or false", () => { + const selected = { provider: "acpx" as const, acpxAgent: "copilot" as const, acpxPermissionMode: "deny-all" as const }; + const params = { model: "gpt-5.6-luna", approvalPolicy: "never", permissions: "paperclip-runner-workspace-read-only" }; + const baseline = runnerProviderLaunchRequirements(selected, requirements.identity, params, null, "Copilot policy"); + const scopedProbe = runnerProviderLaunchRequirements({ ...selected, disableCodexWebSearch: false }, + requirements.identity, params, null, "Copilot policy"); + expect(scopedProbe).toEqual(baseline); + expect(runnerLaunchRequirementsDigest(scopedProbe)).toBe(runnerLaunchRequirementsDigest(baseline)); + expect(createRunnerProviderLaunch(scopedProbe, environment)).toEqual(createRunnerProviderLaunch(baseline, environment)); + }); + it("preserves the ACPX profile, run identity, and permission policy", () => { const launch = createRunnerProviderLaunch({ ...requirements, provider: "acpx", model: "claude-selected", settings: { acpxAgent: "claude", acpxPermissionMode: "deny-all" } }, environment); diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.ts b/packages/paperclip-runner/src/live/runner-provider-launch.ts index 6abb9a00db0..5f2ae436d1e 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.ts @@ -23,7 +23,7 @@ export interface RunnerProviderLaunchRequirements { includeCollaborationModeInstructions: boolean; settings: Pick; + "piThinkingLevel" | "claudeEffort" | "managedProfile" | "agentCoreProfile" | "opencodePermissionMode" | "disableCodexWebSearch">; instructions: string; runtimeContext: NativeRuntimeContextSnapshot | null; } @@ -59,6 +59,7 @@ export function runnerProviderLaunchRequirements( piThinkingLevel: options.piThinkingLevel, claudeEffort: options.claudeEffort, managedProfile: options.managedProfile, agentCoreProfile: options.agentCoreProfile, opencodePermissionMode: options.opencodePermissionMode, + ...(options.disableCodexWebSearch === true ? { disableCodexWebSearch: true } : {}), }, approvalPolicy: params.approvalPolicy === "on-request" || params.approvalPolicy === "untrusted" ? params.approvalPolicy : "never", @@ -109,6 +110,14 @@ export function validatePreparedRunnerLaunch( if (descriptor.kind !== requirements.provider || descriptor.model !== requirements.model) reject(); if (requirements.provider === "codex" && (descriptor.approvalPolicy !== requirements.approvalPolicy || Boolean(descriptor.readOnly) !== requirements.readOnly)) reject(); + if (requirements.provider === "codex" && requirements.settings.disableCodexWebSearch === true) { + if (!Array.isArray(descriptor.args) || descriptor.args.some(arg => typeof arg !== "string")) reject(); + const args = descriptor.args as string[]; + const commandIndex = args.indexOf("app-server"); + const boundary = commandIndex < 0 ? args.length : commandIndex; + if (args.some(arg => arg === "--" || arg === "--search" || arg.startsWith("--search=")) || (commandIndex >= 0 && commandIndex !== args.length - 1) + || args[boundary - 2] !== "-c" || args[boundary - 1] !== 'web_search="disabled"') reject(); + } if (requirements.provider === "acpx") { const policy = descriptor.providerPolicy as Record | undefined; if (descriptor.agent !== (requirements.settings.acpxAgent ?? "codex") @@ -303,8 +312,9 @@ export function createRunnerProviderLaunch( args: provider === "opencode" ? [opencodeProxyPath] - : (input.options.codexArgs ?? - createRunnerdCodexAppServerArgs({ + : createRunnerdCodexAppServerArgs({ + args: input.options.codexArgs, + disableWebSearch: settings.disableCodexWebSearch, environment: input.options.environment, codexHome, workingDirectory: input.cwd, @@ -326,7 +336,7 @@ export function createRunnerProviderLaunch( ] : []), ], - })), + }), cwd: input.cwd, model: requirements.model, approvalPolicy: @@ -408,12 +418,14 @@ export function createRunnerdCodexAppServerArgs(input: { readOnlyRoots?: string[]; instructionWorkingCopyRoot?: string; disableProjectInstructions?: boolean; + disableWebSearch?: boolean; + args?: string[]; }): string[] { // The filesystem policy denies HOME and CODEX_HOME to keep credentials and // runner state outside provider reach. Always bind those names to the actual // isolated runner home; a stale controller environment must never cause the // execution workspace itself to become an explicit deny root. - const args = [ + const args = input.args ? [...input.args] : [ ...(input.disableProjectInstructions ? ["-c", "project_doc_max_bytes=0"] : []), ...createIsolatedCodexAppServerArgs( { @@ -426,8 +438,20 @@ export function createRunnerdCodexAppServerArgs(input: { usesPreconfiguredHome(input.environment) && input.runtimeStateRoot ? [input.runtimeStateRoot] : [], ), ]; - if (usesPreconfiguredHome(input.environment) && input.workingDirectory) { + if (!input.args && usesPreconfiguredHome(input.environment) && input.workingDirectory) { args.unshift("-c", `projects={${JSON.stringify(input.workingDirectory)}={trust_level="trusted"}}`); } + if (input.disableWebSearch) { + // Apply Core's restriction after host overrides. Keep the override before + // the Codex subcommand and after any earlier provider-home configuration. + const commandIndex = args.indexOf("app-server"); + // Restricted launches use the owned app-server argument shape. Refuse + // trailing subcommand flags and option terminators rather than guessing + // whether another CLI spelling could override or neutralize this policy. + if (args.some(arg => arg === "--" || arg === "--search" || arg.startsWith("--search=")) || (commandIndex >= 0 && commandIndex !== args.length - 1)) { + throw new Error("runner_codex_web_search_policy_override"); + } + args.splice(commandIndex < 0 ? args.length : commandIndex, 0, "-c", 'web_search="disabled"'); + } return args; } diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts index 71b37c8ff25..14e6351c9c9 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts @@ -27,6 +27,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) } async start() {} async stop() {} + async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} issueBootstrapTicket() { return {}; } diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts index 3e0c5314b78..1dcd3de3875 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts @@ -25,6 +25,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) } async start() {} async stop() {} + async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} issueBootstrapTicket() { return {}; } diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 47ac7a92091..d5a33c7853b 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -3343,6 +3343,29 @@ const fakeCodex = resolve( "../../runner/target/debug/fake-codex-app-server", ); +it("keeps the Copilot authentication launch requirements free of Codex search policy", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-copilot-probe-policy-")); + let captured: RunnerProviderLaunchRequirements | undefined; + const launch = vi.fn(() => { throw new Error("provider process must not start"); }); + try { + await expect(probeNativeRunnerEnvironment({ runtimeDirectory: root, provider: "acpx", model: "gpt-5.6-luna", + environment: {}, timeoutMs: 1_000, + transportOptions: { acpxAgent: "copilot", acpxCandidateProfile: "copilot", disableCodexWebSearch: true, + runnerArtifact: { version: "offline-fixture", digest: `sha256:${"a".repeat(64)}` }, + runnerProcessLauncher: launch, + controlPlaneRegistration: async () => { throw new Error("registration must not start"); }, + prepareLaunch: async requirements => { + captured = requirements; + throw new Error("offline-stop-at-Copilot-preparation"); + }, + }, + })).rejects.toThrow("offline-stop-at-Copilot-preparation"); + expect(captured).toBeDefined(); + expect(captured?.settings).not.toHaveProperty("disableCodexWebSearch"); + expect(launch).not.toHaveBeenCalled(); + } finally { await rm(root, { recursive: true, force: true }); } +}); + it.each(["stuck", "late-release", "late-release-error", "stuck-release", "cooperative-release", "launch-failure", "launch-failure-sandbox", "launch-failure-unsandboxed"] as const)( "bounds setup registration teardown and retains unconfirmed state (%s)", async mode => { @@ -3381,7 +3404,7 @@ it.each(["stuck", "late-release", "late-release-error", "stuck-release", "cooper ...(mode === "launch-failure-sandbox" ? { externallySandboxed: true } : mode === "launch-failure-unsandboxed" ? { externallySandboxed: false } : {}), runnerBinary: resolve(import.meta.dirname, "../../package.json"), codexCommand: resolve(import.meta.dirname, "../../package.json"), - sourceCodexHome: "", runnerProcessLauncher: launch, + sourceCodexHome: "", runnerProcessLauncher: launch, disableCodexWebSearch: false, controlPlaneRegistration: async (_authority, _identity, signal) => { template = structuredClone(_authority.store.state.runAttachTemplate!); registrationSignal = signal; @@ -3405,6 +3428,8 @@ it.each(["stuck", "late-release", "late-release-error", "stuck-release", "cooper expect(template).toMatchObject({ provider: { externallySandboxed: mode === "launch-failure-sandbox", readOnly: true, approvalPolicy: "never", collaborationMode: "plan" } }); const args = (template!.provider as Record).args as string[]; expect(args).toContain("project_doc_max_bytes=0"); + expect(args).toContain('web_search="disabled"'); + expect(args.indexOf('web_search="disabled"')).toBeLessThan(args.indexOf("app-server")); expect(args[args.indexOf("project_doc_max_bytes=0") - 1]).toBe("-c"); expect(args.indexOf("project_doc_max_bytes=0")).toBeLessThan(args.indexOf("app-server")); expect(args).not.toContain("--dangerously-bypass-approvals-and-sandbox"); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index b70954e9aa8..d088b3c7916 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -726,6 +726,7 @@ async function releaseRunnerProcessOwnership(input: { ((settlement: "settled" | "unsettled") => Promise | void) | null; forceKill: () => Promise | void; release: (() => Promise | void) | null; + retireController?: () => Promise; }): Promise { let releaseFailure: unknown; try { @@ -735,6 +736,9 @@ async function releaseRunnerProcessOwnership(input: { } let checkpointFailure: unknown; try { + // External ingress must stop reconnecting before admitted journal writes + // can be joined. A failed retirement cannot authorize a new checkpoint. + await input.retireController?.(); if (input.checkpoint !== null) { // Release only the authenticated control route first. In provider-ingress // mode this stops its reconnect loop; it does not release the sandbox @@ -1287,6 +1291,8 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD codexCommand?: string; codexArgs?: string[]; disableCodexProjectInstructions?: boolean; + /** Restrict hosted search without changing the provider default for other runs. */ + disableCodexWebSearch?: boolean; /** Controller-visible Codex home used only to seed the isolated runner home. */ sourceCodexHome?: string | null; opencodeCommand?: string; @@ -3567,6 +3573,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { readonly #closedSignal: Promise; #resolveClosedSignal!: () => void; #closePromise: Promise | null = null; + #detachPromise: Promise | null = null; #startupOperation: Promise | null = null; #attachmentOperation: Promise | null = null; #startupDrainTimedOut = false; @@ -4464,7 +4471,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { close(reason?: string): Promise { // Detachment relinquishes process ownership. A late execution finalizer // must not suspend or signal the runner now owned by the next controller. - if (this.#controllerDetachedForRestart) return Promise.resolve(); + if (this.#controllerDetachedForRestart) + return this.#detachPromise ?? Promise.resolve(); if (reason) { this.#diagnostic( `runner transport close requested: ${reason.replaceAll(/[\r\n]/g, " ").slice(0, 1_000)}`, @@ -4474,8 +4482,33 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { return this.#closePromise; } - async detachControllerForRestart(): Promise { + detachControllerForRestart(): Promise { + this.#detachPromise ??= this.#detachControllerOnce(); + return this.#detachPromise; + } + + async #stopAndDrainAuthority(deadline: number): Promise { + const core = this.#core; + if (!core) return; + await core.stop(); + let timer: ReturnType | undefined; + try { + await Promise.race([ + core.drainPendingConnectionProcessing(), + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error( + "Controller journal drain timed out; preserve the session before replacing its writer.", + )), Math.max(0, deadline - Date.now())); + }), + ]); + } finally { + clearTimeout(timer); + } + } + + async #detachControllerOnce(): Promise { if (this.#closed) return; + const deadline = Date.now() + runnerCloseGraceMs(this.options); this.#controllerDetachedForRestart = true; this.#closed = true; this.#resolveClosedSignal(); @@ -4493,13 +4526,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { `controller route release failed during restart detach: ${String(error)}`, ); }); - await this.#core?.stop().catch((error: unknown) => { - this.#diagnostic( - `controller authority stop failed during restart detach: ${String(error)}`, - ); - }); + // A successor must not share this journal with an admitted async write. + // A timeout rejects detachment, including repeated calls, instead of + // reporting that the old writer has retired. + try { + await this.#stopAndDrainAuthority(deadline); + } finally { + this.#queue.close(); + } this.#handle = null; - this.#queue.close(); this.#diagnostic( "controller authority detached for restart; durable runner left alive", ); @@ -4723,6 +4758,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#evidence.runnerSignal = result.signal as NodeJS.Signals | null; }, release: this.#controlPlaneRelease, + retireController: () => this.#stopAndDrainAuthority(closeDeadline), }); } finally { await this.#core?.stop(); @@ -5529,8 +5565,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { // Keep the durable provider profile and thread identity unchanged. const recordedCommand = record(runAttachTemplate.provider).command; runAttachTemplate.runtimeLaunchArgs = - this.options.codexArgs ?? createRunnerdCodexAppServerArgs({ + args: this.options.codexArgs, + disableWebSearch: this.options.disableCodexWebSearch, environment: this.options.environment, codexHome, workingDirectory: this.options.resumeWorkingDirectory, @@ -7287,6 +7324,7 @@ export async function probeNativeRunnerEnvironment(options: { environment, externallySandboxed: options.transportOptions?.externallySandboxed === true, disableCodexProjectInstructions: true, + disableCodexWebSearch: options.provider === "codex", closeGraceMs: 10_000, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, }); diff --git a/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts b/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts new file mode 100644 index 00000000000..d49672cf964 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-detach-durability.test.ts @@ -0,0 +1,150 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import fs from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { DurablePrpControlPlane } from "../control-plane/durable-prp-control-plane.js"; +import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary, readRunnerdArtifactBinding, runnerdRecoveryInternals } from "./runnerd-codex-transport.js"; + +it.each(["resolved", "rejected"] as const)("keeps the checkpoint behind controller retirement (%s)", async outcome => { + const steps: string[] = []; + let release!: () => void; + let reject!: (error: Error) => void; + const barrier = new Promise((resolveDrain, rejectDrain) => { release = resolveDrain; reject = rejectDrain; }); + const operation = runnerdRecoveryInternals.releaseRunnerProcessOwnership({ + runnerSettled: true, + release: () => { steps.push("route"); }, + retireController: async () => { steps.push("drain"); await barrier; }, + checkpoint: () => { steps.push("checkpoint"); }, + forceKill: () => { steps.push("containment"); }, + }); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(steps).toEqual(["route", "drain"]); + if (outcome === "rejected") { + const failed = expect(operation).rejects.toThrow("journal unavailable"); + reject(new Error("journal unavailable")); + await failed; + expect(steps).toEqual(["route", "drain", "containment"]); + } else { + release(); + await operation; + expect(steps).toEqual(["route", "drain", "checkpoint", "containment"]); + } +}); + +it.each(["drain", "timeout", "external_close"] as const)("joins an output flush before controller retirement permits a successor writer (%s)", async mode => { + const root = await fs.mkdtemp(join(tmpdir(), "runnerd-detach-write-")); + await fs.mkdir(join(root, "diagnostics"), { mode: 0o700 }); + const binary = defaultCapabilityRunnerdBinary(); + const artifact = readRunnerdArtifactBinding(binary); + let core!: DurablePrpControlPlane; + const start = DurablePrpControlPlane.prototype.start; + const startSpy = vi.spyOn(DurablePrpControlPlane.prototype, "start").mockImplementation(function (port) { + core = this; + return start.call(this, port); + }); + let release!: () => void; + const barrier = new Promise(resolveWrite => { release = resolveWrite; }); + let staged = false; + const originalOpen = fs.open; + const openSpy = vi.spyOn(fs, "open").mockImplementation(async (...args) => { + const file = await originalOpen(...args); + if (!staged && String(args[0]).includes(".control-plane-state.json.")) { + const sync = file.sync.bind(file); + file.sync = async () => { staged = true; await barrier; await sync(); }; + } + return file; + }); + let child: ChildProcess | undefined; + let exit: Promise | undefined; + const closeSteps: string[] = []; + let stopSpy: ReturnType | undefined; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: binary, + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider-state.json"), "--split-event-burst", "--split-event-prefix-count", "1"], + stateDirectory: root, + closeGraceMs: mode === "timeout" ? 200 : 1_500, + ...(mode === "external_close" ? { controlPlaneRegistration: async (authority: DurablePrpControlPlane) => { + await authority.start(); + const stop = authority.stop.bind(authority); + stopSpy = vi.spyOn(authority, "stop").mockImplementation(async () => { + closeSteps.push("stop"); + release(); + await stop(); + }); + return { connectUrl: authority.connectUrl, release: () => { closeSteps.push("release"); } }; + } } : {}), + runnerProcessLauncher: spec => { + child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: spec.environment, stdio: ["ignore", "pipe", "pipe"] }); + const owned = child; + let stderr = ""; + owned.stderr?.on("data", chunk => { stderr += String(chunk); }); + exit = new Promise(resolveExit => owned.once("exit", () => resolveExit())); + const completion = new Promise<{ code: number | null; signal: string | null; stdout: string; stderr: string }>((resolveExit, reject) => { + owned.once("error", reject); + owned.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr })); + }); + return { child: owned, completion }; + }, + }); + bundle.transport.setServerRequestHandler(async () => ({ success: true, contentItems: [] })); + let detached: Promise | undefined; + try { + await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [{ + name: "get_task_context", description: "Read fixture context.", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + }] }); + await bundle.transport.request("turn/start", { input: [{ type: "text", text: "Exercise a held output flush." }] }); + await vi.waitFor(() => expect(staged).toBe(true), { timeout: 5_000 }); + if (mode === "external_close") { + await bundle.transport.close().catch(error => { + expect(error.message).toMatch(/journal drain timed out|did not durably suspend/); + }); + expect(closeSteps.indexOf("release")).toBeGreaterThanOrEqual(0); + expect(closeSteps.indexOf("release")).toBeLessThan(closeSteps.indexOf("stop")); + return; + } + let settled = false; + detached = bundle.detachControllerForRestart().then(() => { settled = true; }); + await vi.waitFor(() => expect(core.activeRunnerConnectionCount()).toBe(0)); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(settled).toBe(false); + if (mode === "timeout") { + await expect(detached).rejects.toThrow("journal drain timed out"); + await expect(bundle.detachControllerForRestart()).rejects.toThrow("journal drain timed out"); + await expect(bundle.transport.close()).rejects.toThrow("journal drain timed out"); + release(); + await core.drainPendingConnectionProcessing(); + } else { + release(); + await detached; + } + const successor = new DurablePrpControlPlane({ stateDirectory: join(root, "control-plane"), + identity: core.store.state.identity, expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest }); + successor.queueCommand("session.snapshot", {}, "successor_command"); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + const persisted = JSON.parse(await fs.readFile(successor.store.path, "utf8")); + expect(persisted.commands.some((command: { commandId: string }) => command.commandId === "successor_command")).toBe(true); + } catch (error) { + const diagnostic = await fs.readFile(join(root, "diagnostics", "runnerd.stderr.log"), "utf8").catch(() => ""); + throw new Error(`${String(error)}\n${diagnostic}`, { cause: error }); + } finally { + release(); + await detached?.catch(() => undefined); + openSpy.mockRestore(); + startSpy.mockRestore(); + await bundle.detachControllerForRestart().catch(() => undefined); + const providerPid = bundle.evidence().codexPid; + if (providerPid) { + try { process.kill(providerPid, "SIGKILL"); } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; + } + } + child?.kill("SIGKILL"); + await exit; + await core?.drainPendingConnectionProcessing(); + stopSpy?.mockRestore(); + await fs.rm(root, { recursive: true, force: true }); + } +}, 15_000); diff --git a/packages/paperclip-runner/src/live/runnerd-opencode-assigned-tools.test.ts b/packages/paperclip-runner/src/live/runnerd-opencode-assigned-tools.test.ts new file mode 100644 index 00000000000..e4722449c17 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-opencode-assigned-tools.test.ts @@ -0,0 +1,65 @@ +import { createHash } from "node:crypto"; +import { existsSync, readFileSync } from "node:fs"; +import { chmod, copyFile, mkdir, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it } from "vitest"; +import { codexSemanticToolSpecs, dynamicToolResponse } from "../drivers/codex/codex-driver-values.js"; +import { createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js"; + +const runner = defaultCapabilityRunnerdBinary(); +const proxy = resolve("dist/cli/opencode-app-server-proxy.cjs"); +const digest = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`; + +it.skipIf(process.platform === "win32" || !existsSync(runner) || !existsSync(proxy))( + "retains assigned discovery authority across Rust, the proxy, and HTTP MCP", + async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-opencode-assigned-")); + const workspace = join(root, "workspace"); + const executable = join(root, "fake-opencode-server.mjs"); + await mkdir(workspace); + await copyFile(resolve("test/fixtures/fake-opencode-server.mjs"), executable); + await chmod(executable, 0o755); + const operation = "paperclip_search_assigned_tools"; + const seen: string[] = []; + const { transport } = createRunnerdCodexTransport({ + provider: "opencode", runnerBinary: runner, stateDirectory: join(root, "state"), sourceCodexHome: "", + providerNodeCommand: process.execPath, providerNodeCommandSha256: digest(process.execPath), + opencodeCommand: executable, opencodeCommandSha256: digest(executable), + opencodeProxyPath: proxy, opencodeProxySha256: digest(proxy), + environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "synthetic-fixture-key" }, + }); + transport.setServerRequestHandler(async request => { + expect(request.method).toBe("item/tool/call"); + const tool = request.params.tool; + expect([operation, "paperclip_finish"]).toContain(tool); + seen.push(String(tool)); + return dynamicToolResponse(tool === operation ? { tools: [] } : "Accepted completion."); + }); + let timer: ReturnType | undefined; + try { + await transport.request("thread/start", { + cwd: workspace, model: "openrouter/deepseek/deepseek-v4.1-flash-1017", + completionContract: { revision: "assigned-tools-v1", criterionIds: ["objective"] }, + dynamicTools: [{ name: operation, description: "Search assigned synthetic tools.", inputSchema: { type: "object" } }, ...codexSemanticToolSpecs()], + }); + const completed = (async () => { + for await (const notification of transport.notifications()) { + if (notification.method === "turn/completed") return; + if (notification.method === "turn/failed") throw new Error("Native OpenCode turn failed"); + } + throw new Error("Native OpenCode notifications ended before completion"); + })(); + await transport.request("turn/start", { input: [{ type: "text", text: "Discover assigned synthetic tools." }] }); + await Promise.race([completed, new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error("Native assigned-tool smoke timed out")), 20_000); + })]); + expect(seen).toEqual([operation, "paperclip_finish"]); + } finally { + if (timer) clearTimeout(timer); + await transport.close(); + await rm(root, { recursive: true, force: true }); + } + }, + 30_000, +); diff --git a/packages/paperclip-runner/src/live/runnerd-welcome.integration.test.ts b/packages/paperclip-runner/src/live/runnerd-welcome.integration.test.ts new file mode 100644 index 00000000000..49d44895434 --- /dev/null +++ b/packages/paperclip-runner/src/live/runnerd-welcome.integration.test.ts @@ -0,0 +1,62 @@ +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { expect, it, vi } from "vitest"; +import { createCapabilityRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "./runnerd-codex-transport.js"; + +it("starts one provider after delayed bootstrap admission and retains it for a second turn", async () => { + const root = await mkdtemp(join(tmpdir(), "runnerd-delayed-welcome-")); + const calls = join(root, "provider-calls.log"); + let admissionDelayed = false; + let admissionTimer: ReturnType | undefined; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(root, "provider.json"), "--call-log", calls, "--record-process-start", "--durable-turn-ids"], + stateDirectory: join(root, "state"), + controlPlaneRegistration: async authority => { + const attach = authority.attachWireConnection.bind(authority); + vi.spyOn(authority, "attachWireConnection").mockImplementation(wire => attach({ + sendJson: value => wire.sendJson(value), + close: code => wire.close(code), + onClose: listener => wire.onClose(listener), + onJson: listener => wire.onJson(value => { + if (!admissionDelayed && (value as { kind?: string }).kind === "auth_response") { + admissionDelayed = true; + admissionTimer = setTimeout(() => listener(value), 3_000); + } else listener(value); + }), + })); + await authority.start(); + return { connectUrl: authority.connectUrl, release: () => authority.stop() }; + }, + }); + const terminalTurns: unknown[] = []; + try { + const opened = await bundle.transport.request("thread/start", { cwd: root, dynamicTools: [] }); + expect(admissionDelayed).toBe(true); + for (const text of ["Complete the first turn.", "Continue the same session."]) { + await bundle.transport.request("turn/start", { input: [{ type: "text", text }] }); + for await (const event of bundle.transport.notifications()) { + if (event.method === "turn/completed") { + expect(event.params).toMatchObject({ threadId: opened.thread.id }); + terminalTurns.push(event.params); + break; + } + } + } + await bundle.transport.close(); + const methods = (await readFile(calls, "utf8")).trim().split("\n"); + expect(methods.filter(method => method === "process-start")).toHaveLength(1); + expect(methods.filter(method => method === "thread/start")).toHaveLength(1); + expect(methods.filter(method => method === "turn/start")).toHaveLength(2); + expect(terminalTurns).toHaveLength(2); + const runner = JSON.parse(await readFile(join(root, "state/runner/runner-state.json"), "utf8")); + expect(runner.lifecycle).toBe("suspended"); + expect(runner.outbox).toEqual([]); + } finally { + if (admissionTimer) clearTimeout(admissionTimer); + await bundle.transport.close(); + await rm(root, { recursive: true, force: true }); + } +}, 30_000); diff --git a/packages/paperclip-runner/src/testing.ts b/packages/paperclip-runner/src/testing.ts index b220b2ea73a..5260f4f5114 100644 --- a/packages/paperclip-runner/src/testing.ts +++ b/packages/paperclip-runner/src/testing.ts @@ -9,6 +9,7 @@ export * from "./index.js"; export * from "./conformance/control-plane-port.js"; export * from "./conformance/capability-semantic-conformance.js"; export * from "./conformance/harness-driver.js"; +export { startOpenCodeMcpBridge } from "./drivers/opencode/mcp-bridge.js"; export * from "./conformance/semantic-conformance.js"; export * from "./mock-core/deterministic-harness-driver.js"; export * from "./mock-core/mock-control-plane-adapter.js"; diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v60-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v60-identity.json new file mode 100644 index 00000000000..82b95ed7321 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v60-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:ca3c2b38ed5f6e25cf1c96c1f73781a86cc90c1c0add93f64716c66f28b3f3aa", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 60, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "8ddd5d26854f4a153a9559ea8105aef95a3ec94f03209e9a797c95932c162e2a", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "8503e7f2f8175e413b42e85a9ff259a00f2278cd966bbeafc9e9e85417cf7dbb", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json new file mode 100644 index 00000000000..6a641986fce --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:956276ebee59f880249fa6955adc224af9ca7b6a9da59bcc5da4a8667550c5b3", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 61, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "e6c98128d98a4177c42d68a04c1f749554fd5ec844da1bbbdb3f682670189445", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "f9670b63d68e8b84cb79ab18425028f1ff781adb398cb9955ca80b8e0c9264c7", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/plugins/sdk/src/environment-tasks.ts b/packages/plugins/sdk/src/environment-tasks.ts index 57f0dea51fa..f4f86e26b53 100644 --- a/packages/plugins/sdk/src/environment-tasks.ts +++ b/packages/plugins/sdk/src/environment-tasks.ts @@ -39,6 +39,7 @@ export const environmentRunnerLaunchRequirementsSchema = z.object({ acpxPermissionModePinned: z.boolean().optional(), piThinkingLevel: z.enum(["off", "low", "high", "max"]).optional(), claudeEffort: z.enum(["low", "medium", "high", "xhigh", "max"]).optional(), opencodePermissionMode: z.enum(["allow", "ask", "deny"]).optional(), + disableCodexWebSearch: z.literal(true).optional(), managedProfile: z.record(z.string(), z.unknown()).optional(), agentCoreProfile: z.record(z.string(), z.unknown()).optional(), }).strict(), approvalPolicy: z.enum(["never", "on-request", "untrusted"]), readOnly: z.boolean(), diff --git a/packages/plugins/sdk/tests/environment-tasks.test.ts b/packages/plugins/sdk/tests/environment-tasks.test.ts index 9db7cbfe2d5..6f9d7aa388b 100644 --- a/packages/plugins/sdk/tests/environment-tasks.test.ts +++ b/packages/plugins/sdk/tests/environment-tasks.test.ts @@ -11,3 +11,37 @@ describe("task lifecycle policy",()=>{ expect(environmentTaskOperationSchema.safeParse({...submit,launch:{...submit.launch,lifecyclePolicy:{mode:"warm",idleTimeoutMs}}}).success).toBe(false); }); }); + + +const prepare = { + kind: "prepare", projectIds: [], requirements: { + schema: "paperclip.runner.launch-requirements.v1", + identity: { runnerInstanceId: "runner", environmentLeaseId: "lease", runId: "run", + normalizedSessionId: "session", turnId: "turn", itemId: "item" }, + provider: "codex", model: "selected-model", instructions: "Verify the selected account.", + runtimeContext: null, settings: {}, approvalPolicy: "never", readOnly: true, + preparedConversation: false, includeCollaborationModeInstructions: false, + }, +}; + +describe("task hosted search restriction", () => { + it("preserves Core's explicit restriction through plugin launch preparation", () => { + const operation = { ...prepare, requirements: { ...prepare.requirements, + settings: { disableCodexWebSearch: true } } }; + expect(environmentTaskOperationSchema.parse(operation)).toEqual(operation); + }); + + it("keeps the provider default when the restriction is absent", () => { + expect(environmentTaskOperationSchema.parse(prepare)).toEqual(prepare); + }); + + it.each([false, null, 0, "true", "disabled"])("rejects a non-restricting flag value %s", disableCodexWebSearch => { + expect(environmentTaskOperationSchema.safeParse({ ...prepare, requirements: { ...prepare.requirements, + settings: { disableCodexWebSearch } } }).success).toBe(false); + }); + + it("continues rejecting unreviewed launch settings", () => { + expect(environmentTaskOperationSchema.safeParse({ ...prepare, requirements: { ...prepare.requirements, + settings: { disableCodexWebSearch: true, hostedSearchBillingMode: "unmetered" } } }).success).toBe(false); + }); +}); diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index f844a18bce3..095ca9aa105 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -131,6 +131,7 @@ import { deliverReconciledExecutions } from "../services/execution-recovery-reso import { unadmittedChatWakeupCondition, authorizeFailedChatRunRetryWake, + createDurableChatWakeupRequest, } from "../services/durable-chat-wakeup.js"; import { createChatWebhookDiagnostics, @@ -60661,6 +60662,358 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { return { ...context, issue, action, receipt, runId }; } + async function currentGitHubTaskRetryFixture(withPending = false, pendingCount = 1) { + const f = await failedChatRetryFixture("github", "42", true); + const companyId = f.fixture.companyId, issueId = f.issue.id; + await githubResponseCommentService(db, f.providerFetch).tryAcknowledge(f.action.deliveryId!); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, + nativePhase: "terminal_failure", runnerProfileJson: {} }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: f.runId, phase: "terminal_failure" }); + await db.insert(heartbeatRunEvents).values({ companyId, agentId: f.fixture.assignedAgentId, runId: f.runId, + seq: 1, eventType: "native.local_process_stopped", payload: { processPid: 999999999, processGroupId: null } }); + let pending: typeof chatActions.$inferSelect | undefined; + if (withPending) { + for (let index = 0; index < pendingCount; index++) { + await deliverMessage({ callbacks: f.runtime.configurations.get(f.endpoint.id)!.callbacks, + endpointId: f.endpoint.id, provider: "github", thread: f.thread.thread, + message: makeMessage({ id: String(990092 + index), text: `@maya continue saved follow-up ${index}`, userId: "42", mentioned: true }), + trigger: "mention" }); + const [{ source }] = await db.select({ source: chatActions }).from(chatActions) + .innerJoin(chatDeliveries, eq(chatDeliveries.id, chatActions.deliveryId)).where(and( + eq(chatActions.conversationId, f.conversation.id), eq(chatActions.kind, "inbound_wakeup"), + eq(chatDeliveries.providerEventId, `${f.thread.thread.id}:${990092 + index}`))); + pending ??= source; + await db.update(agentWakeupRequests).set({ status: "deferred_issue_execution", runId: null }) + .where(eq(agentWakeupRequests.id, source.id)); + await githubResponseCommentService(db, f.providerFetch).tryAcknowledge(source.deliveryId!); + } + } + const [originalDelivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, f.action.deliveryId!)); + + await db.update(agents).set({ adapterType: "paperclip_runner", status: "running", + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }) + .where(eq(agents.id, f.fixture.assignedAgentId)); + await db.insert(heartbeatRuns).values({ companyId, agentId: f.fixture.assignedAgentId, + status: "running", startedAt: new Date() }); + f.wakeup.mockImplementation(heartbeatService(db).wakeup); + return { ...f, pending, originalDelivery }; + } + + it.each(["reconnected", "newer_input", "reconnected_newer_input"] as const)( + "admits current GitHub task Retry without replaying obsolete input: %s", async kind => { + const f = await currentGitHubTaskRetryFixture(kind.includes("newer_input")); + const { pending, originalDelivery } = f; + try { + const companyId = f.fixture.companyId, issueId = f.issue.id; + if (kind.includes("reconnected")) await f.service.configure(f.endpoint.id, { action: "reconnect" }, "owner-user"); + const input = { db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }; + const request = await requestTaskRetry(input); + expect(request.accepted).toBe(true); + expect((await requestTaskRetry(input)).requestId).toBe(request.requestId); + expect(await f.service.processFailedChatRunRetry(request.chatActionId!)).toMatchObject({ status: "queued" }); + const [wake] = await readAgentWakeupRequestsById(request.chatActionId!); + expect(wake.runId).toBeTruthy(); + const expectedComment = pending?.payload.commentId ?? f.action.payload.commentId; + const [run] = await readHeartbeatRunsById(wake.runId!); + expect(run).toMatchObject({ status: "queued", retryOfRunId: f.runId, + contextSnapshot: { wakeCommentIds: [expectedComment], taskRetryRequestId: request.requestId } }); + if (pending) expect((await readAgentWakeupRequestsById(pending.id))[0]) + .toMatchObject({ status: "coalesced", runId: wake.runId }); + await f.service.processFailedChatRunRetry(request.chatActionId!); + await heartbeatService(db).resumeQueuedRuns(); + expect((await readHeartbeatRunsByCompanyId(companyId)).filter(r => r.contextSnapshot?.taskRetryRequestId === request.requestId)) + .toHaveLength(1); + expect((await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, originalDelivery.id)))[0].normalizedEvent) + .toEqual(originalDelivery.normalizedEvent); + } finally { await f.service.shutdown(); } + }, + ); + + it.each(["legacy_receipt", "concurrent_clicks", "lost_dispatch_reply", "restart", "retry_failed_successor", "http", "handoff_rollback", "board_permission_wait", "peer_conversation", "backlog_51", "consumed_backlog_52"] as const)( + "preserves one current GitHub Retry across %s", async mode => { + const f = await currentGitHubTaskRetryFixture(true, mode.includes("backlog") ? 51 : mode === "peer_conversation" ? 2 : 1); + try { + const companyId = f.fixture.companyId, issueId = f.issue.id; + if (mode === "consumed_backlog_52") { + const sources = await db.select().from(chatActions).where(and(eq(chatActions.conversationId, f.conversation.id), + eq(chatActions.kind, "inbound_wakeup"))).orderBy(asc(chatActions.createdAt), asc(chatActions.id)); + expect(sources).toHaveLength(52); + const commentIds = sources.map(source => source.payload.commentId); + await db.update(heartbeatRuns).set({ finishedAt: new Date(), contextSnapshot: { + ...(await readHeartbeatRunsById(f.runId))[0].contextSnapshot, + wakeCommentIds: commentIds, wakeCommentId: commentIds.at(-1), + } }).where(eq(heartbeatRuns.id, f.runId)); + await db.update(agentWakeupRequests).set({ runId: f.runId, status: "coalesced", + payload: sql`${agentWakeupRequests.payload} || ${JSON.stringify({ coalescedIntoWakeupRequestId: f.action.id })}::jsonb`, + }).where(and(inArray(agentWakeupRequests.id, sources.map(source => source.id)), + sql`${agentWakeupRequests.id} <> ${f.action.id}::uuid`)); + } + let peerInput: typeof chatActions.$inferSelect | undefined; + if (mode === "peer_conversation") { + const originId = randomUUID(); + await db.insert(chatConversations).values({ ...f.conversation, id: originId, sessionGeneration: 2 }); + await db.update(chatActions).set({ conversationId: originId, + payload: sql`jsonb_set(${chatActions.payload}, '{sessionGeneration}', '2'::jsonb)`, + }).where(inArray(chatActions.id, [f.action.id, f.pending!.id])); + await db.update(chatDeliveries).set({ conversationId: originId }) + .where(inArray(chatDeliveries.id, [f.action.deliveryId!, f.pending!.deliveryId!])); + await db.update(chatMessageLinks).set({ conversationId: originId }) + .where(inArray(chatMessageLinks.deliveryId, [f.action.deliveryId!, f.pending!.deliveryId!])); + [peerInput] = await db.select().from(chatActions).where(and(eq(chatActions.conversationId, f.conversation.id), + eq(chatActions.kind, "inbound_wakeup"))); + expect(peerInput).toBeDefined(); + } + if (mode === "legacy_receipt") { + const [originalEndpoint] = await readChatEndpointsById(f.endpoint.id); + await db.insert(chatActions).values({ companyId, endpointId: f.endpoint.id, + conversationId: f.conversation.id, deliveryId: f.pending!.deliveryId, + kind: "github_response_comment", providerActionId: `github_response:${f.pending!.deliveryId}`, + payload: { botExternalId: originalEndpoint.botExternalId, agentId: f.fixture.assignedAgentId }, + status: "processed" }).onConflictDoNothing(); + await db.update(chatDeliveries).set({ + normalizedEvent: sql`jsonb_set(${chatDeliveries.normalizedEvent}, '{githubAuthority}', + (${chatDeliveries.normalizedEvent}->'githubAuthority') - 'botExternalId')`, + }).where(eq(chatDeliveries.id, f.pending!.deliveryId!)); + } + await f.service.configure(f.endpoint.id, { action: "reconnect" }, "owner-user"); + const input = { db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }; + if (mode === "board_permission_wait") { + input.actorId = `retry-operator-${companyId}`; + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: input.actorId, + status: "active", membershipRole: "operator" }); + } + const queryEvents: Record[] = []; + const capture = mode === "backlog_51" ? createDatabaseResponseCapture({ start: Date.now() - 1, + end: Date.now() + 59_000 }, event => queryEvents.push(event)) : undefined; + if (capture) installDatabaseResponseCapture(db, { chatActions, chatDeliveries, chatMessageLinks, issueComments, + agentWakeupRequests, chatEndpoints, chatConversations }, capture); + let one: Awaited>, two: typeof one; + if (mode === "http") { + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); }); + app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service })); + app.use(errorHandler); + const send = () => request(app).post(`/api/issues/${issueId}/retry`).send({ + idempotencyKey: input.idempotencyKey, observedRunId: input.observedRunId, + }).expect(202); + const replies = await Promise.all([send(), send()]); + [one, two] = replies.map(response => response.body); + } else [one, two] = await Promise.all([requestTaskRetry(input), requestTaskRetry({ ...input, + idempotencyKey: mode === "concurrent_clicks" ? randomUUID() : input.idempotencyKey })]); + if (capture) { + capture.flush(); + expect(Number(queryEvents.find(event => event.event === "paperclip.db_response_capture.summary.v1")?.startedQueries)) + .toBeLessThan(180); + } + expect(two.requestId).toBe(one.requestId); + expect(one).toMatchObject({ accepted: true, chatActionId: expect.any(String) }); + let service = f.service; + if (mode === "restart") { + await service.shutdown(); + service = createService(f.runtime, f.providerFetch, { wakeup: heartbeatService(db).wakeup }).service; + } + if (mode === "lost_dispatch_reply") f.wakeup.mockImplementationOnce(async (...args) => { + await heartbeatService(db).wakeup(...args); + throw new Error("Simulated response loss after dispatch commit"); + }); + if (mode === "handoff_rollback") f.wakeup.mockImplementationOnce(async (agentId, options) => { + const original = options!.durableChatRequest!; + return heartbeatService(db).wakeup(agentId, { ...options, durableChatRequest: createDurableChatWakeupRequest({ + ...original, onAdmitted: async (tx, receipt) => { + await original.onAdmitted!(tx, receipt); + throw new Error("Simulated interruption before admission commit"); + }, + }) }); + }); + if (mode === "board_permission_wait") await db.update(companyMemberships).set({ membershipRole: "viewer" }) + .where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, input.actorId))); + await Promise.all([service.processFailedChatRunRetry(one.chatActionId!), + service.processFailedChatRunRetry(one.chatActionId!)]); + if (mode === "handoff_rollback" || mode === "board_permission_wait") { + expect(await readAgentWakeupRequestsById(one.chatActionId!)).toHaveLength(0); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0]) + .toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect((await readHeartbeatRunsByCompanyId(companyId)) + .filter(run => run.contextSnapshot?.taskRetryRequestId === one.requestId)).toHaveLength(0); + } + if (mode === "board_permission_wait") await db.update(companyMemberships).set({ membershipRole: "operator" }) + .where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, input.actorId))); + await vi.waitFor(async () => { + await service.processFailedChatRunRetry(one.chatActionId!); + expect((await readAgentWakeupRequestsById(one.chatActionId!))[0]?.runId, + JSON.stringify((await db.select().from(chatActions).where(eq(chatActions.id, one.chatActionId!)))[0]?.result)).toBeTruthy(); + }, { timeout: 10_000, interval: 100 }); + const [parent] = await readAgentWakeupRequestsById(one.chatActionId!); + expect(parent.runId).toBeTruthy(); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0]).toMatchObject({ + status: "coalesced", runId: mode === "consumed_backlog_52" ? f.runId : parent.runId, + }); + if (peerInput) expect((await readAgentWakeupRequestsById(peerInput.id))[0]) + .toMatchObject({ status: "deferred_issue_execution", runId: null }); + if (mode === "backlog_51") { + const saved = await db.select({ id: chatActions.id, payload: chatActions.payload, + status: agentWakeupRequests.status, runId: agentWakeupRequests.runId }).from(chatActions) + .innerJoin(agentWakeupRequests, eq(agentWakeupRequests.id, chatActions.id)).where(and( + eq(chatActions.conversationId, f.conversation.id), eq(chatActions.kind, "inbound_wakeup"), + sql`${chatActions.id} <> ${f.action.id}::uuid`, + )).orderBy(asc(chatActions.createdAt), asc(chatActions.id)); + expect(saved).toHaveLength(51); + expect(saved.every(source => source.status === "coalesced" && source.runId === parent.runId)).toBe(true); + expect((await readHeartbeatRunsById(parent.runId!))[0].contextSnapshot?.wakeCommentIds) + .toEqual(saved.map(source => source.payload.commentId)); + } + await service.processFailedChatRunRetry(one.chatActionId!); + expect((await readHeartbeatRunsByCompanyId(companyId)) + .filter(run => run.contextSnapshot?.taskRetryRequestId === one.requestId)).toHaveLength(1); + expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId), + eq(chatActions.kind, "failed_run_retry")))).toHaveLength(1); + if (mode === "retry_failed_successor" || mode === "backlog_51") { + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), + errorCode: "adapter_failed", runtimeMode: "native", nativeIssueId: issueId, nativePhase: "terminal_failure", + }).where(eq(heartbeatRuns.id, parent.runId!)); + await db.update(agentWakeupRequests).set({ status: "failed" }).where(eq(agentWakeupRequests.id, one.chatActionId!)); + await db.update(issues).set({ executionRunId: null, status: "blocked" }).where(eq(issues.id, issueId)); + await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId: parent.runId!, phase: "terminal_failure" }); + await db.insert(heartbeatRunEvents).values({ companyId, agentId: f.fixture.assignedAgentId, runId: parent.runId!, + seq: 1, eventType: "native.local_process_stopped", payload: { processPid: 999999998, processGroupId: null } }); + await service.configure(f.endpoint.id, { action: "reconnect" }, "owner-user"); + const next = await requestTaskRetry({ ...input, observedRunId: parent.runId!, idempotencyKey: randomUUID(), + prepareChatRetry: service.prepareFailedChatRunRetry }); + expect(next.requestId).not.toBe(one.requestId); + expect(await service.processFailedChatRunRetry(next.chatActionId!)).toMatchObject({ status: "queued" }); + const [second] = await readAgentWakeupRequestsById(next.chatActionId!); + expect((await readHeartbeatRunsById(second.runId!))[0]).toMatchObject({ + retryOfRunId: parent.runId, contextSnapshot: { + wakeCommentIds: (await readHeartbeatRunsById(parent.runId!))[0].contextSnapshot!.wakeCommentIds, + }, + }); + } + } finally { await f.service.shutdown(); } + }, + ); + + it("leaves saved GitHub input pending when a persisted Retry wait loses board authority", async () => { + const f = await currentGitHubTaskRetryFixture(true); + try { + const companyId = f.fixture.companyId, issueId = f.issue.id, actorId = `retry-operator-${companyId}`; + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: actorId, + status: "active", membershipRole: "operator" }); + const intent = await requestTaskRetry({ db, companyId, issueId, actorId, actorSource: "cloud_tenant", + observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }); + let savedOptions: Parameters[1]; + f.wakeup.mockImplementationOnce(async (_agentId, options) => { + savedOptions = options; + const request = options!.durableChatRequest!; + // Persist the pre-handoff wait as a server interrupted before queue adoption would. + await db.insert(agentWakeupRequests).values({ id: request.id, companyId, agentId: request.agentId, + source: options!.source, triggerDetail: options!.triggerDetail, reason: options!.reason, + requestedByActorType: request.requestedByActorType, requestedByActorId: request.requestedByActorId, + requestedAt: request.requestedAt, idempotencyKey: request.idempotencyKey, + status: "deferred_issue_execution", payload: { ...options!.payload, + taskRetryRequestId: intent.requestId, _paperclipWakeContext: options!.contextSnapshot } }); + return null; + }); + expect(await f.service.processFailedChatRunRetry(intent.chatActionId!)).toMatchObject({ status: "deferred" }); + await db.update(companyMemberships).set({ membershipRole: "viewer" }).where(and( + eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, actorId))); + expect(await heartbeatService(db).wakeup(f.fixture.assignedAgentId, savedOptions!)).toBeNull(); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0]) + .toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect((await readHeartbeatRunsByCompanyId(companyId)) + .filter(run => run.contextSnapshot?.taskRetryRequestId === intent.requestId)).toHaveLength(0); + await db.update(companyMemberships).set({ membershipRole: "operator" }).where(and( + eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, actorId))); + await heartbeatService(db).wakeup(f.fixture.assignedAgentId, savedOptions!); + const [parent] = await readAgentWakeupRequestsById(intent.chatActionId!); + expect(parent.runId).toBeTruthy(); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0]).toMatchObject({ status: "coalesced", runId: parent.runId }); + } finally { await f.service.shutdown(); } + }); + + it.each(["changed_bot", "disabled_repository", "retargeted_repository", "revoked_sender", "paused_endpoint", + "edited_source", "deleted_source", "reassigned_task", "second_reconnect", "later_message_wrong_bot", "later_message_deleted", "later_message_revoked_policy"] as const)( + "keeps current GitHub task Retry authority gated: %s", async gate => { + const f = await currentGitHubTaskRetryFixture(true, gate.startsWith("later_message") ? 2 : 1); + try { + const companyId = f.fixture.companyId, issueId = f.issue.id; + await f.service.configure(f.endpoint.id, { action: "reconnect" }, "owner-user"); + const input = { db, companyId, issueId, actorId: "owner-user", actorSource: "cloud_tenant", + observedRunId: f.runId, idempotencyKey: randomUUID(), prepareChatRetry: f.service.prepareFailedChatRunRetry }; + if (gate === "later_message_revoked_policy") { + const [resource] = await db.select().from(chatEndpointResources).where(eq(chatEndpointResources.id, f.conversation.resourceId!)); + await db.update(chatEndpointResources).set({ availability: "available", + metadata: { ...resource.metadata, providerRepositoryId: "418" }, + }).where(eq(chatEndpointResources.id, resource.id)); + await db.insert(chatGitHubConfigurations).values({ companyId, endpointId: f.endpoint.id, + updatedByUserId: "owner-user", configuration: { version: 1, toolsEnabled: true, responsibleUserId: "owner-user", + memberAccess: "all_linked", people: [], repositories: {}, + defaults: { ...defaultGitHubReviewPolicy(), invocation: "linked_authors", events: ["opened", "synchronize"] }, + } }); + const sources = await db.select().from(chatActions).where(and(eq(chatActions.conversationId, f.conversation.id), + eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.id} <> ${f.action.id}::uuid`)) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)); + for (const [index, source] of sources.entries()) { + const context = { event: index === 0 ? "opened" : "synchronize", deliveryId: source.deliveryId, + repositoryId: "418", repository: resource.providerResourceId, + pullNumber: 7, title: "Saved review request", body: "", baseSha: "a".repeat(40), headSha: "b".repeat(40), + baseBranch: "master", author: { id: "42", login: "octocat", isBot: false }, + sender: { id: "42", login: "octocat" }, draft: false, labels: [] }; + await db.update(chatDeliveries).set({ + normalizedEvent: sql`${chatDeliveries.normalizedEvent} || ${JSON.stringify({ githubAutomatic: { context } })}::jsonb`, + }).where(eq(chatDeliveries.id, source.deliveryId!)); + } + } + const accepted = await requestTaskRetry(input); + if (gate.startsWith("later_message")) { + const [later] = await db.select().from(chatActions).where(and(eq(chatActions.conversationId, f.conversation.id), + eq(chatActions.kind, "inbound_wakeup"), sql`${chatActions.id} not in (${f.action.id}::uuid, ${f.pending!.id}::uuid)`)); + if (gate === "later_message_wrong_bot") await db.update(chatDeliveries).set({ + normalizedEvent: sql`jsonb_set(${chatDeliveries.normalizedEvent}, '{githubAuthority,botExternalId}', ${JSON.stringify(randomUUID())}::jsonb)`, + }).where(eq(chatDeliveries.id, later.deliveryId!)); + else if (gate === "later_message_deleted") await db.update(issueComments).set({ deletedAt: new Date() }) + .where(eq(issueComments.id, String(later.payload.commentId))); + else await db.update(chatGitHubConfigurations).set({ revision: sql`${chatGitHubConfigurations.revision} + 1`, + configuration: sql`jsonb_set(${chatGitHubConfigurations.configuration}, '{defaults,events}', '["opened"]'::jsonb)`, + }).where(eq(chatGitHubConfigurations.endpointId, f.endpoint.id)); + } + if (gate === "changed_bot") await db.update(chatEndpoints).set({ botExternalId: randomUUID() }) + .where(eq(chatEndpoints.id, f.endpoint.id)); + if (gate === "disabled_repository") await db.update(chatEndpointResources).set({ enabled: false }) + .where(eq(chatEndpointResources.id, f.conversation.resourceId!)); + if (gate === "retargeted_repository") await db.update(chatEndpointResources).set({ providerResourceId: "another/repository" }) + .where(eq(chatEndpointResources.id, f.conversation.resourceId!)); + if (gate === "revoked_sender") await db.update(chatIdentityLinks).set({ status: "revoked" }) + .where(and(eq(chatIdentityLinks.endpointId, f.endpoint.id), eq(chatIdentityLinks.principalId, f.pending!.principalId!))); + if (gate === "paused_endpoint") await db.update(chatEndpoints).set({ status: "paused" }) + .where(eq(chatEndpoints.id, f.endpoint.id)); + if (gate === "edited_source") await db.update(issueComments).set({ body: "Changed after Retry", updatedAt: new Date() }) + .where(eq(issueComments.id, String(f.pending!.payload.commentId))); + if (gate === "deleted_source") await db.update(issueComments).set({ deletedAt: new Date() }) + .where(eq(issueComments.id, String(f.pending!.payload.commentId))); + if (gate === "reassigned_task") await db.update(issues).set({ assigneeAgentId: f.fixture.replacementAgentId }) + .where(eq(issues.id, issueId)); + if (gate === "second_reconnect") await f.service.configure(f.endpoint.id, { action: "reconnect" }, "owner-user"); + expect(await f.service.processFailedChatRunRetry(accepted.chatActionId!)).toMatchObject({ status: "failed" }); + expect((await readAgentWakeupRequestsById(f.pending!.id))[0]).toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect((await readHeartbeatRunsByCompanyId(companyId)) + .filter(run => run.contextSnapshot?.taskRetryRequestId === accepted.requestId)).toHaveLength(0); + const [settled] = await db.select().from(chatActions).where(eq(chatActions.id, accepted.chatActionId!)); + expect(settled.result?.taskRetryInputReconciled).toBe(true); + const [intentReceipt] = await readAgentWakeupRequestsById(accepted.requestId!); + await f.service.processFailedChatRunRetry(accepted.chatActionId!); + expect((await readAgentWakeupRequestsById(accepted.requestId!))[0].updatedAt).toEqual(intentReceipt.updatedAt); + if (gate === "second_reconnect") { + const next = await requestTaskRetry({ ...input, idempotencyKey: randomUUID() }); + expect(next.requestId).not.toBe(accepted.requestId); + expect(await f.service.processFailedChatRunRetry(next.chatActionId!)).toMatchObject({ status: "queued" }); + } + } finally { await f.service.shutdown(); } + }, + ); + it.each([ "settled", "distinct_account", diff --git a/server/src/__tests__/execution-control-work.test.ts b/server/src/__tests__/execution-control-work.test.ts index 9d107b75dd7..0046d8574aa 100644 --- a/server/src/__tests__/execution-control-work.test.ts +++ b/server/src/__tests__/execution-control-work.test.ts @@ -4,7 +4,7 @@ import { localAiLoginService } from "../services/local-ai-login.js"; import { randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; -import { agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, applyPendingMigrations } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, applyPendingMigrations } from "@paperclipai/db"; import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -16,7 +16,13 @@ import { terminalizeLegacyExecution } from "../services/legacy-execution-recover import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js"; import { issueRecoveryActionService } from "../services/issue-recovery-actions.js"; import { markExecutionReconciliation, deliverReconciledExecutions, settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js"; +import { nextWorkspaceRecoveryAt } from "../services/system-recovery.js"; +import { createSystemRecovery } from "../services/system-recovery-state.js"; +import { retainedWorkspaceCandidates } from "../services/workspace-recovery.js"; +import { registerWorkspaceRecoveryWork } from "../services/workspace-recovery-work.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { requestTaskRetry } from "../services/task-retry.js"; const coordinators: ReturnType[] = []; afterEach(async () => { for (const worker of coordinators.splice(0)) await worker.stop(); vi.useRealTimers(); }); @@ -94,13 +100,13 @@ describe("execution-control lifecycle", () => { expect(run).toHaveBeenCalledTimes(1); expect(next).toHaveBeenCalledTimes(1); await c.stop(); expect(c.nextWakeAt()).toBeNull(); }); - it("keeps other obligations responsive while one is waiting, and awaits shutdown", async () => { + it.each([DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.workspaceRecovery])("keeps finalization responsive while %s is waiting, and awaits shutdown", async heldQueue => { vi.useFakeTimers(); const owner = fakeOwner(); const c = coordinator(owner); let release!: () => void; const gate = new Promise(resolve => { release = resolve; }); const status = vi.fn(async () => {}); const started = registerExecutionControlWork(c, [ - { queue: DELIVERY_QUEUES.executionReplacement, run: () => gate, next: async () => null }, - { queue: DELIVERY_QUEUES.executionStatus, run: status, next: async () => null }, + { queue: heldQueue, run: () => gate, next: async () => null }, + { queue: DELIVERY_QUEUES.executionFinalization, run: status, next: async () => null }, ]); await vi.advanceTimersByTimeAsync(1); expect(status).toHaveBeenCalledTimes(1); let stopped = false; const stopping = c.stop().then(() => { stopped = true; }); @@ -120,6 +126,7 @@ describe.skipIf(!support.supported)("execution-control durable work", () => { if (external) await applyPendingMigrations(external); else temporary = await startEmbeddedPostgresTestDatabase("execution-sleep-"); db = createDb(external ?? temporary!.connectionString, { maxConnections: 6 }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.05)`))); }, 90000); afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); beforeEach(async () => { @@ -145,22 +152,140 @@ describe.skipIf(!support.supported)("execution-control durable work", () => { await settle(); expect(c.nextWakeAt()).toBeNull(); const before = calls; await vi.advanceTimersByTimeAsync(90_000); expect(calls).toBe(before); } - async function runRow(status = "running") { - const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status, runtimeMode: "legacy", contextSnapshot: { issueId } }).returning(); + async function runRow(status = "running", sourceIssueId = issueId) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status, runtimeMode: "legacy", contextSnapshot: { issueId: sourceIssueId } }).returning(); return run!; } - it("stops all six pending/deadline scans with empty queues and completed history", async () => { + async function pausedRecoveredSource(actionId = randomUUID()) { + const sourceIssueId = randomUUID(); + await db.insert(issues).values({ id: sourceIssueId, companyId, title: "Recovered execution", + status: "in_progress", assigneeAgentId: agentId }); + const run = await runRow("failed", sourceIssueId); + await db.update(heartbeatRuns).set({ resultJson: { startupCancellation: { beforeNativeSelection: true } } }) + .where(eq(heartbeatRuns.id, run.id)); + await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agentId)); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId, + kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted", + fingerprint: run.id, nextAction: "Saved work recovered", evidence: { runId: run.id, + systemRecovery: { ...createSystemRecovery(run.id), state: "complete" }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id, + status: "recovered", stage: "saving_receipts", attempt: 1, attemptLimit: 6, requestIds: [] } } }); + const retry = await requestTaskRetry({ db, companyId, issueId: sourceIssueId, actorId: "board", + observedRunId: run.id, idempotencyKey: randomUUID() }); + expect(retry.accepted).toBe(true); + return { run, retry }; + } + it("stops all pending/deadline scans with empty queues and completed history", async () => { await runRow("succeeded"); const schedule = executionControlSchedule(observed); const c = coordinator(db, undefined, error => { errors.push(error); }); - const topics = [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.executionContinuation, + const topics = [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.workspaceRecovery, DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.executionContinuation, DELIVERY_QUEUES.executionStatus, DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.localLoginCleanup]; - const runs = [() => reconcileAbandonedExecutionControl(observed), () => reconcileSafeNativeReplacements(observed), + const runs = [() => reconcileAbandonedExecutionControl(observed), async () => {}, () => reconcileSafeNativeReplacements(observed), () => deliverReconciledExecutions(observed, vi.fn(async () => null)), () => deliverExecutionStatuses(observed), () => settleUnrecoverableExecutions(observed), () => localAiLoginService(observed).reapExpired()]; await registerExecutionControlWork(c, Object.values(schedule).map((next, i) => ({ queue: topics[i]!, run: runs[i]!, next }))).ready; await quiet(c); }); + it("schedules only a current unfinished workspace recovery and becomes quiet with retained history", async () => { + const run = await runRow("failed"); + const state = createSystemRecovery(run.id); + const evidence = { runId: run.id, systemRecovery: state, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: run.id, + status: "queued", stage: "verifying_stop", attempt: 0, attemptLimit: 6, requestIds: [] } }; + const [action] = await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, + kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_retry_exhausted", + fingerprint: run.id, nextAction: "Save work", evidence }).returning(); + expect(await nextWorkspaceRecoveryAt(db)).toBe(0); + const due = Date.now() + 60_000; + await db.update(issueRecoveryActions).set({ evidence: { ...evidence, systemRecovery: { ...state, nextAttemptAt: due } } }) + .where(eq(issueRecoveryActions.id, action!.id)); + expect(await executionControlSchedule(db).workspaceRecovery()).toBe(due); + for (const status of ["cancelled", "resolved"] as const) { + await db.update(issueRecoveryActions).set({ status }).where(eq(issueRecoveryActions.id, action!.id)); + expect(await nextWorkspaceRecoveryAt(db)).toBeNull(); + expect(await retainedWorkspaceCandidates(db)).toEqual([]); + } + for (const system of [{ ...state, sourceRunId: randomUUID() }, { ...state, state: "complete" }, { ...state, state: "exhausted" }]) { + await db.update(issueRecoveryActions).set({ status: "active", evidence: { ...evidence, systemRecovery: system } }) + .where(eq(issueRecoveryActions.id, action!.id)); + expect(await nextWorkspaceRecoveryAt(db)).toBeNull(); + } + await db.update(issueRecoveryActions).set({ evidence, cause: "agent_unreachable" }) + .where(eq(issueRecoveryActions.id, action!.id)); + expect(await nextWorkspaceRecoveryAt(db)).toBeNull(); + await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", + evidence: { ...evidence, workspaceRecovery: { ...evidence.workspaceRecovery, runId: randomUUID() } } }) + .where(eq(issueRecoveryActions.id, action!.id)); + expect(await nextWorkspaceRecoveryAt(db)).toBeNull(); + expect(await retainedWorkspaceCandidates(db)).toEqual([]); + await db.update(issueRecoveryActions).set({ status: "cancelled", evidence: { ...evidence, + systemRecovery: { ...state, state: "running" } } }).where(eq(issueRecoveryActions.id, action!.id)); + const c = coordinator(db, undefined, error => { errors.push(error); }); + const worker = registerWorkspaceRecoveryWork(observed, c, + {} as Parameters[2], async () => {}); + try { await worker.ready; await quiet(c); } finally { await worker.stop(); } + }); + + it.each([1, 3, 7])("visits all %i paused Retry sources once and becomes quiet after file recovery", async count => { + const sources = []; + for (let index = 0; index < count; index++) sources.push(await pausedRecoveredSource()); + const resume = vi.fn(heartbeatService(observed).resumeRemoteStopComments); + const c = coordinator(db, undefined, error => { errors.push(error); }); + const worker = registerWorkspaceRecoveryWork(observed, c, + {} as Parameters[2], resume); + try { + await worker.ready; + for (let pass = 0; pass < 10 && resume.mock.calls.length < count; pass++) await settle(); + expect(resume).toHaveBeenCalledTimes(count); + await quiet(c); + for (const result of resume.mock.results) { + await expect(result.value).rejects.toThrow("Agent is not invokable in its current state"); + } + for (const { retry } of sources) { + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)); + expect(saved).toMatchObject({ status: "deferred_issue_execution", runId: null }); + } + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))).toHaveLength(count); + expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]?.status).toBe("paused"); + } finally { await c.stop(); await worker.stop(); } + }); + + it("visits a newly committed earlier Retry during a busy recovered-source pass, then becomes quiet", async () => { + const sources = []; + for (let index = 0; index < 3; index++) { + sources.push(await pausedRecoveredSource(`ffffffff-000${index}-4000-8000-${randomUUID().slice(-12)}`)); + } + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const heartbeat = heartbeatService(observed); + const resume = vi.fn(async (run: Parameters[0]) => { + await gate; + return heartbeat.resumeRemoteStopComments(run); + }); + const c = coordinator(db, undefined, error => { errors.push(error); }); + const worker = registerWorkspaceRecoveryWork(observed, c, + {} as Parameters[2], resume); + try { + await worker.ready; + await vi.waitFor(() => expect(resume).toHaveBeenCalledTimes(2)); + const earlier = await pausedRecoveredSource(`00000000-0000-4000-8000-${randomUUID().slice(-12)}`); + sources.push(earlier); + release(); + for (let pass = 0; pass < 10 && sources.some(({ run }) => + !resume.mock.calls.some(([visited]) => visited.id === run.id)); pass++) await settle(); + expect(resume.mock.calls.map(([run]) => run.id)).toContain(earlier.run.id); + await quiet(c); + for (const { run, retry } of sources) { + expect(resume.mock.calls.map(([visited]) => visited.id)).toContain(run.id); + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)); + expect(saved).toMatchObject({ status: "deferred_issue_execution", runId: null }); + } + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId))).toHaveLength(4); + expect((await db.select().from(agents).where(eq(agents.id, agentId)))[0]?.status).toBe("paused"); + } finally { release(); await c.stop(); await worker.stop(); } + }); + it("bounds real producer bursts with another company's completed history present", async () => { const otherCompany = randomUUID(), otherAgent = randomUUID(); await db.insert(companies).values({ id: otherCompany, name: "Other tenant", issuePrefix: "OTH" }); diff --git a/server/src/index.ts b/server/src/index.ts index 4212387dd04..b4e814959e1 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerWorkspaceRecoveryWork } from "./services/workspace-recovery-work.js"; import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-work.js"; import { loginCleanupSchedule } from "./services/login-cleanup-state.js"; import { registerLoginCleanupWork } from "./services/login-cleanup-work.js"; @@ -1342,6 +1343,9 @@ async function startServerWithDatabaseTeardown( }); const deliveryWork = app.locals.deliveryWork as ReturnType; const executionSchedule = executionControlSchedule(db); + const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager }); + const workspaceRecoveryWork = registerWorkspaceRecoveryWork(db, deliveryWork, workspaceRecoveryRuntime, + accountingHeartbeat.resumeRemoteStopComments, () => !heartbeatSchedulerStopped); registerExecutionControlWork(deliveryWork, [ { queue: DELIVERY_QUEUES.executionFinalization, run: () => reconcileAbandonedExecutionControl(db), next: executionSchedule.finalization }, { queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status }, @@ -2103,6 +2107,7 @@ async function startServerWithDatabaseTeardown( await skillSourceSync.stop(); unsubscribeChatCompletions(); await deliveryWork.stop(); + await workspaceRecoveryWork.stop(); unregisterTerminalWorkspaceProbe?.(); unsubscribeNativeWorkspaceSyncRetention?.(); if (heartbeatSchedulerInterval) { diff --git a/server/src/services/agent-runner-selection.test.ts b/server/src/services/agent-runner-selection.test.ts index afec21d6b45..afc3949acdf 100644 --- a/server/src/services/agent-runner-selection.test.ts +++ b/server/src/services/agent-runner-selection.test.ts @@ -74,7 +74,7 @@ describe("server-owned Codex runner selection", () => { const input = { adapterType: "codex_local", target, adapterConfig: { model: "gpt-5.6-sol", modelReasoningEffort: "high", env: { OPENAI_API_KEY: { type: "secret_ref", secretId: "secret" } }, cwd: "/workspace", instructionsFilePath: "/workspace/AGENTS.md", workspaceStrategy: { type: "git_worktree" }, timeoutSec: 60, search: false, fastMode: false, dangerouslyBypassApprovalsAndSandbox: true } }; const resolved = resolveNewAgentRunner(input); expect(resolved).toMatchObject({ adapterType: "paperclip_runner", adapterConfig: { provider: "codex", codexPermissionMode: "never", modelReasoningEffort: "high", env: input.adapterConfig.env, workspaceStrategy: input.adapterConfig.workspaceStrategy, timeoutSec: 60 } }); - expect(resolved.adapterConfig).not.toHaveProperty("search"); + expect(resolved.adapterConfig).toHaveProperty("search", false); expect(resolveNewAgentRunner(resolved)).toEqual(resolved); expect(input.adapterConfig).toHaveProperty("search", false); }); diff --git a/server/src/services/agent-runner-selection.ts b/server/src/services/agent-runner-selection.ts index 7d87a5b9d89..904e3c7aae3 100644 --- a/server/src/services/agent-runner-selection.ts +++ b/server/src/services/agent-runner-selection.ts @@ -270,7 +270,7 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; incompatible(["env.PAPERCLIP_CODEX_ACP_NETWORK_ACCESS"]); } const effort = config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort; - for (const key of ["engine", "command", "extraArgs", "args", "agentCommand", "mode", "nonInteractivePermissions", "stateDir", "warmHandleIdleMs", "dangerouslyBypassApprovalsAndSandbox", "dangerouslyBypassSandbox", "search", "fastMode", "reasoningEffort", "effort"]) delete config[key]; + for (const key of ["engine", "command", "extraArgs", "args", "agentCommand", "mode", "nonInteractivePermissions", "stateDir", "warmHandleIdleMs", "dangerouslyBypassApprovalsAndSandbox", "dangerouslyBypassSandbox", "fastMode", "reasoningEffort", "effort"]) delete config[key]; if (config.model !== undefined && typeof config.model !== "string") incompatible(["model"]); config = { ...config, provider: "codex", model: typeof config.model === "string" && config.model.trim() ? config.model.trim() : PAPERCLIP_RUNNER_DEFAULT_MODELS.codex, codexPermissionMode: config.codexPermissionMode ?? "never", lifecycleMode: config.lifecycleMode ?? "per_turn", ...(effort === undefined ? {} : { modelReasoningEffort: effort }) }; try { resolvePaperclipRunnerNativeProviderInput({ backend: "codex_app_server", adapterConfig: config }); } diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index e53608b49de..fc41debb711 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,4 @@ +import { credentialFingerprint } from "./chat-channels/credential-fingerprint.js"; import { createChatSdkMessageReceiver } from "./chat-channels/runtime/sdk-messages.js"; import { admitDiscordSdkMessage } from "./chat-channels/discord/messages.js"; import { resolveInboundConversationBinding } from "./chat-channels/inbound/conversation-binding.js"; @@ -312,20 +313,6 @@ function runtimeGeneration(setup: ChatEndpointSetupState): number { return Number.isSafeInteger(value) && value >= 0 ? value : 0; } -function credentialFingerprint(refs: ToolCredentialSecretRef[]): string { - const stable = refs - .map((ref) => ({ - configPath: ref.configPath, - secretId: ref.secretId, - versionSelector: ref.versionSelector ?? "latest", - })) - .sort((left, right) => - `${left.configPath}:${left.secretId}:${left.versionSelector}`.localeCompare( - `${right.configPath}:${right.secretId}:${right.versionSelector}`, - ), - ); - return createHash("sha256").update(JSON.stringify(stable)).digest("hex"); -} type ConversationRow = typeof chatConversations.$inferSelect; type DeliveryRow = typeof chatDeliveries.$inferSelect; @@ -847,7 +834,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { const { lockTelegramCallbackNoticeAuthority } = createTelegramCallbackNoticeAuthority({ nonDirectDestinationAllowed, lockCurrentPrincipalAuthorization }); const { stageProviderEffect, processProviderEffect, processPendingProviderEffects } = createProviderEffects(db, { endpointRecord, runtimeCallbackEndpoint, withCredentialMutationLease, runtimeFor, invalidateRuntime, lockCurrentPrincipalAuthorization, redactError, MAX_ERROR_TEXT, nonDirectDestinationAllowed, lockTelegramCallbackNoticeAuthority }); const { publicationFiles } = createOutboundAttachments(db, { get storage() { return options.storage; } }); - const { authorizeInboundWakeup, authorizeInboundWorkStart, authorizeFastResponse } = createInboundWakeAuthority(db, { runtimeCallbackEndpoint, lockCurrentPrincipalAuthorization }); + const { authorizeInboundWakeup, authorizeGitHubTaskRetryBatch, authorizeInboundWorkStart, authorizeFastResponse } = createInboundWakeAuthority(db, { runtimeCallbackEndpoint, lockCurrentPrincipalAuthorization }); const { ingestAttachments, attachmentOmissionDetail } = createInboundAttachments(db, { authorizeInboundWakeup, lifecycleRuntimeFence, sanitizeFilename, issuesSvc, isExternalActionAuthorizationChange, redactError, get storage() { return options.storage; } }); const { stageInboundWakeup, acceptInboundWakeup, settleRejectedInboundWakeups, processInboundWakeup } = createInboundWakeups(db, { authorizeInboundWakeup, authorizeInboundWorkStart, attachmentOmissionDetail, isExternalActionAuthorizationChange, redactError, fetchImpl, get heartbeat() { return options.heartbeat; } }); const { settleTelegramAttachmentOnlyFailure } = createTelegramAttachmentFailure(db, { @@ -1203,6 +1190,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { pendingFailedRetries, } = createChatFailedRunRetry(db, { authorizeInboundWakeup, + authorizeGitHubTaskRetryBatch, isExternalActionAuthorizationChange, inboundWakeScope, lifecycleRuntimeFence, diff --git a/server/src/services/chat-channels/credential-fingerprint.ts b/server/src/services/chat-channels/credential-fingerprint.ts new file mode 100644 index 00000000000..57f472140c7 --- /dev/null +++ b/server/src/services/chat-channels/credential-fingerprint.ts @@ -0,0 +1,10 @@ +import { createHash } from "node:crypto"; +import type { ToolCredentialSecretRef } from "@paperclipai/shared"; + +export function credentialFingerprint(refs: ToolCredentialSecretRef[]): string { + const stable = refs.map(ref => ({ configPath: ref.configPath, secretId: ref.secretId, + versionSelector: ref.versionSelector ?? "latest" })).sort((left, right) => + `${left.configPath}:${left.secretId}:${left.versionSelector}`.localeCompare( + `${right.configPath}:${right.secretId}:${right.versionSelector}`)); + return createHash("sha256").update(JSON.stringify(stable)).digest("hex"); +} diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 6310d448a66..c18a15bdcb8 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -3,6 +3,8 @@ import { readTaskChatRetryIntent, taskChatRetryCanAwaitRecovery, taskChatRetryEx import { notifyChatActionWork, notifyChatPublicationWork } from "../chat-work-notifications.js"; import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js"; import { readChatAuthorizationRun } from "./authorization-run-read.js"; +import { selectTaskRetryInput, claimTaskRetryInput, settleTaskRetryInput, releaseUnadmittedTaskRetryInput, taskRetryInputSchema, type TaskRetryInput } from "./task-retry-input.js"; +import type { TaskRetryWakeAuthority } from "./github/task-retry-authorization.js"; import { createHash, @@ -93,14 +95,16 @@ type RetryRuntimeFence = { credentialFingerprint: string; generation: number }; export type ChatFailedRetrySource = Awaited["failedChatRetrySource"]>>; type ChatFailedRunRetryOptions = { + authorizeGitHubTaskRetryBatch: ReturnType["authorizeGitHubTaskRetryBatch"]; authorizeInboundWakeup: ( database: DbOrTransaction, action: typeof chatActions.$inferSelect, - notice?: { nonblocking: true; terminal: boolean }, + notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority }, ) => Promise<{ conversation: Pick; endpoint: Pick; delivery: DeliveryRow; + retryRuntimeFence?: RetryRuntimeFence; }>; isExternalActionAuthorizationChange: (error: unknown) => boolean; inboundWakeScope: (action: typeof chatActions.$inferSelect) => string; @@ -112,6 +116,7 @@ type ChatFailedRunRetryOptions = { export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOptions) { const { authorizeInboundWakeup, + authorizeGitHubTaskRetryBatch, isExternalActionAuthorizationChange, inboundWakeScope, lifecycleRuntimeFence, @@ -131,14 +136,14 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti } } - function retryCommentIds(context: Record): string[] { + function retryCommentIds(context: Record, taskRetryInput?: TaskRetryInput): string[] { const ids = context.wakeCommentIds ?? (context.wakeCommentId ? [context.wakeCommentId] : []); if ( !Array.isArray(ids) || ids.length === 0 || - ids.length > 50 || + (!taskRetryInput && ids.length > 50) || ids.some((id) => typeof id !== "string" || !isUuidLike(id)) || new Set(ids).size !== ids.length || (context.wakeCommentId !== undefined && @@ -168,6 +173,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti failedRunId: string; sourceScopeSha256: string; }>; + taskRetryInput?: TaskRetryInput; sourceScopeSha256: string; }; function hashFailedRetrySource( @@ -651,6 +657,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti runId?: string; publication?: boolean; taskRetryIntentId?: string; + taskRetryInput?: TaskRetryInput; + taskRetryReauthorization?: TaskRetryWakeAuthority; /** Internal selector for already committed output, never retry admission. */ committedResponse?: { resultId: string; @@ -734,7 +742,22 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti (context.taskKey !== issue.identifier && context.taskKey !== issue.id) ) throw failedChatRetryDenied(); - const commentIds = retryCommentIds(context); + const selection = input.taskRetryInput && taskRetryInputSchema.safeParse(input.taskRetryInput); + if (selection && !selection.success) throw failedChatRetryDenied(); + const selectedInput = selection?.success ? selection.data : undefined; + if (selectedInput && (!input.taskRetryIntentId || provider !== "github")) throw failedChatRetryDenied(); + const reauthorization = input.taskRetryReauthorization ?? (selectedInput ? { + intentId: input.taskRetryIntentId!, sourceRunId: failedRun.id, runtimeFence: selectedInput.runtimeFence, + } : undefined); + const pendingSources = selectedInput?.sourceActionIds.length ? await tx.select().from(chatActions).where(and( + eq(chatActions.companyId, input.companyId), eq(chatActions.kind, "inbound_wakeup"), + sql`${chatActions.payload}->>'issueId' = ${issue.id}`, + inArray(chatActions.id, selectedInput.sourceActionIds), + )) : null; + if (pendingSources && pendingSources.length !== selectedInput!.sourceActionIds.length) throw failedChatRetryDenied(); + const pendingById = new Map(pendingSources?.map(source => [source.id, source])); + const commentIds = pendingSources ? selectedInput!.sourceActionIds.map(id => + String(pendingById.get(id)!.payload.commentId)) : retryCommentIds(context, selectedInput); const [accepted] = await tx .select() .from(nativeRunResults) @@ -868,7 +891,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti answerDelivery ) throw failedChatRetryDenied(); - if (context.chatFailedRunRetry) { + if (context.chatFailedRunRetry && !pendingSources) { const hint = context.chatFailedRunRetry as Record; const [priorIntent] = await tx .select() @@ -914,6 +937,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti committedResponse: undefined, taskRetryIntentId: typeof priorIntent.payload.taskRetryIntentId === "string" ? priorIntent.payload.taskRetryIntentId : undefined, + taskRetryInput: priorIntent.payload.taskRetryInput as TaskRetryInput | undefined, + retryActionId: priorIntent.id, + taskRetryReauthorization: reauthorization, }, ancestors, ); @@ -945,6 +971,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti return hashFailedRetrySource({ ...parentScope, failedRunId: failedRun.id, + ...(selectedInput ? { taskRetryInput: { ...selectedInput, runtimeFence: reauthorization?.runtimeFence } } : {}), retryAncestors: [ ...parent.retryAncestors, { @@ -958,20 +985,21 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti // Include every receipt that fed this run, including a deferred owner's // coalesced children. A mutable latest-comment hint cannot drop a sibling. const admittedBatch = - await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id + pendingSources ? [] : await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id from agent_wakeup_requests receipt join chat_actions source on source.id = receipt.id where receipt.company_id = ${input.companyId}::uuid and receipt.agent_id = ${input.agentId}::uuid and (receipt.run_id = ${failedRun.id}::uuid or receipt.id = ${failedRun.wakeupRequestId}::uuid or receipt.payload->>'coalescedIntoWakeupRequestId' = ${failedRun.wakeupRequestId}) - and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' limit 51`); + and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' + ${selectedInput ? sql`` : sql`limit 51`}`); if ( - admittedBatch.length !== commentIds.length || + !pendingSources && (admittedBatch.length !== commentIds.length || admittedBatch.some( (entry) => !commentIds.includes(String(entry.comment_id)), - ) + )) ) throw failedChatRetryDenied(); - const sources = await tx + const sources = pendingSources ?? await tx .select() .from(chatActions) .where( @@ -986,8 +1014,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ), ); if (sources.length !== commentIds.length) throw failedChatRetryDenied(); + const sourcesByCommentId = new Map(sources.map(source => [source.payload.commentId, source])); const ordered = commentIds.map( - (id) => sources.find((action) => action.payload.commentId === id)!, + (id) => sourcesByCommentId.get(id)!, ); const first = ordered[0]!; if ( @@ -1011,39 +1040,53 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ); if (!identity[0]?.locked) throw new NativeChatReviewPresentationContentionError(); + const actionIds = ordered.map(action => action.id); + const currentActions = await tx.select().from(chatActions).where(and(eq(chatActions.companyId, input.companyId), + inArray(chatActions.id, actionIds))).orderBy(asc(chatActions.id)).for("update", { noWait: true }); + const deliveries = await tx.select().from(chatDeliveries).where(and(eq(chatDeliveries.companyId, input.companyId), + inArray(chatDeliveries.id, ordered.map(action => action.deliveryId!)))) + .orderBy(asc(chatDeliveries.id)).for("update", { noWait: true }); + const comments = await tx.select().from(issueComments).where(and(eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, issue.id), inArray(issueComments.id, commentIds))) + .orderBy(asc(issueComments.id)).for("share", { noWait: true }); + const receipts = await tx.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, input.companyId), + inArray(agentWakeupRequests.id, [...actionIds, failedRun.wakeupRequestId!, ...(input.retryActionId ? [input.retryActionId] : [])]))) + .orderBy(asc(agentWakeupRequests.id)).for("share", { noWait: true }); + const actionsById = new Map(currentActions.map(action => [action.id, action])); + const deliveriesById = new Map(deliveries.map(delivery => [delivery.id, delivery])); + const commentsById = new Map(comments.map(comment => [comment.id, comment])); + const receiptsById = new Map(receipts.map(receipt => [receipt.id, receipt])); + const lifecycleRows = await tx.select({ + target: sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId'`, + context: sql>`${chatDeliveries.normalizedEvent}->'runtimeContext'`, + }).from(chatDeliveries).where(and(eq(chatDeliveries.companyId, input.companyId), + eq(chatDeliveries.endpointId, first.endpointId), + or(isNull(chatDeliveries.conversationId), eq(chatDeliveries.conversationId, first.conversationId)), + ne(chatDeliveries.state, "filtered"), inArray(chatDeliveries.eventKind, ["message_updated", "message_deleted", "message_restored"]), + inArray(sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId'`, deliveries.map(delivery => delivery.providerEventId)))); + const lifecycleKey = (target: string, context: Record | null) => + JSON.stringify([target, Object.entries(context ?? {}).sort(([left], [right]) => left.localeCompare(right))]); + const invalidated = new Set(lifecycleRows.map(row => lifecycleKey(row.target, row.context))); + const batch = reauthorization ? await authorizeGitHubTaskRetryBatch(tx, ordered, { + nonblocking: true, terminal: input.publication === true, taskRetry: reauthorization, + }).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(); throw error; }) : null; const canonicalSources: Array> = []; let destination: Awaited> | null = null; for (const action of ordered) { - destination = await authorizeInboundWakeup(tx, action, { + destination = batch?.get(action.id) ?? await authorizeInboundWakeup(tx, action, { nonblocking: true, terminal: input.publication === true, + ...(reauthorization ? { taskRetry: reauthorization } : {}), }).catch((error) => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(); throw error; }); - const [delivery] = await tx - .select() - .from(chatDeliveries) - .where(eq(chatDeliveries.id, action.deliveryId!)) - .for("update", { noWait: true }); - const [current] = await tx - .select() - .from(chatActions) - .where(eq(chatActions.id, action.id)) - .for("update", { noWait: true }); - const [comment] = await tx - .select() - .from(issueComments) - .where( - and( - eq(issueComments.id, String(action.payload.commentId)), - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, issue.id), - ), - ) - .for("share", { noWait: true }); + if (reauthorization && !reauthorization.runtimeFence) reauthorization.runtimeFence = destination.retryRuntimeFence; + const delivery = deliveriesById.get(action.deliveryId!); + const current = actionsById.get(action.id); + const comment = commentsById.get(String(action.payload.commentId)); if ( !current || current.status !== "processed" || @@ -1053,43 +1096,17 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti !comment || comment.deletedAt || comment.updatedAt.getTime() !== comment.createdAt.getTime() || - comment.updatedAt > + (!pendingSources && comment.updatedAt > (input.committedResponse ? accepted!.createdAt - : failedRun.finishedAt!) || + : failedRun.finishedAt!)) || destination.endpoint.provider !== provider || JSON.stringify(lifecycleRuntimeFence(delivery)) !== JSON.stringify(lifecycleRuntimeFence(destination.delivery)) ) throw failedChatRetryDenied(); - const [lifecycle] = await tx - .select({ id: chatDeliveries.id }) - .from(chatDeliveries) - .where( - and( - eq(chatDeliveries.companyId, input.companyId), - eq(chatDeliveries.endpointId, first.endpointId), - or( - isNull(chatDeliveries.conversationId), - eq(chatDeliveries.conversationId, first.conversationId), - ), - ne(chatDeliveries.state, "filtered"), - sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(delivery))}::jsonb`, - inArray(chatDeliveries.eventKind, [ - "message_updated", - "message_deleted", - "message_restored", - ]), - sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${delivery.providerEventId}`, - ), - ) - .limit(1); - if (lifecycle) throw failedChatRetryDenied(); - const [receipt] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, action.id)) - .for("share", { noWait: true }); + if (invalidated.has(lifecycleKey(delivery.providerEventId, lifecycleRuntimeFence(delivery)))) throw failedChatRetryDenied(); + const receipt = receiptsById.get(action.id); if (!receipt) throw failedChatRetryDenied(); assertFailedRetryReceipt( createDurableChatWakeupRequest({ @@ -1107,6 +1124,17 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti }), receipt, ); + if (pendingSources) { + const direct = receipt.status === "deferred_issue_execution" && !receipt.runId + && receipt.payload?.coalescedIntoWakeupRequestId === undefined; + const owned = input.retryActionId && receipt.status === "coalesced" + && receipt.payload?.coalescedIntoWakeupRequestId === input.retryActionId; + if (!direct && !owned) throw failedChatRetryDenied(); + if (owned && receipt.runId) { + const owner = receiptsById.get(input.retryActionId!); + if (owner?.runId !== receipt.runId) throw failedChatRetryDenied(); + } + } let owner = receipt; const seen = new Set(); for ( @@ -1126,25 +1154,16 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ) throw failedChatRetryDenied(); seen.add(ownerId); - const [next] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, ownerId), - eq(agentWakeupRequests.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); + const next = receiptsById.get(ownerId); if (!next) throw failedChatRetryDenied(); owner = next; } if ( - owner.id !== failedRun.wakeupRequestId || + (!pendingSources && (owner.id !== failedRun.wakeupRequestId || owner.runId !== failedRun.id || owner.agentId !== input.agentId || owner.requestedByActorType !== receipt.requestedByActorType || - owner.requestedByActorId !== receipt.requestedByActorId || + owner.requestedByActorId !== receipt.requestedByActorId)) || ["skipped", "cancelled"].includes(receipt.status) ) throw failedChatRetryDenied(); @@ -1158,7 +1177,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti bodySha256: createHash("sha256").update(comment.body).digest("hex"), omissionReasons: action.payload.attachmentOmissionReasons ?? {}, receiptId: receipt.id, - ownerId: owner.id, + ownerId: pendingSources ? action.id : owner.id, }); } if (!destination) throw failedChatRetryDenied(); @@ -1204,6 +1223,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti | "system", requestedByActorId: String(first.payload.requestedByActorId), retryAncestors: [], + ...(selectedInput ? { taskRetryInput: { ...selectedInput, + runtimeFence: selectedInput.runtimeFence ?? destination.retryRuntimeFence } } : {}), }; return hashFailedRetrySource(scope); } @@ -1240,7 +1261,8 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti retryIntentId?: string; }, ): Promise<{ actionId: string; issueId: string }> { - const source = await failedChatRetrySource(tx, { ...input, taskRetryIntentId: input.retryIntentId }); + const taskRetryInput = input.retryIntentId ? await selectTaskRetryInput(tx, input) : undefined; + const source = await failedChatRetrySource(tx, { ...input, taskRetryIntentId: input.retryIntentId, taskRetryInput }); const providerActionId = input.retryIntentId ? `task-retry:${input.retryIntentId}` : `failed-run-retry:${source.failedRunId}`; const [existing] = await tx @@ -1351,6 +1373,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti failedRunId: snapshot.payload.failedRunId, taskRetryIntentId: typeof snapshot.payload.taskRetryIntentId === "string" ? snapshot.payload.taskRetryIntentId : undefined, + taskRetryInput: snapshot.payload.taskRetryInput as TaskRetryInput | undefined, }); if (typeof snapshot.payload.taskRetryIntentId === "string" && !await readTaskChatRetryIntent(tx as unknown as Db, { companyId: input.companyId, issueId: input.issueId, agentId: input.agentId, @@ -1444,7 +1467,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti context.source !== `chat:${source.provider}` || context.taskKey !== source.taskKey || context.retryOfRunId !== source.failedRunId || - JSON.stringify(retryCommentIds(context)) !== + JSON.stringify(retryCommentIds(context, source.taskRetryInput)) !== JSON.stringify(source.commentIds) || hint?.version !== 1 || hint.actionId !== source.intent.id || @@ -1518,8 +1541,19 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti }; } + async function reconcileFailedTaskRetry(action: typeof chatActions.$inferSelect) { + if (!["failed", "cancelled"].includes(action.status) || !action.payload.taskRetryIntentId || + action.result?.taskRetryInputReconciled === true) return action; + await releaseUnadmittedTaskRetryInput(db, { companyId: action.companyId, retryActionId: action.id }); + const [settled] = await db.update(chatActions).set({ + result: sql`coalesce(${chatActions.result}, '{}'::jsonb) || '{"taskRetryInputReconciled":true}'::jsonb`, + }).where(and(eq(chatActions.companyId, action.companyId), eq(chatActions.id, action.id), + inArray(chatActions.status, ["failed", "cancelled"]))).returning(); + return settled ?? action; + } + async function claimFailedChatRunRetry(actionId: string) { - const [candidate] = await db + const [snapshot] = await db .select() .from(chatActions) .where( @@ -1528,7 +1562,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti eq(chatActions.kind, "failed_run_retry"), ), ); - if (!candidate) throw notFound("Chat retry request not found"); + if (!snapshot) throw notFound("Chat retry request not found"); + // Reconcile once at settlement; a crash before the marker is safely + // retried here. Status projection itself performs no writes or row locks. + const candidate = await reconcileFailedTaskRetry(snapshot); const now = new Date(); if ( isShuttingDown() || @@ -1571,7 +1608,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti ) .returning(); }); - return failedChatRetryReceipt(current ?? claimed); + return failedChatRetryReceipt(await reconcileFailedTaskRetry(current ?? claimed)); }; try { const scope = { @@ -1644,6 +1681,12 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti true, // The durable wake callback validates admission; later phases recheck containment. ); }, + onAdmitted: async (tx, receipt) => { + if (!source.taskRetryInput) return; + await claimTaskRetryInput(tx, { companyId: claimed.companyId, + retryActionId: claimed.id, sourceActionIds: source.taskRetryInput.sourceActionIds }); + await settleTaskRetryInput(tx, { companyId: claimed.companyId, retryActionId: claimed.id, runId: receipt.runId }); + }, }), }); } @@ -1652,6 +1695,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti .from(agentWakeupRequests) .where(eq(agentWakeupRequests.id, claimed.id)); if (!receipt) throw new Error("chat_retry_receipt_not_committed"); + await settleTaskRetryInput(db, { companyId: claimed.companyId, retryActionId: claimed.id, runId: receipt.runId }); return settle( ["skipped", "failed", "cancelled"].includes(receipt.status) ? "failed" diff --git a/server/src/services/chat-channels/github/task-preparation.ts b/server/src/services/chat-channels/github/task-preparation.ts index 403b45c537e..d44ce61de02 100644 --- a/server/src/services/chat-channels/github/task-preparation.ts +++ b/server/src/services/chat-channels/github/task-preparation.ts @@ -37,7 +37,7 @@ export async function retainGitHubDeliveryAuthority( linkedUserId: string | null, taskUserId: string | null, ): Promise { const guest = !linkedUserId; - await taskTx.update(chatDeliveries).set({ normalizedEvent: sql`${chatDeliveries.normalizedEvent} || ${JSON.stringify({ githubAuthority: { guest, responsibleUserId: taskUserId, sponsorUserId: guest ? taskEndpoint.sponsorUserId : null } })}::jsonb` }).where(eq(chatDeliveries.id, deliveryId)); + await taskTx.update(chatDeliveries).set({ normalizedEvent: sql`${chatDeliveries.normalizedEvent} || ${JSON.stringify({ githubAuthority: { guest, botExternalId: taskEndpoint.botExternalId, responsibleUserId: taskUserId, sponsorUserId: guest ? taskEndpoint.sponsorUserId : null } })}::jsonb` }).where(eq(chatDeliveries.id, deliveryId)); } export async function retainGitHubReviewReceipt(taskTx: DbOrTransaction, endpoint: EndpointRow, issueId: string, deliveryId: string, githubAutomatic: AutomaticRequest): Promise { diff --git a/server/src/services/chat-channels/github/task-retry-authorization.ts b/server/src/services/chat-channels/github/task-retry-authorization.ts new file mode 100644 index 00000000000..52e51f38ebf --- /dev/null +++ b/server/src/services/chat-channels/github/task-retry-authorization.ts @@ -0,0 +1,126 @@ +import { and, eq, inArray } from "drizzle-orm"; +import { chatActions, chatConversations, chatDeliveries, chatEndpointResources, chatMessageLinks, issueComments, toolConnections } from "@paperclipai/db"; +import { forbidden } from "../../../errors.js"; +import type { DbOrTransaction, EndpointRow, LifecycleRuntimeFence } from "../runtime-types.js"; +import { credentialFingerprint } from "../credential-fingerprint.js"; +import { readTaskChatRetryIntent } from "../task-retry-authority.js"; + +export type TaskRetryWakeAuthority = { + intentId: string; + sourceRunId: string; + runtimeFence?: LifecycleRuntimeFence; +}; + +export function githubTaskRetryAppMatches(endpoint: EndpointRow, delivery: typeof chatDeliveries.$inferSelect, + response: { payload: Record } | undefined) { + const original = delivery.normalizedEvent.githubAuthority as { botExternalId?: unknown } | undefined; + return original?.botExternalId !== undefined ? original.botExternalId === endpoint.botExternalId + : response?.payload.botExternalId === endpoint.botExternalId && response.payload.agentId === endpoint.assignedAgentId; +} + +export function githubTaskRetryDestinationMatches(delivery: typeof chatDeliveries.$inferSelect, + conversation: typeof chatConversations.$inferSelect, resource: typeof chatEndpointResources.$inferSelect | null) { + const originalResource = delivery.normalizedEvent.resource as { providerResourceId?: unknown } | undefined; + const originalThread = delivery.normalizedEvent.conversation as { externalThreadId?: unknown } | undefined; + const automatic = (delivery.normalizedEvent.githubAutomatic ?? delivery.normalizedEvent.githubIssue) as + { context?: { repositoryId?: unknown } } | undefined; + return resource && originalResource?.providerResourceId === resource.providerResourceId + && originalThread?.externalThreadId === conversation.externalThreadId + && (automatic?.context?.repositoryId === undefined + || automatic.context.repositoryId === resource.metadata.providerRepositoryId); +} + +export async function githubTaskRetryRuntimeFence(tx: DbOrTransaction, endpoint: EndpointRow, + action: typeof chatActions.$inferSelect, delivery: typeof chatDeliveries.$inferSelect, + authority: TaskRetryWakeAuthority): Promise { + const deny = () => forbidden("This accepted chat message is no longer authorized to start work", + { code: "chat_action_authorization_changed" }); + const retry = await readTaskChatRetryIntent(tx as Parameters[0], { + companyId: action.companyId, issueId: String(action.payload.issueId), + agentId: String(action.payload.agentId), sourceRunId: authority.sourceRunId, intentId: authority.intentId, + }); + if (!retry || endpoint.provider !== "github" || !endpoint.botExternalId || action.kind !== "inbound_wakeup" + || action.status !== "processed" || delivery.state !== "processed") throw deny(); + const [response] = await tx.select({ payload: chatActions.payload }).from(chatActions).where(and( + eq(chatActions.companyId, action.companyId), eq(chatActions.endpointId, endpoint.id), + eq(chatActions.conversationId, action.conversationId!), eq(chatActions.deliveryId, delivery.id), + eq(chatActions.kind, "github_response_comment"), + )); + if (!githubTaskRetryAppMatches(endpoint, delivery, response)) throw deny(); + const [connection] = await tx.select({ refs: toolConnections.credentialSecretRefs, + enabled: toolConnections.enabled, status: toolConnections.status }).from(toolConnections).where(and( + eq(toolConnections.companyId, endpoint.companyId), eq(toolConnections.id, endpoint.connectionId), + )); + if (!connection?.enabled || connection.status !== "active") throw deny(); + const generation = Number((endpoint.setup as { runtimeGeneration?: unknown }).runtimeGeneration ?? 0); + if (!Number.isSafeInteger(generation) || generation < 0) throw deny(); + return authority.runtimeFence ?? { generation, credentialFingerprint: credentialFingerprint(connection.refs ?? []) }; +} + +type GitHubRetryBinding = { + endpoint: EndpointRow; + conversation: typeof chatConversations.$inferSelect; + resource: typeof chatEndpointResources.$inferSelect | null; + delivery: typeof chatDeliveries.$inferSelect; + retryRuntimeFence?: LifecycleRuntimeFence; +}; + +export function createGitHubTaskRetryBatchAuthority(authorize: ( + tx: DbOrTransaction, action: typeof chatActions.$inferSelect, + notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority }, +) => Promise) { + async function authorizeGitHubTaskRetryBatch(tx: DbOrTransaction, actions: Array, + notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority }) { + const deny = () => forbidden("This accepted chat message is no longer authorized to start work", + { code: "chat_action_authorization_changed" }); + const first = actions[0]; + if (!first) throw deny(); + const commonScope = (action: typeof chatActions.$inferSelect) => JSON.stringify([ + action.companyId, action.endpointId, action.conversationId, action.principalId, action.kind, action.status, + ...["version", "issueId", "agentId", "sessionGeneration", "requestedByActorType", "requestedByActorId"] + .map(key => action.payload[key]), + ]); + if (actions.some(action => commonScope(action) !== commonScope(first))) throw deny(); + const deliveries = await tx.select().from(chatDeliveries).where(and(eq(chatDeliveries.companyId, first.companyId), + inArray(chatDeliveries.id, actions.map(action => action.deliveryId!)))); + const byDelivery = new Map(deliveries.map(delivery => [delivery.id, delivery])); + const legacy = await tx.select().from(chatActions).where(and(eq(chatActions.companyId, first.companyId), + eq(chatActions.endpointId, first.endpointId), eq(chatActions.conversationId, first.conversationId!), + eq(chatActions.kind, "github_response_comment"), inArray(chatActions.deliveryId, deliveries.map(delivery => delivery.id)))); + const legacyByDelivery = new Map(legacy.map(action => [action.deliveryId, action])); + const links = await tx.select({ deliveryId: chatMessageLinks.deliveryId, commentId: chatMessageLinks.commentId, + authorUserId: issueComments.authorUserId, deletedAt: issueComments.deletedAt }).from(chatMessageLinks) + .innerJoin(issueComments, and(eq(issueComments.id, chatMessageLinks.commentId), + eq(issueComments.companyId, first.companyId), eq(issueComments.issueId, String(first.payload.issueId)))) + .where(and(eq(chatMessageLinks.companyId, first.companyId), eq(chatMessageLinks.endpointId, first.endpointId), + eq(chatMessageLinks.conversationId, first.conversationId!), eq(chatMessageLinks.direction, "inbound"), + inArray(chatMessageLinks.deliveryId, deliveries.map(delivery => delivery.id)))); + const expectedUserId = first.payload.requestedByActorType === "user" ? first.payload.requestedByActorId : null; + const linked = new Set(links.filter(link => (!link.deletedAt || notice.terminal) && link.authorUserId === expectedUserId) + .map(link => `${link.deliveryId}:${link.commentId}`)); + const groups = new Map(); + const authorized = new Map(); + for (const action of actions) { + const delivery = byDelivery.get(action.deliveryId!); + if (!delivery || !linked.has(`${delivery.id}:${action.payload.commentId}`)) throw deny(); + // Reuse only the same live grant within this transaction. Automatic review + // policies with different contexts still receive their own full check. + const grant = JSON.stringify(delivery.normalizedEvent.githubAutomatic ?? delivery.normalizedEvent.githubIssue ?? null); + let current = groups.get(grant); + if (!current) { + current = await authorize(tx, action, notice); + if (current.endpoint.provider !== "github") throw deny(); + notice.taskRetry.runtimeFence ??= current.retryRuntimeFence; + groups.set(grant, current); + } + if (delivery.state !== "processed" || delivery.endpointId !== current.endpoint.id || + delivery.conversationId !== current.conversation.id || delivery.principalId !== action.principalId || + !githubTaskRetryAppMatches(current.endpoint, delivery, legacyByDelivery.get(delivery.id)) || + !githubTaskRetryDestinationMatches(delivery, current.conversation, current.resource)) throw deny(); + authorized.set(action.id, { ...current, delivery }); + } + return authorized; + } + + return authorizeGitHubTaskRetryBatch; +} diff --git a/server/src/services/chat-channels/inbound/wakeup-authorization.ts b/server/src/services/chat-channels/inbound/wakeup-authorization.ts index ecdca242e9b..f4da4fa7cce 100644 --- a/server/src/services/chat-channels/inbound/wakeup-authorization.ts +++ b/server/src/services/chat-channels/inbound/wakeup-authorization.ts @@ -11,6 +11,7 @@ import { agentWakeupRequests, agents, chatActions, chatConversations, chatDelive import { forbidden } from "../../../errors.js"; import { resolveGitHubWakeAuthority, isGitHubAutomaticDelivery } from "../github/wake-authority.js"; import { assertSlackWakeMention } from "../slack/wake-authority.js"; +import { githubTaskRetryRuntimeFence, githubTaskRetryDestinationMatches, createGitHubTaskRetryBatchAuthority, type TaskRetryWakeAuthority } from "../github/task-retry-authorization.js"; import type { createChatInboundAuthority } from "./authorization.js"; import type { createChatMessageLifecycle } from "./lifecycle.js"; type WakeAuthorityOptions = { @@ -41,7 +42,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions async function authorizeInboundWakeup( tx: DbOrTransaction, action: typeof chatActions.$inferSelect, - notice?: { nonblocking: true; terminal: boolean }, + notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority }, ) { const payload = action.payload; const deny = () => @@ -106,6 +107,8 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions .limit(1) .then((rows) => rows[0] ?? null); const fence = delivery ? lifecycleRuntimeFence(delivery) : null; + const currentFence = endpoint && delivery && notice?.taskRetry + ? await githubTaskRetryRuntimeFence(tx, endpoint, action, delivery, notice.taskRetry) : fence; if ( !endpoint || endpoint.assignedAgentId !== payload.agentId || @@ -114,8 +117,8 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions delivery.endpointId !== endpoint.id || delivery.conversationId !== action.conversationId || delivery.principalId !== action.principalId || - !fence || - !(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, fence, [ + !currentFence || + !(await runtimeCallbackEndpoint(tx as unknown as DbTransaction, endpoint.id, currentFence, [ "verifying", "active", ])) @@ -160,6 +163,7 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions endpoint, action.principalId, ); + if (notice?.taskRetry && !githubTaskRetryDestinationMatches(delivery, conversation, resource)) throw deny(); if (endpoint.provider === "github") { const automatic = await resolveGitHubWakeAuthority(tx, endpoint, delivery); if (automatic.kind === "denied") throw deny(); @@ -214,9 +218,12 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions source.authorUserId !== expectedUserId ) throw deny(); - return { issue, delivery, endpoint, conversation }; + return { issue, delivery, endpoint, conversation, resource, + ...(notice?.taskRetry && currentFence ? { retryRuntimeFence: currentFence } : {}) }; } + const authorizeGitHubTaskRetryBatch = createGitHubTaskRetryBatchAuthority(authorizeInboundWakeup); + async function authorizeInboundWorkStart( tx: DbOrTransaction, action: typeof chatActions.$inferSelect, @@ -241,5 +248,5 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions attachments: normalized.message?.attachments?.map(a => `${a.name ?? "Attachment"} (${a.mimeType ?? "unknown type"})`) }; } - return { authorizeInboundWakeup, authorizeInboundWorkStart, authorizeFastResponse }; + return { authorizeInboundWakeup, authorizeGitHubTaskRetryBatch, authorizeInboundWorkStart, authorizeFastResponse }; } diff --git a/server/src/services/chat-channels/task-retry-input.ts b/server/src/services/chat-channels/task-retry-input.ts new file mode 100644 index 00000000000..b5fb8a30233 --- /dev/null +++ b/server/src/services/chat-channels/task-retry-input.ts @@ -0,0 +1,95 @@ +import { z } from "zod"; +import { and, asc, eq, inArray, isNull, sql } from "drizzle-orm"; +import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import type { DbOrTransaction } from "./runtime-types.js"; + +export const taskRetryInputSchema = z.object({ + // This is a server-selected snapshot of the durable queue, not client input. + // Truncating it strands older requests or makes the newer-input guard reject + // every Retry. Dispatch already carries the full queue with bounded inline text. + sourceActionIds: z.array(z.string().uuid()).refine(ids => new Set(ids).size === ids.length), + runtimeFence: z.object({ generation: z.number().int().nonnegative(), + credentialFingerprint: z.string().regex(/^[a-f0-9]{64}$/) }).optional(), +}); +export type TaskRetryInput = z.infer; + +export async function selectTaskRetryInput(tx: DbOrTransaction, input: { + companyId: string; issueId: string; agentId: string; failedRunId: string; +}) { + const [conversation] = await tx.select({ id: chatConversations.id }).from(chatConversations) + .innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)) + .innerJoin(chatActions, and(eq(chatActions.companyId, chatConversations.companyId), + eq(chatActions.conversationId, chatConversations.id))) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, chatConversations.companyId), + eq(heartbeatRuns.wakeupRequestId, chatActions.id))).where(and( + eq(chatConversations.companyId, input.companyId), eq(chatConversations.issueId, input.issueId), + eq(heartbeatRuns.id, input.failedRunId), eq(heartbeatRuns.agentId, input.agentId), + eq(chatEndpoints.provider, "github"), eq(chatEndpoints.assignedAgentId, input.agentId), + eq(chatEndpoints.externalExecutionPolicy, "restricted"), + )).limit(1); + if (!conversation) return undefined; + const pending = await tx.select({ id: chatActions.id }).from(chatActions) + .innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.companyId, chatActions.companyId), + eq(agentWakeupRequests.id, chatActions.id))).where(and( + eq(chatActions.companyId, input.companyId), eq(chatActions.conversationId, conversation.id), + eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"), + eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + isNull(agentWakeupRequests.runId), + )).orderBy(asc(chatActions.createdAt), asc(chatActions.id)).for("update", { of: agentWakeupRequests }); + return taskRetryInputSchema.parse({ sourceActionIds: pending.map(row => row.id) }); +} + +export async function claimTaskRetryInput(tx: DbOrTransaction, input: { + companyId: string; retryActionId: string; sourceActionIds: string[]; +}) { + if (!input.sourceActionIds.length) return; + await tx.update(agentWakeupRequests).set({ status: "coalesced", updatedAt: new Date(), + payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({ + coalescedIntoWakeupRequestId: input.retryActionId })}::jsonb`, + }).where(and(eq(agentWakeupRequests.companyId, input.companyId), + inArray(agentWakeupRequests.id, input.sourceActionIds), eq(agentWakeupRequests.status, "deferred_issue_execution"), + isNull(agentWakeupRequests.runId))); +} + +export async function settleTaskRetryInput(tx: DbOrTransaction, input: { companyId: string; retryActionId: string; runId: string | null }) { + if (!input.runId) return; + await tx.update(agentWakeupRequests).set({ runId: input.runId, updatedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.status, "coalesced"), + isNull(agentWakeupRequests.runId), + sql`${agentWakeupRequests.payload}->>'coalescedIntoWakeupRequestId' = ${input.retryActionId}`, + )); +} + +export async function releaseUnadmittedTaskRetryInput(db: Db, input: { companyId: string; retryActionId: string }) { + await db.transaction(async tx => { + const [snapshot] = await tx.select({ payload: chatActions.payload }).from(chatActions).where(and( + eq(chatActions.companyId, input.companyId), eq(chatActions.id, input.retryActionId), + )); + const issueId = z.string().uuid().safeParse(snapshot?.payload.issueId); + if (!issueId.success) return; + const [issue] = await tx.select({ id: issues.id }).from(issues).where(and( + eq(issues.companyId, input.companyId), eq(issues.id, issueId.data), + )).for("update"); + if (!issue) return; + const [action] = await tx.select().from(chatActions).where(and(eq(chatActions.companyId, input.companyId), + eq(chatActions.id, input.retryActionId), eq(chatActions.kind, "failed_run_retry"), + sql`${chatActions.payload}->>'issueId' = ${issue.id}`, + inArray(chatActions.status, ["failed", "cancelled"]))).for("update"); + if (!action || typeof action.payload.taskRetryIntentId !== "string") return; + const [receipt] = await tx.select({ runId: agentWakeupRequests.runId }).from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.id, input.retryActionId), + )).for("update"); + if (receipt?.runId) return; + await tx.update(agentWakeupRequests).set({ status: "deferred_issue_execution", finishedAt: null, + payload: sql`${agentWakeupRequests.payload} - 'coalescedIntoWakeupRequestId'`, updatedAt: new Date(), + }).where(and(eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.status, "coalesced"), + isNull(agentWakeupRequests.runId), + sql`${agentWakeupRequests.payload}->>'coalescedIntoWakeupRequestId' = ${input.retryActionId}`)); + await tx.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date(), updatedAt: new Date() }).where(and( + eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.id, action.payload.taskRetryIntentId), + eq(agentWakeupRequests.reason, "task_retry"), isNull(agentWakeupRequests.runId), + inArray(agentWakeupRequests.status, ["deferred_issue_execution", "queued", "claimed"]), + sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${action.id}`, + )); + }); +} diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index c4b3b9f2334..66f14532600 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -3,6 +3,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { executionFinalization: "execution-finalization", + workspaceRecovery: "workspace-recovery", executionReplacement: "execution-replacement", executionContinuation: "execution-continuation", executionStatus: "execution-status", diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index 62324a15ece..b9d9f721a28 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -1,6 +1,6 @@ import { createHash } from "node:crypto"; import { agentWakeupRequests, chatActions, chatDeliveries, chatEndpoints, type Db } from "@paperclipai/db"; -import { and, eq, or, sql, type SQLWrapper } from "drizzle-orm"; +import { and, eq, inArray, or, sql, type SQLWrapper } from "drizzle-orm"; import { HttpError } from "../errors.js"; export interface CommittedChatResponseAuthorizationInput { @@ -211,6 +211,7 @@ export interface DurableChatWakeupRequest { readonly idempotencyKey: string; readonly failedRunRetry?: { readonly failedRunId: string }; readonly authorize: (tx: Db) => Promise; + readonly onAdmitted?: (tx: Db, receipt: { runId: string | null }) => Promise; } const trustedRequests = new WeakSet(); @@ -290,6 +291,26 @@ export function assertDurableChatWakeupReceipt( } } +export function withDurableChatAdmission(request: DurableChatWakeupRequest | undefined, + admit: (tx: Parameters[0]>[0]) => Promise) { + return async (tx: Parameters[0]>[0]) => { + const outcome = await admit(tx); + // An existing deferred receipt is not proof this transaction passed its + // gates. Only dispatch or an explicitly persisted wait admits the input. + if (request?.onAdmitted && (outcome.admitted === true || ["queued", "durable", "coalesced"].includes(outcome.kind))) { + const [receipt] = await tx.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, request.companyId), eq(agentWakeupRequests.id, request.id), + inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution", "coalesced"]), + )); + if (receipt) { + assertDurableChatWakeupReceipt(request, receipt); + await request.onAdmitted(tx as unknown as Db, { runId: receipt.runId }); + } + } + return outcome; + }; +} + /** Work-start policy only: ongoing tool and publication authorization must not * revoke an already-started run when the operator enables mention gating. */ export async function slackMentionAllowsRunStart( diff --git a/server/src/services/execution-control-reconciliation.test.ts b/server/src/services/execution-control-reconciliation.test.ts index 9d46777715c..1b873eb4e1e 100644 --- a/server/src/services/execution-control-reconciliation.test.ts +++ b/server/src/services/execution-control-reconciliation.test.ts @@ -2,7 +2,7 @@ import { enableRunFailureReporting } from "../__tests__/helpers/run-failure-repo import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db"; +import { agents, companies, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issues, issueRecoveryActions, issueThreadInteractions, nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -153,6 +153,65 @@ describeEmbeddedPostgres("reconcileAbandonedExecutionControl reports a genuine f expect(result.surfaced).toBe(1); expect(mockCaptureRunFailure.mock.calls.slice(captureCallsBefore)).toHaveLength(0); }); + + it.each(["prepared", "finalized", "mismatched", "reassigned", "accepted"] as const)("preserves only an exact unfinished native workspace after its deadline (%s)", async state => { + const f = await seedAbandonedRunFixture(); + const [environment] = await db.insert(environments).values({ name: `Deadline ${f.runId}`, driver: "sandbox" }).returning(); + const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, + heartbeatRunId: f.runId, environmentId: environment.id, provider: "daytona", providerLeaseId: randomUUID(), + status: "active", leasePolicy: "reuse_by_environment", metadata: { remoteCwd: "/work" }, + }).returning(); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId, runnerProfileJson: { + nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: state === "finalized" ? state : "prepared", + descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, + workspaceId: f.runId, leaseId: lease.id, providerLeaseId: state === "mismatched" ? randomUUID() : lease.providerLeaseId, + remoteCwd: "/work", resourceDisposition: "keep_running" }, + } }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunFinalizations).values({ runId: f.runId, companyId: f.companyId, issueId: f.issueId, + phase: "observed", leaseOwner: "abandoned-controller", controlDeadlineAt: new Date(Date.now() - 60_000) }); + if (state === "reassigned") { + const nextAgentId = randomUUID(); + await db.insert(agents).values({ id: nextAgentId, companyId: f.companyId, name: "New owner", adapterType: "codex_local" }); + await db.update(issues).set({ assigneeAgentId: nextAgentId, executionRunId: null, checkoutRunId: null }) + .where(eq(issues.id, f.issueId)); + } + if (state === "accepted") { + const contractId = randomUUID(), resultId = randomUUID(); + await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId, + revision: 1, schemaVersion: "1", policyVersion: "test", risk: "low", completionAuthority: "agent_claim_policy", + incompleteCriteriaPolicy: "block", contractJson: {}, canonicalSha256: randomUUID(), createdByActorType: "agent", createdByActorId: f.agentId }); + await db.update(heartbeatRuns).set({ completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId)); + await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId, + completionContractId: contractId, serverFingerprint: randomUUID(), schemaStatus: "valid", resultJson: {}, canonicalSha256: randomUUID() }); + await db.update(nativeRunFinalizations).set({ resultId }).where(eq(nativeRunFinalizations.runId, f.runId)); + } + + await reconcileAbandonedExecutionControl(db); + await reconcileAbandonedExecutionControl(db); + + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); + const [retained] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)); + expect(run).toMatchObject(state === "accepted" ? { status: "running", errorCode: null } + : { status: "failed", errorCode: "execution_finalization_deadline_exceeded" }); + if (state === "prepared" || state === "reassigned") { + expect(run.resultJson).toMatchObject({ workspaceRestoreRecovery: { leaseIds: [lease.id] } }); + expect(retained).toMatchObject({ status: "pending_cleanup", leasePolicy: "retain_on_failure", + metadata: { sandboxStopAndRetain: { runId: f.runId, providerLeaseId: lease.providerLeaseId } } }); + const recovery = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); + expect(recovery.filter(action => action.evidence?.workspaceRestoreRecovery)).toMatchObject([ + { evidence: { runId: f.runId, workspaceRestoreRecovery: { leaseIds: [lease.id] } } }, + ]); + } else { + expect(run.resultJson ?? {}).not.toHaveProperty("workspaceRestoreRecovery"); + expect(retained).toMatchObject({ status: "active", leasePolicy: "reuse_by_environment" }); + } + const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + if (state === "reassigned") { + expect(task.assigneeAgentId).not.toBe(f.agentId); + expect(task).toMatchObject({ status: "in_progress", executionRunId: null, checkoutRunId: null }); + } + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }); }); describeEmbeddedPostgres("native review execution reconciliation admission", () => { diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 5b3a7aa3735..dd82e02b56a 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -14,6 +14,7 @@ import { issueRecoveryActionService } from "./issue-recovery-actions.js"; import { hasLiveLegacyController } from "./legacy-controller-lease.js"; import { reportRunFailure } from "./run-failure-report.js"; import { getNativeReviewAssignment, readNativeReviewAssignmentContext } from "./native-runtime/native-review-participant.js"; +import { preserveNativeFailedWorkspace } from "./native-runtime/native-failed-workspace-recovery.js"; /** Only newly recorded control deadlines are eligible. Upgrades never replay ambiguous historical runs. */ export async function reconcileAbandonedExecutionControl( @@ -233,8 +234,10 @@ export async function reconcileAbandonedExecutionControl( ["done", "cancelled"].includes(task.status) || (task.executionRunId && task.executionRunId !== run.id) || (task.checkoutRunId && task.checkoutRunId !== run.id) - ) + ) { + if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun); return true; + } await tx .update(issues) .set({ executionRunId: null, checkoutRunId: null, updatedAt: now }) @@ -261,6 +264,7 @@ export async function reconcileAbandonedExecutionControl( maxAttempts: 3, supersedeOnIdentityChange: true, }); + if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun); return true; }); if (repaired) surfaced += 1; diff --git a/server/src/services/execution-control-work.ts b/server/src/services/execution-control-work.ts index 1c7a1a6d376..69863954dbc 100644 --- a/server/src/services/execution-control-work.ts +++ b/server/src/services/execution-control-work.ts @@ -1,3 +1,4 @@ +import { nextWorkspaceRecoveryAt } from "./system-recovery.js"; import { and, asc, eq, isNotNull, sql } from "drizzle-orm"; import { adapterAuthSessions, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; @@ -12,6 +13,7 @@ export function registerExecutionControlWork( queue: DeliveryQueue; run: () => Promise; next: () => Promise; + onSettled?: () => void; }>, canRun: () => boolean = () => true, ) { @@ -20,10 +22,10 @@ export function registerExecutionControlWork( let reconciled = 0; let next: number | null = null; let earliestRunAt = 0; - return coordinator.register(duty.queue, { + const worker = coordinator.register(duty.queue, { retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS, pausePassiveMaintenance: true, - onSettled: () => { requested += 1; }, + onSettled: () => { requested += 1; duty.onSettled?.(); }, run: async () => { if (!canRun()) { requested += 1; next = null; return; } const now = Date.now(); @@ -41,8 +43,9 @@ export function registerExecutionControlWork( hasPending: async () => reconciled !== requested, nextRunAt: () => next, }); + return { ...worker, wake: () => { requested += 1; worker.wake(); } }; }); - return { ready: Promise.all(workers.map(worker => worker.ready)) }; + return { ready: Promise.all(workers.map(worker => worker.ready)), wake: () => { for (const worker of workers) worker.wake(); } }; } export function executionControlSchedule(db: Db) { @@ -63,6 +66,10 @@ export function executionControlSchedule(db: Db) { ]); return deadline([native[0]?.at, legacy[0]?.at]); }, + async workspaceRecovery() { + const at = await nextWorkspaceRecoveryAt(db); + return at === null ? null : deadline([new Date(at)]); + }, async replacement() { const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) diff --git a/server/src/services/heartbeat-runner-provider-config.test.ts b/server/src/services/heartbeat-runner-provider-config.test.ts index 5e874641c19..8090eb3cc04 100644 --- a/server/src/services/heartbeat-runner-provider-config.test.ts +++ b/server/src/services/heartbeat-runner-provider-config.test.ts @@ -7,6 +7,14 @@ import { resolvePaperclipRunnerNativeProviderInput, } from "./native-runtime/provider-profile.js"; +describe("native Codex hosted search policy", () => { + it.each([undefined, true, false])("preserves the saved search=%s choice", search => { + const input = resolvePaperclipRunnerNativeProviderInput({ backend: "codex_app_server", adapterConfig: { provider: "codex", model: "gpt-5.4-mini", ...(search === undefined ? {} : { search }) } }); + if (search === false) expect(input).toHaveProperty("codexWebSearch", "disabled"); + else expect(input).not.toHaveProperty("codexWebSearch"); + }); +}); + describe("Paperclip Runner native provider configuration", () => { it.each([ ["gpt-6.1-sol", "0.158.0", "gpt-6-sol"], diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 35ff18e6e88..2ac1a3f0960 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,5 +1,4 @@ -import { authorizationService } from "../authorization.js"; -import { assertTaskRetryConversationAllowed, readTaskRetryIntent, retireSupersededTaskRetry } from "../task-retry.js"; +import { assertTaskRetryConversationAllowed, readTaskRetryIntent, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js"; import { insertIssueComment } from "../issue-comment-writes.js"; import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js"; import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; @@ -50,6 +49,7 @@ import { slackMentionAllowsRunStart, assertDurableChatWakeupReceipt, assertDurableChatWakeupRequest, + withDurableChatAdmission, authorizeFailedChatRunRetryWake, FailedChatRunRetryAuthorizationError, unadmittedChatWakeupCondition, @@ -2809,7 +2809,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = []; const outcome = await db - .transaction(async (tx) => { + .transaction(withDurableChatAdmission(durableRequest, async (tx) => { await tx.execute( sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`, ); @@ -2847,11 +2847,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (pending.id !== opts.taskRetryRequestId || retry?.generationId !== pending.id || retry.chatActionId || retry.actorId !== opts.requestedByActorId || retry.sourceRunId !== enrichedContextSnapshot.previousRunId) return { kind: "deferred" as const }; - const permission = await authorizationService(tx as unknown as Db).decide({ - actor: { type: "board", userId: retry.actorId, source: retry.actorSource }, action: "issue:mutate", - resource: { type: "issue", companyId: agent.companyId, issueId }, - }); - if (!permission.allowed) return { kind: "deferred" as const }; + if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId, + actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const }; } if (opts.queuedCommentRequestId) { const ids = await undeliveredLegacyUserCommentIds(tx as unknown as Db, @@ -3004,11 +3001,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (!failedChatRetry || retry?.chatActionId !== durableRequest.id || retry.actorId !== taskRetryActorId || retry.sourceRunId !== durableRequest.failedRunRetry?.failedRunId) throw new FailedChatRunRetryAuthorizationError(); - const permission = await authorizationService(tx as unknown as Db).decide({ - actor: { type: "board", userId: retry.actorId, source: retry.actorSource }, action: "issue:mutate", - resource: { type: "issue", companyId: agent.companyId, issueId }, - }); - if (!permission.allowed) return { kind: "deferred" as const }; + if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: agent.companyId, issueId, + actorId: retry.actorId, actorSource: retry.actorSource })) return { kind: "deferred" as const }; } } else { if ( @@ -3311,7 +3305,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', ${JSON.stringify(condition)}::jsonb)`, updatedAt: new Date(), }).where(eq(agentWakeupRequests.id, executionWaitRequestId)); - return { kind: "deferred" as const }; + return { kind: "deferred" as const, admitted: true }; } if (durableRequest || wakeCommentId || hasInteractionContinuationWakeContext(enrichedContextSnapshot) || @@ -3348,7 +3342,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }, }); } - return { kind: "deferred" as const }; + return { kind: "deferred" as const, admitted: true }; }; const explicitContinuationRunId = randomUUID(); const executionBlocker = await getExecutionBlocker( @@ -3974,7 +3968,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }; } if (admission.kind === "deferred") { - return { kind: "deferred" as const }; + return { kind: "deferred" as const, admitted: true }; } // admission.kind === "proceed": no active run absorbed this wake, // so fall through to the ordinary queue path below. @@ -4357,7 +4351,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende // transitions to "running" — Fix A (lazy locking). return { kind: "queued" as const, run: newRun }; - }) + })) .catch((error) => { if (isExternalChatWaitAuthorizationContention(error)) return { kind: "deferred" as const }; diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 425a2fee60a..05a7b039a2f 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -971,6 +971,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele nativeToolContractFingerprint: nativeToolContractFingerprintForTarget( executionTarget?.kind ?? "local", + nativeExecution!.provider.kind, ), ...(lockedProfile.sessionCheckpoint != null ? { sessionCheckpoint: lockedProfile.sessionCheckpoint } diff --git a/server/src/services/native-runtime/assigned-mcp-tools.test.ts b/server/src/services/native-runtime/assigned-mcp-tools.test.ts index 7fa7b7ccb71..21cacfd0845 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.test.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { ToolGatewayHttpError, type ToolGatewayDescriptor, type ToolGatewayService } from "../tool-gateway.js"; import { createAssignedMcpTools, getAssignedMcpGateway, registerAssignedMcpGateway } from "./assigned-mcp-tools.js"; +import { startOpenCodeMcpBridge } from "../../vendor/paperclip-runner/testing.js"; function descriptor(name: string, risk: ToolGatewayDescriptor["risk"] = "read"): ToolGatewayDescriptor { return { name, displayName: name, description: `Use ${name}`, parametersSchema: { type: "object", properties: { query: { type: "string" } } }, pluginId: "fixture", providerType: "mcp_remote_http", risk }; @@ -18,6 +19,93 @@ describe("assigned MCP runner tools", () => { const searchName = "paperclip_search_assigned_tools"; const callName = "paperclip_call_assigned_tool"; + it.each(["page", "escaped schema"])("keeps on-demand %s results valid through the OpenCode MCP bridge", async variant => { + const schema = { type: "object", description: variant === "page" ? "x".repeat(20_000) : "\"\\\u0000🙂".repeat(30_000) }; + const f = fixture(Array.from({ length: variant === "page" ? 5 : 1 }, (_, i) => ({ + ...descriptor(`large_${i}`), parametersSchema: schema, + }))); + const assigned = await createAssignedMcpTools({ ...f, delivery: "on_demand" }); + const bridge = await startOpenCodeMcpBridge({ tools: assigned.definitions(), handler: call => assigned.execute(call) }); + let id = 0; + const search = async (args: Record) => { + const response = await fetch(bridge.url, { + method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", id: ++id, method: "tools/call", params: { name: searchName, arguments: args } }), + }); + const rpc = await response.json() as { result: { isError?: boolean; content: Array<{ text: string }> } }; + expect(rpc.result.isError).not.toBe(true); + return JSON.parse(rpc.result.content[0]!.text); + }; + try { + let page = await search({ query: "", limit: 20 }); + if (variant === "page") { + const seen = [...page.tools]; + while (page.nextOffset !== null) { + page = await search({ query: "", offset: page.nextOffset, limit: 20 }); + seen.push(...page.tools); + } + expect(seen).toHaveLength(5); + expect(seen.every(tool => JSON.stringify(tool.inputSchema) === JSON.stringify(schema))).toBe(true); + } else { + const name = page.tools[0]!.inputSchemaRef; + expect(name).toBeTypeOf("string"); + let offset: number | null = 0; + let reconstructed = ""; + do { + const chunk = await search({ query: "", schemaTool: name, schemaOffset: offset }); + reconstructed += chunk.schemaJson; + offset = chunk.nextSchemaOffset; + } while (offset !== null); + expect(JSON.parse(reconstructed)).toEqual(schema); + } + } finally { + await bridge.close(); + } + }); + + it.each([1, 741])("uses bounded on-demand delivery for %i tools even when the direct catalog fits", async count => { + const f = fixture(Array.from({ length: count }, (_, i) => descriptor(`app.operation_${i}`))); + const direct = await createAssignedMcpTools(f); + const assigned = await createAssignedMcpTools({ ...f, delivery: "on_demand" }); + const fits = vi.fn(() => true); + expect(assigned.definitions(fits).map(tool => tool.name)).toEqual([searchName, callName]); + expect(fits).toHaveBeenCalledExactlyOnceWith(assigned.definitions()); + expect(() => assigned.definitions(() => false)).toThrow("assigned_mcp_tool_catalog_capacity_exceeded"); + const page = await assigned.execute({ tool: searchName, arguments: { query: "", limit: 1 } }) as { tools: Array<{ name: string }> }; + expect(page.tools[0]!.name).toBe(direct.definitions().map(tool => String(tool.name)).sort()[0]); + await assigned.execute({ tool: callName, arguments: { name: page.tools[0]!.name, arguments: { query: "fixture" }, gatewayPublicId: "untrusted", bearerToken: "untrusted" } }); + expect(f.executeTool).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ gatewayPublicId: f.gatewayPublicId, sessionToken: f.bearerToken })); + f.listToolsForNamedGateway.mockResolvedValue([]); + await expect(assigned.execute({ tool: searchName, arguments: { query: "" } })).resolves.toEqual({ tools: [], nextOffset: null }); + }); + + it("keeps on-demand aliases pinned, mode restricted, revocable and approval gated", async () => { + const read = descriptor("mcp.first:search"); + const write = descriptor("mcp.second:search", "write"); + const f = fixture([read, write]); + const assigned = await createAssignedMcpTools({ ...f, delivery: "on_demand" }); + const page = await assigned.execute({ tool: searchName, arguments: { query: "" } }) as { tools: Array<{ name: string; description: string }> }; + expect(new Set(page.tools.map(tool => tool.name)).size).toBe(2); + const writeAlias = page.tools.find(tool => tool.description.includes(write.name))!.name; + const call = { tool: callName, arguments: { name: writeAlias, arguments: {} } }; + await expect(assigned.execute(call, "planning")).rejects.toThrow("paperclip_runner_tool_mode_denied"); + await expect(assigned.execute(call, "ask")).rejects.toThrow("paperclip_runner_tool_mode_denied"); + expect(f.executeTool).not.toHaveBeenCalled(); + const denied = new ToolGatewayHttpError(403, "Approval required", "approval_required"); + f.executeTool.mockRejectedValueOnce(denied); + await expect(assigned.execute(call)).rejects.toBe(denied); + f.listToolsForNamedGateway.mockResolvedValue([read, descriptor("newly.granted")]); + const refreshed = await assigned.execute({ tool: searchName, arguments: { query: "" } }) as { tools: Array<{ name: string }> }; + expect(refreshed.tools).toHaveLength(1); + expect(refreshed.tools[0]!.name).not.toBe(writeAlias); + // Execution still goes through the original gateway, which reauthorizes + // each call; discovery never grants an added tool to this pinned session. + const revoked = new ToolGatewayHttpError(403, "Revoked", "connection_revoked"); + f.executeTool.mockRejectedValueOnce(revoked); + await expect(assigned.execute(call)).rejects.toBe(revoked); + await expect(assigned.execute({ tool: callName, arguments: { name: "newly.granted", arguments: {} } })).rejects.toThrow("assigned_mcp_tool_unknown"); + }); + it("deduplicates repeated app tools in direct and on-demand catalogs while retaining revocation", async () => { const read = descriptor("memory.search"); const write = descriptor("memory.store", "write"); diff --git a/server/src/services/native-runtime/assigned-mcp-tools.ts b/server/src/services/native-runtime/assigned-mcp-tools.ts index 535b265d4ad..ff43b86a6dd 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.ts @@ -86,6 +86,8 @@ export async function createAssignedMcpTools(input: { gatewayPublicId: string; bearerToken: string; workMode?: WorkMode; + /** Bound provider grammar complexity independently of protocol byte capacity. */ + delivery?: "on_demand"; }) { const listed = await input.gateway.listToolsForNamedGateway({ gatewayPublicId: input.gatewayPublicId, @@ -103,6 +105,11 @@ export async function createAssignedMcpTools(input: { tools.set(name, descriptor); } const permits = (tool: ToolGatewayDescriptor, mode: WorkMode = input.workMode ?? "standard") => mode === "standard" || tool.risk === "read"; + // OpenCode's HTTP bridge returns at most 64K JSON characters. Leave room + // for the envelope and for JSON escaping when returning a schema chunk. + // Other harnesses retain the larger PRP page and chunk budgets. + const searchPageBytes = input.delivery === "on_demand" ? 32 * 1024 : SEARCH_PAGE_BYTES; + const schemaChunkCharacters = input.delivery === "on_demand" ? 16 * 1024 : SCHEMA_CHUNK_CHARACTERS; async function search(argumentsValue: unknown, currentWorkMode?: WorkMode) { const args = object(argumentsValue); @@ -132,7 +139,7 @@ export async function createAssignedMcpTools(input: { const selected = authorized.find(([name]) => name === args.schemaTool); if (!selected) throw new Error("assigned_mcp_tool_unknown"); const schema = JSON.stringify(selected[1].parametersSchema); - const end = schemaOffset + SCHEMA_CHUNK_CHARACTERS; + const end = schemaOffset + schemaChunkCharacters; return { name: selected[0], schemaJson: schema.slice(schemaOffset, end), nextSchemaOffset: end < schema.length ? end : null, @@ -144,13 +151,13 @@ export async function createAssignedMcpTools(input: { const page: ToolDefinition[] = []; for (const [name, tool] of matches.slice(offset, offset + limit)) { let next = definition(name, tool); - if (Buffer.byteLength(JSON.stringify([next]), "utf8") > SEARCH_PAGE_BYTES) { + if (Buffer.byteLength(JSON.stringify([next]), "utf8") > searchPageBytes) { // One large schema must not block browsing the tools after it. Expose // a reference whose schema can be fetched in bounded, reauthorized // chunks, without weakening the gateway's argument validation. next = { name, description: String(next.description).slice(0, 2000), inputSchemaRef: name }; } - if (Buffer.byteLength(JSON.stringify([...page, next]), "utf8") > SEARCH_PAGE_BYTES) { + if (Buffer.byteLength(JSON.stringify([...page, next]), "utf8") > searchPageBytes) { break; } page.push(next); @@ -161,9 +168,9 @@ export async function createAssignedMcpTools(input: { return { definitions(fits?: (definitions: ToolDefinition[]) => boolean): ToolDefinition[] { const direct = [...tools].filter(([, tool]) => permits(tool)).map(([name, tool]) => definition(name, tool)); - if (!fits || fits(direct)) return direct; + if (input.delivery !== "on_demand" && (!fits || fits(direct))) return direct; const compact = structuredClone(ON_DEMAND_TOOLS); - if (!fits(compact)) throw new Error("assigned_mcp_tool_catalog_capacity_exceeded"); + if (fits && !fits(compact)) throw new Error("assigned_mcp_tool_catalog_capacity_exceeded"); return compact; }, has(name: string): boolean { diff --git a/server/src/services/native-runtime/native-execution-input.test.ts b/server/src/services/native-runtime/native-execution-input.test.ts index f0cdc8aca17..f10000ce8c7 100644 --- a/server/src/services/native-runtime/native-execution-input.test.ts +++ b/server/src/services/native-runtime/native-execution-input.test.ts @@ -21,12 +21,13 @@ describe("LCA-05 explicit native work mode", () => { normalizedSessionId: null, provider: "codex", codexReasoningEffort: "high", + codexWebSearch: "disabled", planningContext: workMode === "planning" ? { documentId: null, baseRevisionId: null, baseRevisionNumber: 0, markdown: "", sha256: "a".repeat(64), reviewContext: {} } : null, completionContract: { id: "70000000-0000-4000-8000-000000000007", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", contract: { revision: "1", objective: "Deliver the requested work", criteria: [{ id: "output", requirement: "Deliver the requested work" }] } }, runtimeContext: nativeRuntimeContextFixture(), }); expect(input.task.workMode).toBe(workMode); - expect(input.provider).toMatchObject({ kind: "codex", reasoningEffort: "high" }); + expect(input.provider).toMatchObject({ kind: "codex", reasoningEffort: "high", webSearch: "disabled" }); expect(input.executionMode).toBe(workMode === "planning" ? "plan" : "default"); expect(input.task.title).toBe(text); } diff --git a/server/src/services/native-runtime/native-execution-input.ts b/server/src/services/native-runtime/native-execution-input.ts index 313e2f7405a..974ea876de6 100644 --- a/server/src/services/native-runtime/native-execution-input.ts +++ b/server/src/services/native-runtime/native-execution-input.ts @@ -102,6 +102,7 @@ export interface BuildNativeExecutionInput { homeMode?: "isolated" | "preconfigured"; codexApprovalPolicy?: NativeCodexApprovalPolicy; codexReasoningEffort?: string; + codexWebSearch?: "disabled"; opencodePermissionMode?: NativeOpenCodePermissionMode; acpxPermissionMode?: NativeAcpxPermissionMode; acpxSessionMode?: "agent" | "plan" | "ask"; @@ -364,6 +365,7 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat model: input.model ?? null, approvalPolicy: input.codexApprovalPolicy ?? "never", ...(input.codexReasoningEffort ? { reasoningEffort: input.codexReasoningEffort } : {}), + ...(input.codexWebSearch ? { webSearch: input.codexWebSearch } : {}), }, completionContract: { id: input.completionContract.id, diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 39862333f8c..c0d18a93957 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1471,6 +1471,7 @@ describe("remote provider pack manifest", () => { try { await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, runnerInstanceId: "bundled-image-runner", runnerIngressAuthorized: true, + ...(provider === "opencode" ? { runnerEnvironment: { PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute, syncIn } } as never }); @@ -1478,8 +1479,13 @@ describe("remote provider pack manifest", () => { state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const transport = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { controlPlaneRegistration: (authority: unknown) => Promise; + environment?: NodeJS.ProcessEnv; }; expect(transport.runnerBinary).toBe(state.bundledRunnerBinary); + if (provider === "opencode") { + expect(state.createAssignedMcpTools).toHaveBeenCalledWith(expect.objectContaining({ gatewayPublicId: "assigned-fixture", bearerToken: "private-fixture", delivery: "on_demand" })); + for (const name of ["PAPERCLIP_NATIVE_MCP_NAME", "PAPERCLIP_NATIVE_MCP_URL", "PAPERCLIP_NATIVE_MCP_TOKEN"]) expect(transport.environment?.[name]).toBeUndefined(); + } if (claude) { expect(state.resolveRemoteRunnerBinary).toHaveBeenCalledWith(expect.objectContaining({ remoteCwd: "/workspace", configuredPath: undefined })); const { bundledRemoteProviderPackManifestPath } = await import("../../vendor/paperclip-runner/index.js"); @@ -7800,7 +7806,7 @@ describe("bounded stopped instruction collection", () => { describe("native warm session supervision", () => { it.each(["collect", "close-failure", "stopped", "destroyed", "missing-lease", "foreign-run", "foreign-environment", "deleted-environment", "claim-edit", "claim-add", "claim-remove", "claim-remove-entry", "claim-canonical", "claim-config", "claim-uncertain"])("composes remote warm ownership with real DB successor leases and current-run collection: %s", async ending => { const tables = await import("@paperclipai/db"); - const { eq } = await import("drizzle-orm"); + const { eq, sql } = await import("drizzle-orm"); const { randomUUID } = await import("node:crypto"); const { promisify } = await import("node:util"); const { execFile } = await import("node:child_process"); @@ -7813,7 +7819,11 @@ describe("native warm session supervision", () => { const home = await realpath(await mkdtemp(join(tmpdir(), "warm-remote-db-"))); const previousHome = process.env.PAPERCLIP_HOME; process.env.PAPERCLIP_HOME = home; - const database = await startEmbeddedPostgresTestDatabase("warm-remote-db-"); + const { resolveExternalTestDatabaseUrl } = await import("../../__tests__/helpers/external-test-database.js"); + const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); + const database = externalUrl + ? { connectionString: externalUrl, cleanup: () => tables.closeRegisteredClients(externalUrl) } + : await startEmbeddedPostgresTestDatabase("warm-remote-db-"); const db = tables.createDb(database.connectionString); const companyId = randomUUID(), agentId = randomUUID(), userId = randomUUID(), environmentId = randomUUID(), issueId = randomUUID(); const { resolveManagedInstructionsRoot } = await import("../agent-instructions.js"); @@ -8049,6 +8059,9 @@ describe("native warm session supervision", () => { finally { const cleanupFailures: unknown[] = []; try { await closeWarmNativeSessionsForEnvironment({ environmentId, reason: "test cleanup" }); } catch (error) { cleanupFailures.push(error); } + if (externalUrl) { + try { await db.execute(sql`TRUNCATE TABLE ${tables.companies}, ${tables.environments}, ${tables.authUsers} CASCADE`); } catch (error) { cleanupFailures.push(error); } + } try { await database.cleanup(); } catch (error) { cleanupFailures.push(error); } if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; try { @@ -11828,7 +11841,10 @@ describe("runnerd provider runtime wiring", () => { } finally {owner.mockRestore();await rm(home,{recursive:true,force:true});} }); - it("keeps assigned MCP credentials on the control plane for remote Codex", async () => { + it.each(["codex", "opencode"] as const)("keeps assigned MCP credentials on the control plane for %s", async (providerKind) => { + const selectedExecution: NativeExecutionInput = providerKind === "opencode" + ? { ...execution, provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", permissionMode: "allow" } } + : execution; const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); const exporter = new InMemorySpanExporter(); @@ -11841,13 +11857,13 @@ describe("runnerd provider runtime wiring", () => { state.createTransport.mockClear(); state.toolAuthorityDefinitions.mockClear(); await createRunnerdBackend({ - db: leaseDb(execution), execution, runnerInstanceId: "runner-assigned-mcp", + db: leaseDb(selectedExecution), execution: selectedExecution, runnerInstanceId: "runner-assigned-mcp", runnerEnvironment: { PAPERCLIP_NATIVE_MCP_NAME: "paperclip-assigned", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1:3217/mcp/gateways/assigned-test", PAPERCLIP_NATIVE_MCP_TOKEN: "private-run-token", }, - runnerExecutionTarget: { + runnerExecutionTarget: providerKind === "opencode" ? undefined : { kind: "remote", transport: "sandbox", providerKey: "daytona", leaseId: "lease-assigned-mcp", remoteCwd: "/home/daytona/paperclip-workspace", runner: { execute: vi.fn() }, @@ -11855,6 +11871,7 @@ describe("runnerd provider runtime wiring", () => { }); expect(state.createAssignedMcpTools).toHaveBeenCalledWith(expect.objectContaining({ gatewayPublicId: "assigned-test", bearerToken: "private-run-token", + ...(providerKind === "opencode" ? { delivery: "on_demand" } : {}), })); expect(state.toolAuthorityDefinitions).toHaveBeenCalledWith(expect.objectContaining({ assignedMcpTools })); state.createBackend.mock.calls[0]![1].codexTransportFactory!(); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 234f673e664..7bea16d6e0d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5563,7 +5563,7 @@ function nativeSessionConfigDigest( // changes so an older thread is rotated instead of falsely resuming // without newly required tools. nativeToolContractFingerprint: - nativeToolContractFingerprintForTarget(executionTargetKind), + nativeToolContractFingerprintForTarget(executionTargetKind, execution.provider.kind), runtimeContract: nativeRuntimeContractForProvider(execution.provider), }), ) @@ -11828,10 +11828,10 @@ async function createRunnerdBackendWithinSessionClaim( if (nativeReview && !await getNativeReviewAssignment(input.db, { ...input.execution.binding, contextSnapshot: nativeReview, })) throw new Error("native_review_assignment_no_longer_available"); - // Remote Codex already sends dynamic tool calls over authenticated PRP. Keep + // Codex and OpenCode send dynamic tool calls over authenticated PRP. Keep // the assigned gateway on the control plane instead of asking the sandbox to // reach the host's HTTP origin (which may be private or loopback-only). - const relayAssignedMcp = Boolean(pluginTask) || (remoteTarget !== null && input.execution.provider.kind === "codex") || input.execution.schema === "paperclip.native-execution-input.v6"; + const relayAssignedMcp = Boolean(pluginTask) || input.execution.provider.kind === "opencode" || (remoteTarget !== null && input.execution.provider.kind === "codex") || input.execution.schema === "paperclip.native-execution-input.v6"; const assignedMcpUrl = input.runnerEnvironment?.PAPERCLIP_NATIVE_MCP_URL; const assignedMcpToken = input.runnerEnvironment?.PAPERCLIP_NATIVE_MCP_TOKEN; const assignedMcpName = input.runnerEnvironment?.PAPERCLIP_NATIVE_MCP_NAME; @@ -11853,6 +11853,7 @@ async function createRunnerdBackendWithinSessionClaim( gatewayPublicId: assignedGatewayPublicId!, bearerToken: assignedMcpToken, workMode: input.execution.task.workMode, + ...(input.execution.provider.kind === "opencode" ? { delivery: "on_demand" as const } : {}), })) : undefined), ]); @@ -12012,6 +12013,7 @@ async function createRunnerdBackendWithinSessionClaim( }, codexTransportFactory: recovery => createRunnerdCodexTransport({ provider: provider.kind, stateDirectory: root, ...owner.driver, runnerArtifact: description.runner, + ...(provider.kind === "codex" ? { disableCodexWebSearch: "webSearch" in provider && provider.webSearch === "disabled" } : {}), ...(provider.kind === "acpx" ? { acpxAgent: provider.agent, acpxCandidateProfile: provider.agent === "pi" ? undefined : resolveAcpxQualification(provider, process.env), acpxPermissionMode: provider.permissionMode, acpxMode: provider.mode, piThinkingLevel: provider.piThinkingLevel, @@ -13396,6 +13398,7 @@ async function createRunnerdBackendWithinSessionClaim( codexTransportFactory: (recoveryContext) => createRunnerdCodexTransport({ onSpawn: input.onSpawn, + ...(input.execution.provider.kind === "codex" ? { disableCodexWebSearch: "webSearch" in input.execution.provider && input.execution.provider.webSearch === "disabled" } : {}), homeMode: preconfiguredHome ? "preconfigured" : "isolated", onCredentialsUpdated: (event) => appendNativeCredentialUpdate(input.db, { companyId: input.execution.binding.companyId, diff --git a/server/src/services/native-runtime/native-session-resume.test.ts b/server/src/services/native-runtime/native-session-resume.test.ts index bacf0ab22e9..c0f274b08e4 100644 --- a/server/src/services/native-runtime/native-session-resume.test.ts +++ b/server/src/services/native-runtime/native-session-resume.test.ts @@ -32,7 +32,7 @@ import { } from "@paperclipai/paperclip-runner"; import { getEmbeddedPostgresTestSupport, - startEmbeddedPostgresTestDatabase, + startEmbeddedPostgresTestDatabase as startEmbeddedDatabase, } from "../../__tests__/helpers/embedded-postgres.js"; import { canonicalNativeRuntimeContextDigest, @@ -72,6 +72,20 @@ import { selectNativeSessionResumeRun, } from "./native-session-resume.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; + +const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); + +async function startEmbeddedPostgresTestDatabase(prefix: string) { + const externalUrl = externalTestDatabaseUrl; + if (!externalUrl) return startEmbeddedDatabase(prefix); + const { closeRegisteredClients } = await import("@paperclipai/db"); + return { connectionString: externalUrl, cleanup: async () => { + const db = createDb(externalUrl); + try { await db.execute(sql`TRUNCATE TABLE companies CASCADE`); } + finally { await closeRegisteredClients(externalUrl); } + } }; +} const companyId = "10000000-0000-4000-8000-000000000001"; const issueId = "20000000-0000-4000-8000-000000000002"; @@ -283,12 +297,13 @@ it("wires exact-session recovery and guarded selected identity into heartbeat pe }); const embeddedSupport = await getEmbeddedPostgresTestSupport(); +const databaseSupported = Boolean(externalTestDatabaseUrl) || embeddedSupport.supported; const recoveryFakeCodex = resolve( import.meta.dirname, "../../../../packages/paperclip-runner/test/fixtures/fake-final-burst-codex-app-server.mjs", ); -(embeddedSupport.supported && existsSync(defaultCapabilityRunnerdBinary()) +(databaseSupported && existsSync(defaultCapabilityRunnerdBinary()) ? it : it.skip)( "keeps actual normalized driver events separate from raw maintenance receipts", @@ -520,7 +535,7 @@ const recoveryFakeCodex = resolve( 30_000, ); -(embeddedSupport.supported && +(databaseSupported && existsSync(defaultCapabilityRunnerdBinary()) && existsSync(recoveryFakeCodex) ? it @@ -1062,7 +1077,7 @@ const recoveryFakeCodex = resolve( 60_000, ); -(embeddedSupport.supported ? describe : describe.skip)( +(databaseSupported ? describe : describe.skip)( "native session recovery database orchestration", () => { it("uses scoped latest checkpoints, preserves progress barriers, and guards bootstrap persistence after the row lock", async () => { @@ -1825,6 +1840,23 @@ describe("rebindNativeSessionCheckpoint", () => { + it.each(["local", "remote"] as const)("refreshes only OpenCode's former direct MCP catalog on %s", target => { + const legacy = nativeToolContractFingerprintForTarget(target); + expect(nativeToolContractFingerprintForTarget(target, "codex")).toBe(legacy); + expect(nativeToolContractFingerprintForTarget(target, "acpx")).toBe(legacy); + const current = nativeToolContractFingerprintForTarget(target, "opencode"); + expect(current).not.toBe(legacy); + const shortenedCatalog = target === "local" + ? "sha256:2ae35faccbb63cfe009d94170ef1ef13f0d57e120af6160540598a3ec72001d7" + : "sha256:ee5285ca5118b8468c0afd1df4ef9c05d85b0934e882bccd0b193dc8722dbb7a"; + expect(current).not.toBe(shortenedCatalog); + const currentExecution = { ...execution(currentRunId), provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4.1-flash-1017", permissionMode: "allow" }, session: { ...execution(currentRunId).session, driverKind: "opencode_server" } } as NativeExecutionInput; + const previousExecution = { ...execution(previousRunId), provider: currentExecution.provider, session: currentExecution.session }; + expect(rebindNativeSessionCheckpoint({ previousRun: previousRun({ nativeToolContractFingerprint: shortenedCatalog, nativeExecutionInput: previousExecution }), currentExecution, executionTargetKind: target })).toBeNull(); + expect(rebindNativeSessionCheckpoint({ previousRun: previousRun({ nativeToolContractFingerprint: legacy, nativeExecutionInput: previousExecution }), currentExecution, executionTargetKind: target })).toBeNull(); + expect(rebindNativeSessionCheckpoint({ previousRun: previousRun({ nativeToolContractFingerprint: current, nativeExecutionInput: previousExecution }), currentExecution, executionTargetKind: target })).not.toBeNull(); + }); + it("binds the provider tool catalog to the local or remote execution target", () => { const remoteFingerprint = nativeToolContractFingerprintForTarget("remote"); expect(remoteFingerprint).not.toBe(NATIVE_TOOL_CONTRACT_FINGERPRINT); diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index 3243de9980e..e3ea827cbdf 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -29,6 +29,7 @@ export function nativeRuntimeContractForProvider( */ export function nativeToolContractFingerprintForTarget( executionTargetKind: NativeToolExecutionTargetKind, + providerKind?: NativeExecutionInput["provider"]["kind"], ): string { return `sha256:${createHash("sha256") .update( @@ -47,6 +48,9 @@ export function nativeToolContractFingerprintForTarget( structuredHumanInput: "always_advertised_run_issue_agent_binding_gated_current_task_description.v2", semanticCompletion: "finish_accessible_deliverable_evidence.v4", + ...(providerKind === "opencode" + ? { opencodeAssignedMcpTools: "server_gateway_prp_on_demand_exact_names.v2" } + : {}), connectorTools: "assigned_resources_and_pinned_skill_bundle.v1", chatTaskTools: "durable_task_binding_and_accepted_origin.v1", githubResponses: "bound_working_comment_progress_and_final.v1", @@ -482,7 +486,7 @@ export function rebindNativeSessionCheckpoint(input: { const previousProfile = record(input.previousRun.runnerProfileJson); if ( previousProfile.nativeToolContractFingerprint !== - nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local") + nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local", input.currentExecution.provider.kind) ) { return null; } diff --git a/server/src/services/native-runtime/provider-profile.ts b/server/src/services/native-runtime/provider-profile.ts index 22f906c59c3..d217c3a05ca 100644 --- a/server/src/services/native-runtime/provider-profile.ts +++ b/server/src/services/native-runtime/provider-profile.ts @@ -84,6 +84,7 @@ export type PaperclipRunnerNativeProviderInput = homeMode?: "isolated" | "preconfigured"; codexApprovalPolicy: "never" | "on-request" | "untrusted"; codexReasoningEffort?: string; + codexWebSearch?: "disabled"; } | { provider: "opencode"; @@ -843,5 +844,6 @@ export function resolvePaperclipRunnerNativeProviderInput(input: { config.codexPermissionMode, ) as "never" | "on-request" | "untrusted", ...(typeof effort === "string" && effort ? { codexReasoningEffort: effort } : {}), + ...(config.search === false ? { codexWebSearch: "disabled" as const } : {}), }; } diff --git a/server/src/services/system-recovery.ts b/server/src/services/system-recovery.ts index 75eea95666f..8dba366e1af 100644 --- a/server/src/services/system-recovery.ts +++ b/server/src/services/system-recovery.ts @@ -1,10 +1,17 @@ import { and, eq, sql } from "drizzle-orm"; -import { issueComments, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { heartbeatRuns, issueComments, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { writeExecutionControl } from "./execution-control-notifications.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { createSystemRecovery, readSystemRecovery, type SystemRecoveryState } from "./system-recovery-state.js"; -import type { WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js"; +import { workspaceRecoverySourceEligible, type WorkspaceRecoverySourceIdentity } from "./workspace-recovery-source.js"; export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) { return db.transaction(async tx => { + const [existing] = await tx.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions) + .where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId), + eq(issueRecoveryActions.sourceIssueId, source.issueId))).for("update"); + if (!existing || existing.evidence.systemRecovery) return; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery); const state = createSystemRecovery(source.runId); // Atomic, once per source. An exhausted previous execution budget is history; // adoption grants only the explicitly versioned maintenance capability. @@ -49,17 +56,22 @@ export async function updateSystemRecovery(db: Db, source: WorkspaceRecoverySour stage: next.phase === "verify_execution" ? "verifying_stop" : next.phase === "preserve_files" ? "restoring_files" : "saving_receipts", attempt: next.attempts, attemptLimit: 6, retryable: true, retryAt: next.nextAttemptAt ?? undefined, }; - const [updated] = await db.update(issueRecoveryActions).set({ - ownerType: "system", ownerAgentId: null, ownerUserId: null, - evidence: sql`${issueRecoveryActions.evidence} || jsonb_build_object('systemRecovery', ${JSON.stringify(next)}::jsonb, - 'workspaceRecovery', ${JSON.stringify(compatibility)}::jsonb || jsonb_build_object('requestIds', - coalesce(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}', '[]'::jsonb)))`, - updatedAt: new Date(), - }).where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId), - sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${previous.generationId}`, - sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(previous.revision)}`)) - .returning({ id: issueRecoveryActions.id }); - return Boolean(updated); + return writeExecutionControl(db, DELIVERY_QUEUES.workspaceRecovery, async tx => { + const [updated] = await tx.update(issueRecoveryActions).set({ + ownerType: "system", ownerAgentId: null, ownerUserId: null, + evidence: sql`${issueRecoveryActions.evidence} || jsonb_build_object('systemRecovery', ${JSON.stringify(next)}::jsonb || + ${next.generationId === previous.generationId && next.intentId === previous.intentId + ? sql`jsonb_build_object('intentId', ${issueRecoveryActions.evidence}#>'{systemRecovery,intentId}')` + : sql`'{}'::jsonb`}, + 'workspaceRecovery', ${JSON.stringify(compatibility)}::jsonb || jsonb_build_object('requestIds', + coalesce(${issueRecoveryActions.evidence}#>'{workspaceRecovery,requestIds}', '[]'::jsonb)))`, + updatedAt: new Date(), + }).where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId), + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,generationId}' = ${previous.generationId}`, + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,revision}' = ${String(previous.revision)}`)) + .returning({ id: issueRecoveryActions.id }); + return Boolean(updated); + }); } export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySourceIdentity) { @@ -68,3 +80,19 @@ export async function loadSystemRecovery(db: Db, source: WorkspaceRecoverySource eq(issueRecoveryActions.id, source.actionId))); return readSystemRecovery(row?.evidence); } + +export async function nextWorkspaceRecoveryAt(db: Db): Promise { + const rows = await db.select({ at: sql`case + when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}') = 'number' + then (${issueRecoveryActions.evidence}#>>'{systemRecovery,nextAttemptAt}')::bigint else 0 end` }) + .from(issueRecoveryActions).innerJoin(heartbeatRuns, and( + eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), + eq(heartbeatRuns.id, sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + then (${issueRecoveryActions.evidence}->>'runId')::uuid end`), + sql`${heartbeatRuns.status} in ('failed', 'interrupted', 'timed_out', 'cancelled', 'succeeded')`, + )).where(and( + workspaceRecoverySourceEligible(), + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')`, + )).orderBy(sql`1`).limit(1); + return rows.length ? Number(rows[0].at) : null; +} diff --git a/server/src/services/task-retry.ts b/server/src/services/task-retry.ts index 1937712ac23..afbfbb34d08 100644 --- a/server/src/services/task-retry.ts +++ b/server/src/services/task-retry.ts @@ -1,3 +1,5 @@ +import { writeExecutionControl } from "./execution-control-notifications.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { z } from "zod"; import { randomUUID } from "node:crypto"; import { and, desc, eq, inArray, or, sql } from "drizzle-orm"; @@ -8,6 +10,20 @@ import { createSystemRecovery, readSystemRecovery } from "./system-recovery-stat import { updateSystemRecovery } from "./system-recovery.js"; import { instanceSettingsService } from "./instance-settings.js"; import { notifyChatActionWork } from "./chat-work-notifications.js"; +import { authorizationService } from "./authorization.js"; + +export async function taskRetryMutationAllowed(db: Db, input: { + companyId: string; issueId: string; actorId: string; actorSource: TaskRetryIntent["actorSource"]; +}) { + const [issue] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId, + assigneeUserId: issues.assigneeUserId }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))); + if (!issue) return false; + const permission = await authorizationService(db).decide({ + actor: { type: "board", userId: input.actorId, source: input.actorSource }, action: "issue:mutate", + resource: { type: "issue", companyId: input.companyId, issueId: input.issueId, ...issue }, + }); + return permission.allowed; +} export async function assertTaskRetryConversationAllowed( db: Db, @@ -59,7 +75,7 @@ export async function requestTaskRetry(input: { prepareChatRetry?: (tx: Db, scope: { companyId: string; issueId: string; agentId: string; failedRunId: string; initiatedByUserId: string; retryIntentId?: string }) => Promise<{ actionId: string }>; }) { - return input.db.transaction(async tx => { + return writeExecutionControl(input.db, DELIVERY_QUEUES.workspaceRecovery, async tx => { const db = tx as unknown as Db; const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); @@ -115,6 +131,7 @@ export async function requestTaskRetry(input: { const intent: TaskRetryIntent = { schema: "paperclip.task-retry.v1", generationId: active?.id ?? id, sourceRunId: latest.id, actorId: input.actorId, actorSource: taskRetryIntentSchema.shape.actorSource.parse(input.actorSource ?? "session"), strategy: "evaluate", + ...(active && activeIntent?.chatActionId ? { chatActionId: activeIntent.chatActionId } : {}), ...(active ? { canonicalIntentId: active.id } : {}) }; // Preserve external chat authority through its existing admission owner. // Generic board Retry must never manufacture a chat-inbound request. @@ -139,6 +156,11 @@ export async function requestTaskRetry(input: { for (const action of records) { const previous = readSystemRecovery(action.evidence); if (!previous || previous.state === "complete") continue; + if (previous.state === "running") { + await updateSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id, runId: latest.id, + actionId: action.id }, previous, { ...previous, intentId: active?.id ?? id }); + continue; + } // Active repair keeps its attempt and cooldown. An exhausted generation // gets a new budget only because this click is a new durable intent. const next = previous.state === "exhausted" @@ -148,8 +170,7 @@ export async function requestTaskRetry(input: { : { ...previous, revision: previous.revision + 1, intentId: active?.id ?? id, // Retry may advance our backoff, never a provider's cooldown and // never an operation whose physical owner is still running. - nextAttemptAt: previous.state === "running" ? previous.nextAttemptAt - : Math.max(Date.now(), previous.providerCooldownAt ?? 0) }; + nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) }; if (next.generationId !== previous.generationId) await persistActivity(db, { companyId: issue.companyId, actorType: "system", actorId: "task-retry", entityType: "issue", entityId: issue.id, action: "issue.recovery_generation_replaced", runId: latest.id, diff --git a/server/src/services/workspace-recovery-source.ts b/server/src/services/workspace-recovery-source.ts index ab8517aa88c..61b9320b7d7 100644 --- a/server/src/services/workspace-recovery-source.ts +++ b/server/src/services/workspace-recovery-source.ts @@ -10,6 +10,29 @@ export const workspaceExportCauses = ["native_workspace_sync_out_unrecoverable", export const terminalWorkspaceRunStates = ["failed", "interrupted", "timed_out", "cancelled", "succeeded"]; export type WorkspaceRecoverySourceIdentity = { companyId: string; issueId: string; runId: string; actionId: string }; +export function workspaceRecoverySourceEligible(includeCompleted = false) { + const actionEligible = sql`(${issueRecoveryActions.status} in ('active', 'escalated') or + (${issueRecoveryActions.status} = 'resolved' and ( + (${issueRecoveryActions.evidence}#>>'{automaticRecovery,policy}' = 'preserve_without_replay_v1' + and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,replay}' = 'blocked' + and ${issueRecoveryActions.evidence}#>>'{automaticRecovery,runId}' = ${heartbeatRuns.id}::text) or + (${issueRecoveryActions.evidence}#>>'{historicalWorkspaceSource,schema}' = 'paperclip.historical-workspace-source.v1' + and ${issueRecoveryActions.evidence}#>>'{historicalWorkspaceSource,runId}' = ${heartbeatRuns.id}::text))))`; + return and( + sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`, + includeCompleted ? sql`(${actionEligible} or ${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete')` : actionEligible, + sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueRecoveryActions.sourceIssueId}::text`, + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,sourceRunId}' = ${heartbeatRuns.id}::text`, + sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,runId}' = ${heartbeatRuns.id}::text`, + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,policyVersion}' = '1'`, + sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,schema}' = 'paperclip.system-recovery.v1'`, + sql`${issueRecoveryActions.evidence}#>>'{workspaceRecovery,schema}' = 'paperclip.workspace-recovery.v1'`, + sql`(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1' + or ${issueRecoveryActions.evidence}->'historicalWorkspaceSource'->>'schema' = 'paperclip.historical-workspace-source.v1' + or ${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted'))`, + ); +} + /** The preservation owner may reopen only its exact stop-only allocation. * This does not release the source's ordinary execution or destruction fence. */ export async function ownedWorkspaceRecoveryLease(db: Db, input: WorkspaceRecoverySourceIdentity & { diff --git a/server/src/services/workspace-recovery-work.ts b/server/src/services/workspace-recovery-work.ts new file mode 100644 index 00000000000..dea2bf28b46 --- /dev/null +++ b/server/src/services/workspace-recovery-work.ts @@ -0,0 +1,37 @@ +import type { Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { executionControlSchedule, registerExecutionControlWork } from "./execution-control-work.js"; +import type { EnvironmentRuntimeService } from "./environment-runtime.js"; +import { reconcileRetainedWorkspaces } from "./workspace-recovery.js"; + +export function registerWorkspaceRecoveryWork(db: Db, coordinator: ReturnType, + runtime: EnvironmentRuntimeService, onRecovered: Parameters[2], + canRun: () => boolean = () => true) { + const pending = new Set>(); + let stopped = false; + let scan: Promise | undefined; + let wakeWhileBusy = false; + let passiveSweep: { cursor?: string } = {}; + const schedule = executionControlSchedule(db); + const worker = registerExecutionControlWork(coordinator, [{ + queue: DELIVERY_QUEUES.workspaceRecovery, + onSettled: () => { passiveSweep = {}; if (pending.size >= 2) wakeWhileBusy = true; }, + run: () => scan = (async () => { + const work = await reconcileRetainedWorkspaces(db, runtime, onRecovered, true, continueSweep => { + if (!stopped && (continueSweep || wakeWhileBusy)) { wakeWhileBusy = false; worker.wake(); } + }, passiveSweep); + for (const job of work ?? []) { + pending.add(job); + const settled = () => { pending.delete(job); }; + void job.then(settled, settled); + } + })(), + next: () => pending.size >= 2 ? Promise.resolve(null) : schedule.workspaceRecovery(), + }], () => !stopped && canRun()); + return { ready: worker.ready, async stop() { + stopped = true; + await scan?.catch(() => {}); + await Promise.all(pending); + } }; +} diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 9edfdf1d391..e9519384bab 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -20,21 +20,29 @@ import { generateKeyPairSync, randomUUID, sign } from "node:crypto"; import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; +import express from "express"; +import request from "supertest"; import { and, eq, inArray, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db"; +import { agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations } from "@paperclipai/db"; import { resumePreservedResultFinalization } from "./preserved-result-finalization.js"; import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { EnvironmentLease } from "@paperclipai/shared"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { prepareNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js"; +import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import { prepareStoppedSandboxForRecovery } from "./sandbox-stop-and-retain.js"; import { recoverRetainedWorkspace, requestWorkspaceRecovery, scheduleRecoveredWorkspaceCleanup, reconcileRetainedWorkspaces, retainedWorkspaceCandidates } from "./workspace-recovery.js"; import * as activity from "./activity-log.js"; +import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerWorkspaceRecoveryWork } from "./workspace-recovery-work.js"; import { heartbeatService } from "./heartbeat.js"; import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js"; import { getExecutionBlocker } from "./execution-blocker.js"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/index.js"; import * as workerProof from "./native-runtime/native-workspace-platform-proof.js"; const targets = vi.hoisted(() => new Map()); vi.mock("./environment-execution-target.js", async importOriginal => { @@ -51,7 +59,13 @@ describe("automatic retained workspace recovery", () => { beforeAll(async () => { root = await mkdtemp(path.join(os.tmpdir(), "workspace-recovery-")); process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "recovery"; - temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-"); db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL"); + if (external) { + await applyPendingMigrations(external); + temporary = { connectionString: external, cleanup: async () => {} }; + } else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-"); + db = createDb(temporary.connectionString); + if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }, 30_000); afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); await rm(root, { recursive: true, force: true }); @@ -123,6 +137,178 @@ describe("automatic retained workspace recovery", () => { const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!; const run = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!; + it("wakes physical preservation from HTTP Retry before passive history without replaying the provider", async () => { + for (let index = 0; index < 5; index++) { + const old = await seed(); + await recoverRetainedWorkspace(old.input); + await db.update(issueRecoveryActions).set({ + id: `00000000-000${index}-4000-8000-${randomUUID().slice(-12)}`, + }).where(eq(issueRecoveryActions.id, old.actionId)); + } + const f = await seed(); + await adoptSystemRecovery(db, f); + const before = await action(f.actionId); + const exhausted = failSystemRecovery(readSystemRecovery(before.evidence)!, + new SystemRecoveryFailure("source_binding_invalid", "service_required")); + await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", + evidence: { ...before.evidence, systemRecovery: exhausted } }) + .where(eq(issueRecoveryActions.id, f.actionId)); + const errors = vi.fn(); + const competingCleanup = await retainedWorkspaceCandidates(db, undefined, false, true); + expect(competingCleanup).toHaveLength(5); + expect(competingCleanup.every(row => row.action.id.startsWith("00000000"))).toBe(true); + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors }); + const recovered = vi.fn(async () => {}); + const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, recovered); + try { + await worker.ready; + await db.update(agents).set({ status: "paused" }).where(eq(agents.companyId, f.companyId)); + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "board" }; next(); }); + app.use("/api", issueRoutes(db, {} as Parameters[1])); + app.use(errorHandler); + const body = { idempotencyKey: randomUUID(), observedRunId: f.runId }; + const retry = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202); + const repeated = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202); + const coalesced = await request(app).post(`/api/issues/${f.issueId}/retry`) + .send({ ...body, idempotencyKey: randomUUID() }).expect(202); + expect(repeated.body.requestId).toBe(retry.body.requestId); + expect(coalesced.body.requestId).toBe(retry.body.requestId); + await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state, + { timeout: 20_000 }).toBe("complete"); + expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n"); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0); + const reloaded = await request(app).get(`/api/issues/${f.issueId}/recovery-actions`).expect(200); + expect(JSON.stringify(reloaded.body)).toContain('"state":"complete"'); + expect(errors).not.toHaveBeenCalled(); + } finally { await coordinator.stop(); await worker.stop(); } + }, 30_000); + + it("reconstructs a resolved files-only preservation obligation at startup without replay", async () => { + const f = await seed("legacy"); + await adoptSystemRecovery(db, f); + const before = await action(f.actionId); + const automaticRecovery = { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" }; + await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked", + evidence: { ...before.evidence, automaticRecovery } }).where(eq(issueRecoveryActions.id, f.actionId)); + const errors = vi.fn(); + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors }); + const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, async () => {}); + try { + await worker.ready; + await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state, + { timeout: 20_000 }).toBe("complete"); + expect((await action(f.actionId)).evidence.automaticRecovery).toEqual(automaticRecovery); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); + expect(errors).not.toHaveBeenCalled(); + } finally { await coordinator.stop(); await worker.stop(); } + }, 30_000); + + it("fills a free export slot while another export is held and joins both at shutdown", async () => { + const held = await seed(), fast = await seed(); + const fixtures = [held, fast]; + const runtimeFor = (runId: string) => fixtures.find(fixture => fixture.runId === runId)!.environmentRuntime; + const runtime = { ...held.environmentRuntime, + resumeWorkspaceRecoveryLease: (input: Parameters[0]) => + runtimeFor(input.source.runId).resumeWorkspaceRecoveryLease(input), + retryPendingSandboxTeardown: (input: Parameters[0]) => + runtimeFor(input.lease.heartbeatRunId!).retryPendingSandboxTeardown(input), + }; + let active = 0, maximum = 0, release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const countExport = (fixture: typeof held, pause = false) => { + const resume = fixture.runtime.resumeRunLease.getMockImplementation()!; + fixture.runtime.resumeRunLease.mockImplementation(async input => { + active += 1; maximum = Math.max(maximum, active); + try { if (pause) await gate; return await resume(input); } finally { active -= 1; } + }); + }; + countExport(held, true); countExport(fast); + const errors = vi.fn(); + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: errors }); + const worker = registerWorkspaceRecoveryWork(db, coordinator, runtime, async () => {}); + try { + await expect.poll(() => held.runtime.resumeRunLease.mock.calls.length, { timeout: 20_000 }).toBe(1); + await expect.poll(async () => readSystemRecovery((await action(fast.actionId)).evidence)?.state, + { timeout: 20_000 }).toBe("complete"); + const third = await seed(); fixtures.push(third); countExport(third); + await requestTaskRetry({ db, companyId: third.companyId, issueId: third.issueId, actorId: "board", + observedRunId: third.runId, idempotencyKey: randomUUID() }); + await expect.poll(async () => readSystemRecovery((await action(third.actionId)).evidence)?.state, + { timeout: 20_000 }).toBe("complete"); + expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("running"); + expect(maximum).toBeLessThanOrEqual(2); + expect(await readFile(path.join(third.host, "saved.txt"), "utf8")).toBe("finished work\n"); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, third.companyId))).toHaveLength(1); + let stopped = false; + const stopping = Promise.all([coordinator.stop(), worker.stop()]).then(() => { stopped = true; }); + await Promise.resolve(); expect(stopped).toBe(false); + release(); await stopping; + expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("complete"); + expect(errors).not.toHaveBeenCalled(); + } finally { release(); await coordinator.stop(); await worker.stop(); } + }, 60_000); + + it("rotates past due sources whose physical locks belong to other workers", async () => { + const blocked = [await seed(), await seed()], available = await seed(); + for (const [index, fixture] of [...blocked, available].entries()) { + await adoptSystemRecovery(db, fixture); + const id = `00000001-000${index}-4000-8000-${randomUUID().slice(-12)}` as const; + await db.update(issueRecoveryActions).set({ id }).where(eq(issueRecoveryActions.id, fixture.actionId)); + fixture.actionId = id; fixture.input.actionId = id; + } + let unlock!: () => void; + const gate = new Promise(resolve => { unlock = resolve; }); + const held = blocked.map(fixture => { + let ready!: () => void; + const acquired = new Promise(resolve => { ready = resolve; }); + const lock = db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(fixture.companyId, fixture.runId)}, 0))`); + ready(); await gate; + }); + return { acquired, lock }; + }); + try { + await Promise.all(held.map(item => item.acquired)); + await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []); + expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true); + await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []); + expect(readSystemRecovery((await action(available.actionId)).evidence)?.state).toBe("complete"); + expect(await readFile(path.join(available.host, "saved.txt"), "utf8")).toBe("finished work\n"); + expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true); + } finally { unlock(); await Promise.all(held.map(item => item.lock)); } + }, 30_000); + + it("coalesces Retry while preservation owns its physical operation", async () => { + const f = await seed(); + await adoptSystemRecovery(db, f); + const resume = f.runtime.resumeRunLease.getMockImplementation()!; + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + f.runtime.resumeRunLease.mockImplementation(async input => { await gate; return resume(input); }); + const preserving = recoverRetainedWorkspace(f.input); + let requestId: string | null | undefined; + try { + await expect.poll(() => f.runtime.resumeRunLease.mock.calls.length).toBe(1); + const request = { db, companyId: f.companyId, issueId: f.issueId, actorId: "board", + idempotencyKey: randomUUID(), observedRunId: f.runId }; + const first = await requestTaskRetry(request); + requestId = first.requestId; + const second = await requestTaskRetry({ ...request, idempotencyKey: randomUUID() }); + expect(second.requestId).toBe(first.requestId); + } finally { release(); await preserving; } + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + expect(readSystemRecovery((await action(f.actionId)).evidence)?.intentId).toBe(requestId); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); + expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n"); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }, 30_000); + it("adopts only the exact historical infrastructure notice, preserving prose and user-action notices", async () => { const f = await seed("legacy", false); const metadata = { version: 1 as const, sourceRunId: f.runId, @@ -285,7 +471,7 @@ describe("automatic retained workspace recovery", () => { expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0); expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); await recoverRetainedWorkspace(f.input); - expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce(); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.phase, "workspace_recovery")))).toHaveLength(1); }, 30_000); @@ -304,9 +490,9 @@ describe("automatic retained workspace recovery", () => { expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered"); expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n"); expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n"); - expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce(); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); await recoverRetainedWorkspace({ ...f.input, environmentRuntime: runtime }); - expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce(); + expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); }, 30_000); @@ -405,7 +591,7 @@ describe("automatic retained workspace recovery", () => { f.runtime.resumeRunLease.mockImplementationOnce(async () => { ready(); await gate; return { providerLeaseId: f.providerLeaseId, metadata: { remoteCwd: f.remote } }; }); const first = recoverRetainedWorkspace(f.input); await started; - try { await recoverRetainedWorkspace(f.input); expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce(); } + try { await recoverRetainedWorkspace(f.input); expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1); } finally { release(); await first; } expect(readWorkspaceRecovery((await action(f.actionId)).evidence)?.status).toBe("recovered"); }, 30_000); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index e2fae8179e5..28b0f301d28 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -1,3 +1,5 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { beginIdleTrackedWork } from "./task-admission.js"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { resumePreservedResultFinalization } from "./preserved-result-finalization.js"; import { and, asc, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; @@ -19,7 +21,7 @@ import { classifyNativeWorkspaceFailure } from "./native-runtime/native-workspac import { adoptSystemRecovery, loadSystemRecovery, updateSystemRecovery } from "./system-recovery.js"; import { createSystemRecovery, failSystemRecovery, readSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure, type SystemRecoveryState } from "./system-recovery-state.js"; -import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js"; +import { inspectWorkspaceRecoverySource, workspaceRecoverySourceEligible } from "./workspace-recovery-source.js"; import { preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js"; import { adoptHistoricalWorkspaceRecoveries } from "./workspace-recovery-backfill.js"; @@ -48,12 +50,13 @@ export async function requestWorkspaceRecovery(input: Source & { db: Db; idempot requestIds: [...(prior?.requestIds ?? []), input.idempotencyKey] }; const previous = readSystemRecovery(action.evidence); const active = previous && ["scheduled", "running", "awaiting_evidence"].includes(previous.state); - const system = active ? { ...previous, revision: previous.revision + 1, - nextAttemptAt: previous.state === "running" ? previous.nextAttemptAt : Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id), + const system = previous?.state === "running" ? previous : active ? { ...previous, revision: previous.revision + 1, + nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) } : { ...createSystemRecovery(run.id), revision: (previous?.revision ?? -1) + 1, preservationOperationIds: previous?.preservationOperationIds ?? [], providerCooldownAt: previous?.providerCooldownAt ?? null, nextAttemptAt: previous?.providerCooldownAt ?? null }; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery); await tx.update(issueRecoveryActions).set({ ownerType: "system", ownerAgentId: null, ownerUserId: null, evidence: { ...action.evidence, workspaceRecovery: state, systemRecovery: system }, updatedAt: new Date() }) .where(eq(issueRecoveryActions.id, action.id)); @@ -286,6 +289,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ const cursors = new WeakMap(); const workInFlight = new WeakMap>(); +const unfinishedCursors = new WeakMap(); /** Destruction is a later cleanup obligation, authorized only by a matching * saved-file receipt and the original recorded disposition. */ @@ -309,29 +313,31 @@ export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof hear sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`, sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`))); } -export function retainedWorkspaceCandidates(db: Db, cursor?: string) { +export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false) { + const cleanup = sql`(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy' and exists ( + select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} + and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id} + and ${environmentLeases.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' + and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success' + and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null))`; + const waitingInput = sql`exists (select 1 from ${agentWakeupRequests} + where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId} + and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user' + and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text)`; return db.select({ action: issueRecoveryActions, run: heartbeatRuns }).from(issueRecoveryActions) .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, issueRecoveryActions.companyId), eq(heartbeatRuns.id, sql`case when ${issueRecoveryActions.evidence}->>'runId' ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' then (${issueRecoveryActions.evidence}->>'runId')::uuid end`))) .where(and(inArray(heartbeatRuns.status, terminal), - sql`(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1' - or ${issueRecoveryActions.evidence}->'historicalWorkspaceSource'->>'schema' = 'paperclip.historical-workspace-source.v1' - or ${issueRecoveryActions.cause} in ('native_workspace_sync_out_unrecoverable', 'native_workspace_sync_out_retry_exhausted'))`, - sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`, + workspaceRecoverySourceEligible(true), + unfinishedOnly ? sql`${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence')` : undefined, + obligationsOnly ? sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence') + or (${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' = 'complete' and ${cleanup}) or ${waitingInput})` : undefined, sql`(${issueRecoveryActions.evidence}#>>'{systemRecovery,state}' in ('scheduled', 'running', 'awaiting_evidence', 'complete', 'exhausted') and (${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' = 'null'::jsonb or ${issueRecoveryActions.evidence}#>'{systemRecovery,nextAttemptAt}' <= to_jsonb(${Date.now()}::bigint)))`, sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or - (${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy' and exists ( - select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} - and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id} - and ${environmentLeases.id}::text = ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,leaseId}' - and ${environmentLeases.status} = 'released' and ${environmentLeases.cleanupStatus} = 'success' - and ${environmentLeases.metadata}->'workspaceRecoveryCleanup' is null)) or exists ( - select 1 from ${agentWakeupRequests} where ${agentWakeupRequests.companyId} = ${issueRecoveryActions.companyId} - and ${agentWakeupRequests.status} = 'deferred_issue_execution' and ${agentWakeupRequests.requestedByActorType} = 'user' - and ${agentWakeupRequests.payload}->>'issueId' = ${issueRecoveryActions.sourceIssueId}::text))`, + ${cleanup} or ${waitingInput})`, cursor ? gt(issueRecoveryActions.id, cursor) : undefined, )).orderBy(asc(issueRecoveryActions.id)).limit(5); } @@ -339,19 +345,29 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string) { /** Bounded discovery shares the periodic recovery lifecycle. A repaired source * still owes cleanup even without saved input or after task reconciliation. */ export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRuntimeService, - onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise) { + onRecovered: (run: typeof heartbeatRuns.$inferSelect) => Promise, obligationsOnly = false, + onFinished?: (continueSweep: boolean) => void, passiveSweep?: { cursor?: string }) { await adoptHistoricalWorkspaceRecoveries(db); let pending = workInFlight.get(db); if (!pending) { pending = new Set(); workInFlight.set(db, pending); } if (pending.size >= 2) return; - let rows = await retainedWorkspaceCandidates(db, cursors.get(db)); - if (!rows.length) rows = await retainedWorkspaceCandidates(db); + let rows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true); + if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, true); + const unfinished = rows.length > 0; + if (!rows.length) rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly); + if (!rows.length && !passiveSweep) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly); + const work: Promise[] = []; + let hasUnvisited = false; for (const { action, run } of rows) { - if (pending.size >= 2) break; - cursors.set(db, action.id); + if (pending.size >= 2) { hasUnvisited = true; break; } + if (unfinished) unfinishedCursors.set(db, action.id); + else if (passiveSweep) passiveSweep.cursor = action.id; + else cursors.set(db, action.id); if (pending.has(run.id)) continue; pending.add(run.id); - void (async () => { try { + const finish = beginIdleTrackedWork(); + const physical = ["scheduled", "running", "awaiting_evidence"].includes(readSystemRecovery(action.evidence)?.state ?? ""); + work.push((async () => { try { await recoverRetainedWorkspace({ db, environmentRuntime: runtime, companyId: action.companyId, issueId: action.sourceIssueId, runId: run.id, actionId: action.id }); const [updated] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)); const recovered = readWorkspaceRecovery(updated?.evidence ?? null)?.status === "recovered"; @@ -370,6 +386,7 @@ export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRu } } } catch { logger.warn({ runId: run.id }, "Retained workspace remains pending"); } - finally { pending.delete(run.id); } })(); + finally { pending.delete(run.id); finish(); onFinished?.(physical || hasUnvisited); } })()); } + return work; } diff --git a/server/src/vendor/paperclip-runner/testing.ts b/server/src/vendor/paperclip-runner/testing.ts index e889df919de..06a85b6f36d 100644 --- a/server/src/vendor/paperclip-runner/testing.ts +++ b/server/src/vendor/paperclip-runner/testing.ts @@ -26,6 +26,10 @@ const sourceUrl = new URL( ); const runnerTesting = await import(sourceUrl.href) as RunnerTestingModule; +export const startOpenCodeMcpBridge: + RunnerTestingModule["startOpenCodeMcpBridge"] = + runnerTesting.startOpenCodeMcpBridge; + export const CAPABILITY_HIGH_RISK_SEMANTIC_VECTORS: RunnerTestingModule["CAPABILITY_HIGH_RISK_SEMANTIC_VECTORS"] = runnerTesting.CAPABILITY_HIGH_RISK_SEMANTIC_VECTORS; diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index c9ad6c6d91d..72dc383a7a9 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -506,6 +506,17 @@ select arbitrary destinations. Gateway response caching and fallback are refused Request journal and usage receipts contain no prompt/response bodies or keys. The manifest, journal, and receipts are not public report projections. +`boundedEvalPayload` is the shared, credential-free request validator and +normalizer. Qualification callers must apply it before recording budget-denial +proof. The forwarding path applies the same policy before a ledger reservation. +OpenAI tools may use one namespace level around local function or custom tools. +Function and custom leaves require nonblank string names. +Namespace wrappers contain only `type`, `name`, optional `description`, and +`tools`. Nested namespaces, hosted tools, and tool lists attached to leaves fail +closed. The request may contain at most 1,024 tool definitions, counting namespace +wrappers and their children. Anthropic tool policy stays separate. Output limits, +billing-mode restrictions, and multimodal exclusions apply in both paths. + ## Configured repository and conversion proof The repository proof flow seeds only the exact `workspace` child of its resolved diff --git a/tests/runner-e2e/eval-gateway.test.ts b/tests/runner-e2e/eval-gateway.test.ts index c4c9e785b8d..150fac62f5b 100644 --- a/tests/runner-e2e/eval-gateway.test.ts +++ b/tests/runner-e2e/eval-gateway.test.ts @@ -6,7 +6,7 @@ import { connect } from "node:net"; import { afterEach, expect, it, vi } from "vitest"; import { openEvalCostLedger, readEvalCostLedger } from "./cost-ledger.js"; import { beginEvalAgentPhase, createEvalAttemptPhase, readEvalAttemptPhase } from "./attempt-phase.js"; -import { evalRequestReservation, EvalUsageStream, forwardEvalRequest, startEvalGateway, type EvalGatewayRoute, type EvalGatewayReceipt } from "./eval-gateway.js"; +import { boundedEvalPayload, evalRequestReservation, EvalUsageStream, forwardEvalRequest, startEvalGateway, type EvalGatewayRoute, type EvalGatewayReceipt } from "./eval-gateway.js"; const directories: string[] = [], closes: Array<() => Promise> = []; afterEach(async () => { for (const close of closes.splice(0).reverse()) await close(); for (const dir of directories.splice(0)) await rm(dir, { recursive: true, force: true }); }); const identity = { executionId: "hello", attemptId: "attempt", phase: "agent" as const }; @@ -135,6 +135,71 @@ it("admits local function and custom tools inside a Codex namespace", async () = expect(result.outcome).toBe("succeeded"); expect(transport).toHaveBeenCalledOnce(); }); +it("validates the native daemon's namespace envelope before reserving and forwarding", async () => { + const f = await fixture(); + const localTool = (name: string) => ({ type: "function", name, description: "Synthetic local tool", + parameters: { type: "object", properties: {}, additionalProperties: false }, strict: true }); + const tools = [...Array.from({ length: 7 }, (_, index) => localTool(`local_${index}`)), + { type: "namespace", name: "local_tools", description: "Synthetic local namespace", + tools: Array.from({ length: 5 }, (_, index) => localTool(`operation_${index}`)) }]; + const payload = Object.freeze({ ...f.payload, tools }); + const normalized = boundedEvalPayload(payload, { provider: route.provider, price: route.price, maximumOutputTokens: 100 }); + expect(normalized).toEqual({ ...payload, max_output_tokens: 100, service_tier: "default" }); + expect(payload).not.toHaveProperty("service_tier"); + expect(payload.max_output_tokens).toBe(500); + expect(f.ledger.snapshot().requests).toHaveLength(0); + const transport = vi.fn(async (_url, init) => { + expect(JSON.parse(init!.body as string)).toEqual(normalized); + expect((await readEvalCostLedger(f.file)).state.snapshot().inflight).toBe(1); + return response(); + }); + const receipt = await forwardEvalRequest({ ...f, payload, transport }); + expect(receipt).toMatchObject({ outcome: "succeeded", costUsd: "0.000068000", final: true }); + expect(transport).toHaveBeenCalledOnce(); +}); +it.each([ + { type: "namespace", name: " ", tools: [{ type: "function", name: "local" }] }, + { type: "namespace", name: "local", description: {}, tools: [{ type: "function", name: "local" }] }, + { type: "namespace", name: "local", tools: [] }, + { type: "namespace", name: "local", tools: null }, + { type: "namespace", name: "local", tools: [null] }, + { type: ["function"], name: "local" }, + { type: ["custom"], name: "local" }, + { type: "namespace", name: "local", tools: [{ type: ["function"], name: "local" }] }, + { type: "namespace", name: "local", tools: [{ type: ["custom"], name: "local" }] }, + ...[undefined, null, 7, "", " "].flatMap(name => ["function", "custom"].flatMap(type => + [{ type, name }, { type: "namespace", name: "local", tools: [{ type, name }] }])), + { type: "namespace", name: "local", tools: [{ type: "function", name: "local" }], future_hosted_tool: true }, + { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "function", name: "local" }] }] }, + ...["web_search", "file_search", "image_generation", "code_interpreter", "computer", "mcp", "unknown"].map(type => + ({ type: "namespace", name: "local", tools: [{ type }] })), +])("rejects malformed, nested, or hosted namespace tools before any side effects", async tool => { + const f = await fixture(), transport = vi.fn(), persist = vi.fn(); + const payload = { ...f.payload, tools: [tool] }; + expect(() => boundedEvalPayload(payload, f.route)).toThrow("unqualified_hosted_tool"); + await expect(forwardEvalRequest({ ...f, payload, transport, persist })).rejects.toThrow("unqualified_hosted_tool"); + expect(transport).not.toHaveBeenCalled(); expect(persist).not.toHaveBeenCalled(); + expect(f.ledger.snapshot().requests).toHaveLength(0); +}); +it.each(["flat", "namespace"] as const)("bounds the total %s tool definitions including namespace wrappers", async shape => { + const f = await fixture(), transport = vi.fn(), persist = vi.fn(); + const tool = { type: "function", name: "local_tool" }; + const tools = shape === "flat" ? Array.from({ length: 1024 }, () => tool) + : [{ type: "namespace", name: "local", tools: Array.from({ length: 1023 }, () => tool) }]; + expect(boundedEvalPayload({ ...f.payload, tools }, f.route).tools).toEqual(tools); + const payload = { ...f.payload, tools: [...tools, tool] }; + await expect(forwardEvalRequest({ ...f, payload, transport, persist })).rejects.toThrow("unqualified_hosted_tool"); + expect(transport).not.toHaveBeenCalled(); expect(persist).not.toHaveBeenCalled(); + expect(f.ledger.snapshot().requests).toHaveLength(0); +}); +it("keeps Anthropic custom tools and output limits separate from OpenAI namespaces", () => { + const anthropic = { provider: "anthropic" as const, price: { ...route.price, provider: "anthropic" as const }, maximumOutputTokens: 100 }; + const tools = [{ name: "local_tool", input_schema: { type: "object" } }, { type: "custom", name: "other_tool" }]; + expect(boundedEvalPayload({ model: "fixture", tools, max_tokens: 500 }, anthropic)) + .toEqual({ model: "fixture", tools, max_tokens: 100, service_tier: "standard_only" }); + expect(() => boundedEvalPayload({ model: "fixture", tools: [{ type: "namespace", name: "local", tools }] }, anthropic)) + .toThrow("unqualified_hosted_tool"); +}); it.each([ { type: "namespace", name: "local", tools: [{ type: "web_search" }] }, { type: "namespace", name: "local", tools: [{ type: "namespace", name: "nested", tools: [{ type: "web_search" }] }] }, diff --git a/tests/runner-e2e/eval-gateway.ts b/tests/runner-e2e/eval-gateway.ts index 6d42ce680e5..a90cd28047f 100644 --- a/tests/runner-e2e/eval-gateway.ts +++ b/tests/runner-e2e/eval-gateway.ts @@ -65,19 +65,34 @@ function modelReservationBound(route: EvalGatewayRoute, reservation: string) { }) : undefined; } -function boundedPayload(value: unknown, route: EvalGatewayRoute) { +function qualifiedOpenAiTools(tools: unknown): boolean { + if (!Array.isArray(tools)) return false; + let remaining = 1024; + const leaf = (tool: unknown): boolean => { + const definition = row(tool); + return --remaining >= 0 && (definition.type === "function" || definition.type === "custom") + && typeof definition.name === "string" && Boolean(definition.name.trim()) && !Object.hasOwn(definition, "tools"); + }; + return tools.every(tool => { + const definition = row(tool); + if (definition.type !== "namespace") return leaf(tool); + return --remaining >= 0 && typeof definition.name === "string" && Boolean(definition.name.trim()) + && (definition.description === undefined || typeof definition.description === "string") + && Object.keys(definition).every(key => ["type", "name", "description", "tools"].includes(key)) + && Array.isArray(definition.tools) && definition.tools.length > 0 && definition.tools.every(leaf); + }); +} + +export function boundedEvalPayload(value: unknown, route: Pick): Record { const body = row(value); if (body.model !== route.price.model) throw new EvalRequestError("unqualified_model"); if (body.background === true || body.mcp_servers !== undefined || body.service_tier !== undefined && !["auto", "default", "standard_only"].includes(String(body.service_tier))) throw new EvalRequestError("unqualified_billing_mode"); if (body.speed !== undefined && body.speed !== "standard" || body.inference_geo !== undefined) throw new EvalRequestError("unqualified_billing_mode"); - if (body.tools !== undefined && (!Array.isArray(body.tools) || body.tools.some(tool => { - const definition = row(tool), type = definition.type; - if (route.provider !== "openai") return type !== undefined && type !== "custom" || Object.hasOwn(definition, "tools"); - if (type === "namespace") return typeof definition.name !== "string" || !definition.name - || !Array.isArray(definition.tools) || !definition.tools.length - || definition.tools.some(nested => !["function", "custom"].includes(String(row(nested).type)) || Object.hasOwn(row(nested), "tools")); - return !["function", "custom"].includes(String(type)) || Object.hasOwn(definition, "tools"); - }))) throw new EvalRequestError("unqualified_hosted_tool"); + if (body.tools !== undefined && (route.provider === "openai" ? !qualifiedOpenAiTools(body.tools) + : !Array.isArray(body.tools) || body.tools.some(tool => { + const definition = row(tool), type = definition.type; + return type !== undefined && type !== "custom" || Object.hasOwn(definition, "tools"); + }))) throw new EvalRequestError("unqualified_hosted_tool"); const inspect = (item: unknown): void => { if (Array.isArray(item)) { item.forEach(inspect); return; } if (!item || typeof item !== "object") return; @@ -119,8 +134,8 @@ export async function forwardEvalRequest(input: { transport?: typeof fetch; }): Promise { const { route, ledger } = input; - let body: ReturnType; - try { body = boundedPayload(input.payload, route); } + let body: ReturnType; + try { body = boundedEvalPayload(input.payload, route); } catch (error) { throw error instanceof EvalRequestError ? error : new EvalRequestError("invalid_request_payload"); } const reservation = evalRequestReservation(route); if (!route.apiKey || /[\r\n]/.test(route.apiKey)) throw new Error("missing_gateway_credential"); diff --git a/ui/src/components/Layout.production.tsx b/ui/src/components/Layout.production.tsx index 56afb99fc06..cdb984d85d7 100644 --- a/ui/src/components/Layout.production.tsx +++ b/ui/src/components/Layout.production.tsx @@ -45,6 +45,7 @@ import { useSidebar } from "../context/SidebarContext"; import { useKeyboardShortcuts } from "../hooks/useKeyboardShortcuts"; import { useCompanyPageMemory } from "../hooks/useCompanyPageMemory"; import { useMobileNavVisibility } from "../hooks/useMobileNavVisibility"; +import { useMobileVisualViewport } from "../hooks/useMobileVisualViewport"; import { healthApi } from "../api/health"; import { resolveArchivedCompanyBounce, @@ -178,6 +179,7 @@ export function Layout() { const scrollMemory = useRef(new NavigationScrollMemory()); const activeScrollKey = useRef(location.key); const mobileNavVisible = useMobileNavVisibility(isMobile, location.pathname); + const mobileViewportRef = useMobileVisualViewport(isMobile); const [shortcutsOpen, setShortcutsOpen] = useState(false); const matchedCompany = useMemo(() => { if (!companyPrefix) return null; @@ -588,8 +590,10 @@ export function Layout() { return (
@@ -663,12 +667,13 @@ export function Layout() {
) : null}
-
+
(location.key); const mobileNavVisible = useMobileNavVisibility(isMobile, location.pathname); + const mobileViewportRef = useMobileVisualViewport(isMobile); const [shortcutsOpen, setShortcutsOpen] = useState(false); const matchedCompany = useMemo(() => { if (!companyPrefix) return null; @@ -574,8 +576,10 @@ export function Layout({ sidebarSections }: { sidebarSections?: ReactNode }) { return (
-
+
{isMobile && sidebarOpen && ( + : null + ); const renderBrief = useCallback( - () => issueBrief ? <>{props.historyControl} : null, - [issueBrief, props.historyControl], + () => issueBrief ? <>{historyControl} : null, + [issueBrief, historyControl], ); const assignedAgentForNotice = useMemo(() => { @@ -3040,25 +3048,19 @@ export function TaskChatThread(props: TaskChatThreadProps) { -
@@ -3093,8 +3095,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
{items.length === 0 && !tailRunId && !optimisticRunnerStartup ? (
{threadHeaderWithBlockers ? (
{threadHeaderWithBlockers}{!issueBrief ? props.historyControl : null}} + header={<>{threadHeaderWithBlockers}{!issueBrief ? historyControl : null}} renderInteraction={renderInteraction} renderBrief={renderBrief} renderMessageActions={renderMessageActions} @@ -3241,7 +3240,6 @@ export function TaskChatThread(props: TaskChatThreadProps) { } contentKey={`${autoFollowContentKey}:${historyRevealed}`} className={isMobile ? undefined : "pt-3"} - scroll={!isMobile} /> )} diff --git a/ui/src/components/ai-connections/AiConnectionField.test.tsx b/ui/src/components/ai-connections/AiConnectionField.test.tsx index cc9e6395173..ecf976d41cb 100644 --- a/ui/src/components/ai-connections/AiConnectionField.test.tsx +++ b/ui/src/components/ai-connections/AiConnectionField.test.tsx @@ -17,7 +17,6 @@ vi.mock("./AiConnectionCredentialStep", () => ({ return
Provider sign-in
; }, })); -vi.mock("./AiConnectionManagement", () => ({ AiConnectionLegacyNotice: () => null })); vi.mock("./AiProviderSetup", () => ({ AiProviderSetup: () =>
Advanced provider setup
})); vi.mock("@/pages/apps/AppLogo", () => ({ AppLogo: () => null })); @@ -82,6 +81,33 @@ it("reconnects the unavailable personal default in place", async () => { expect(onChange).toHaveBeenCalledWith({ provider: "anthropic", method: "subscription", mode: "responsible_user" }); }); +it("identifies duplicate-named accounts before adopting the exact personal grant", async () => { + const personal = account({ id: "00000000-0000-4000-8000-000000000001", grantId: "00000000-0000-4000-8000-000000000002", provider: "openai", method: "api_key", name: "OpenAI", ownerName: "Avery", status: "connected", isDefault: false }); + const shared = account({ id: "00000000-0000-4000-8000-000000000003", grantId: "00000000-0000-4000-8000-000000000004", provider: "openai", method: "subscription", name: "OpenAI", accountLabel: "engineering@example.test", ownership: "shared", ownerUserId: undefined, status: "connected", isDefault: false }); + const sharedKey = account({ id: "00000000-0000-4000-8000-000000000005", grantId: "00000000-0000-4000-8000-000000000006", provider: "openai", method: "api_key", name: "OpenAI", ownerName: "Engineering", ownership: "shared", ownerUserId: undefined, status: "connected", isDefault: false }); + const unavailable = account({ id: "expired-account", grantId: "expired-grant", provider: "openai", method: "api_key", name: "Expired account", ownerName: "Avery" }); + mocks.list.mockResolvedValue({ currentUserId: "owner", connections: [shared, personal, sharedKey, unavailable], canManageConnections: true }); + flushSync(() => root.render( + + )); + await settle(); + await click("Choose a managed connection"); + const trigger = container.querySelector('[role="combobox"][aria-label="Connection"]')!; + flushSync(() => trigger.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }))); + await settle(); + const choices = [...document.querySelectorAll('[role="option"]')]; + expect(choices.map(choice => choice.textContent)).toContain("OpenAI · API key · Personal · Avery"); + expect(choices.map(choice => choice.textContent)).toContain("OpenAI · ChatGPT subscription · Company shared · engineering@example.test"); + expect(choices.map(choice => choice.textContent)).toContain("OpenAI · API key · Company shared · Engineering"); + expect(choices.find(choice => choice.textContent?.includes("Expired account"))?.getAttribute("aria-disabled")).toBe("true"); + flushSync(() => choices.find(choice => choice.textContent === "OpenAI · API key · Personal · Avery")!.click()); + await settle(); + expect(document.querySelector('[role="dialog"]')?.textContent).toContain("OpenAI · API key · Personal · Avery"); + expect(onChange).not.toHaveBeenCalled(); + await click("Use this binding when saved"); + expect(onChange).toHaveBeenCalledExactlyOnceWith({ provider: "openai", method: "api_key", mode: "delegated", connectionId: personal.id, grantId: personal.grantId }); +}); + it("opens provider setup directly when connecting from the advanced mode", async () => { await mount([], true, true); await click("Connect another account"); @@ -163,7 +189,7 @@ it("offers Pi account selection before choosing its required model", async () => }); it.each(["openai", "anthropic", "google", "openrouter"] as const)("keeps a matching saved Pi %s API-key account selectable", async provider => { - const saved = account({ provider, method: "api_key", name: "Pi provider account", status: "connected" }); + const saved = account({ provider, method: "api_key", name: "Pi provider account", accountLabel: "avery@example.test", status: "connected" }); mocks.list.mockResolvedValue({ currentUserId: "owner", connections: [saved], canManageConnections: true }); flushSync(() => root.render( trigger.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }))); await settle(); const choice = [...document.querySelectorAll('[role="option"]')].find(node => node.textContent?.includes("Pi provider account")); diff --git a/ui/src/components/ai-connections/AiConnectionField.tsx b/ui/src/components/ai-connections/AiConnectionField.tsx index 5fcbe5e2757..d7d40336afd 100644 --- a/ui/src/components/ai-connections/AiConnectionField.tsx +++ b/ui/src/components/ai-connections/AiConnectionField.tsx @@ -14,6 +14,7 @@ import { AiConnectionSelect } from "./AiConnectionSelect"; import { AiProviderSetup } from "./AiProviderSetup"; import { AiConnectionCredentialStep } from "./AiConnectionCredentialStep"; import { AiConnectionLegacyNotice } from "./AiConnectionManagement"; +import { aiConnectionDisplayLabel } from "./model"; import { Button } from "@/components/ui/button"; import { Checkbox } from "@/components/ui/checkbox"; import { @@ -89,6 +90,11 @@ export function AiConnectionField({ enabled: Boolean(provider), }); const personalDefault = accounts.data?.connections.find((account) => account.provider === provider && account.isDefault && account.ownership === "personal" && account.ownerUserId === accounts.data.currentUserId); + const pendingAccount = pendingAdoption?.mode === "responsible_user" + ? personalDefault + : pendingAdoption && pendingAdoption.mode !== "router" + ? accounts.data?.connections.find(account => account.id === pendingAdoption.connectionId && account.grantId === pendingAdoption.grantId) + : undefined; const selectDefault = useMutation({ mutationFn: async (result: NonNullable) => { // Reconnect retains the existing default and its access. A new account @@ -177,10 +183,8 @@ export function AiConnectionField({ {pendingAdoption?.mode !== "router" &&

{pendingAdoption?.mode === "responsible_user" - ? `Responsible user’s default. For you: ${accounts.data?.connections.find((account) => account.isDefault && account.provider === provider)?.name ?? "Not connected"}. Other users use their own default.` - : accounts.data?.connections.find( - (account) => account.id === pendingAdoption?.connectionId, - )?.name} + ? `Responsible user’s default. For you: ${pendingAccount ? aiConnectionDisplayLabel(pendingAccount) : "Not connected"}. Other users use their own default.` + : pendingAccount && aiConnectionDisplayLabel(pendingAccount)}

} {pendingAdoption?.mode !== "router" &&

After adoption, missing credentials block execution. Previous diff --git a/ui/src/components/ai-connections/AiConnectionSelect.tsx b/ui/src/components/ai-connections/AiConnectionSelect.tsx index 4eb82c01f1a..1eb36785f1e 100644 --- a/ui/src/components/ai-connections/AiConnectionSelect.tsx +++ b/ui/src/components/ai-connections/AiConnectionSelect.tsx @@ -14,15 +14,15 @@ import { } from "@/components/ui/select"; import { AppLogo } from "@/pages/apps/AppLogo"; import { Button } from "@/components/ui/button"; -import { aiConnectionProblem, personalAiDefault } from "./model"; +import { aiConnectionDisplayLabel, aiConnectionProblem, personalAiDefault } from "./model"; import type { AiConnectionPickerProps } from "./AiConnectionPicker"; import type { AiConnectionSummary } from "./model"; -function ConnectionLabel({ connection, label }: { connection: AiConnectionSummary; label?: string }) { +function ConnectionLabel({ connection, label, wrap = false }: { connection: AiConnectionSummary; label?: string; wrap?: boolean }) { return ( - {label ?? connection.name} + {label ?? aiConnectionDisplayLabel(connection)} ); } @@ -119,7 +119,7 @@ export function AiConnectionSelect({ onValueChange={change} disabled={readOnly || loading || Boolean(error)} > - + {incompatible && selected && ( - + )} {compatible.map((c) => ( @@ -144,7 +144,7 @@ export function AiConnectionSelect({ value={c.id} disabled={Boolean(aiConnectionProblem(c))} > - + ))} {(canUseDefault || value?.mode === "responsible_user") && ( diff --git a/ui/src/components/ai-connections/model.ts b/ui/src/components/ai-connections/model.ts index e6e645be500..3ce4ad2e080 100644 --- a/ui/src/components/ai-connections/model.ts +++ b/ui/src/components/ai-connections/model.ts @@ -57,6 +57,16 @@ export function aiMethodLabel(provider: AiProvider, method: AiAuthMethod) { : provider === "github" ? "Personal access token" : "API key"; } +export function aiConnectionDisplayLabel(connection: AiConnectionSummary) { + const personal = connection.ownership === "personal"; + return [ + connection.name, + aiMethodLabel(connection.provider, connection.method), + personal ? "Personal" : "Company shared", + connection.accountLabel || connection.ownerName, + ].filter(Boolean).join(" · "); +} + export function matchesAiRequirement( connection: AiConnectionSummary, requirement: AiConnectionRequirement, diff --git a/ui/src/components/task-chat/TaskChatComposer.test.tsx b/ui/src/components/task-chat/TaskChatComposer.test.tsx index 4310bd4e545..0c739124229 100644 --- a/ui/src/components/task-chat/TaskChatComposer.test.tsx +++ b/ui/src/components/task-chat/TaskChatComposer.test.tsx @@ -856,7 +856,7 @@ describe("TaskChatComposer", () => { render(); expect(editable().dataset.contentClassName).toContain("min-h-(--sz-48px)"); - expect(editable().dataset.contentClassName).toContain("max-h-(--sz-28dvh)"); + expect(editable().dataset.contentClassName).toContain("max-h-(--tc-composer-editor-max-height)"); }); it("submits the trimmed body on Cmd+Enter and clears the draft", async () => { diff --git a/ui/src/components/task-chat/TaskChatComposer.tsx b/ui/src/components/task-chat/TaskChatComposer.tsx index 8d162be068f..bf332d3a40b 100644 --- a/ui/src/components/task-chat/TaskChatComposer.tsx +++ b/ui/src/components/task-chat/TaskChatComposer.tsx @@ -1375,7 +1375,7 @@ export function TaskChatComposer({ className={cn(disabled && "opacity-60")} contentClassName={ mobile - ? "max-h-(--sz-28dvh) min-h-(--sz-48px) overflow-y-auto px-1 py-1 text-base scrollbar-auto-hide" + ? "max-h-(--tc-composer-editor-max-height) min-h-(--sz-48px) overflow-y-auto px-1 py-1 text-base scrollbar-auto-hide" : "max-h-(--sz-28dvh) min-h-(--sz-48px) overflow-y-auto px-1 py-1 text-sm scrollbar-auto-hide" } /> diff --git a/ui/src/components/task-chat/TaskChatComposerDock.tsx b/ui/src/components/task-chat/TaskChatComposerDock.tsx index 03324b36aed..667f29a9ea2 100644 --- a/ui/src/components/task-chat/TaskChatComposerDock.tsx +++ b/ui/src/components/task-chat/TaskChatComposerDock.tsx @@ -8,7 +8,7 @@ interface TaskChatComposerDockProps { concealed?: boolean; } -/** The task thread and its Storybook phone previews share the same composer spacing. */ +/** A normal flex footer beside the message scroller, on desktop and mobile. */ export function TaskChatComposerDock({ children, mobile, streamlined, concealed = false }: TaskChatComposerDockProps) { return

n + l.length, 0); diff --git a/ui/src/components/task-chat/TaskMessageScroller.test.tsx b/ui/src/components/task-chat/TaskMessageScroller.test.tsx index dabc297f03e..379fba4d6d9 100644 --- a/ui/src/components/task-chat/TaskMessageScroller.test.tsx +++ b/ui/src/components/task-chat/TaskMessageScroller.test.tsx @@ -125,6 +125,36 @@ describe("TaskMessageScroller", () => { vi.unstubAllGlobals(); }); + it("does not measure historical rows on each streaming update while following the bottom", () => { + let initialized = false; + let rowMeasurements = 0; + const update = (tick: number) => flushSync(() => root.render( + +
{ + if (node && !initialized) { + fakeGeometry(node.parentElement!, { scrollHeight: 20000, clientHeight: 400 }); + initialized = true; + } + }}> + {Array.from({ length: 200 }, (_, index) =>
{ + if (node) node.getBoundingClientRect = () => { + rowMeasurements++; + const top = index * 100 - scroller().scrollTop; + return { top, bottom: top + 100, height: 100 } as DOMRect; + }; + }}>History {index}
)} +

Live response {tick}

+
+ , + )); + update(0); + rowMeasurements = 0; + for (let tick = 1; tick <= 30; tick++) update(tick); + expect(container.textContent).toContain("Live response 30"); + expect(scroller().scrollTop).toBe(20000); + expect(rowMeasurements).toBe(0); + }); + it("applies same-task hash changes and restores each history entry without remounting", async () => { let initialized = false; function navigate(key: string, hash: string, restore = false) { @@ -178,6 +208,42 @@ describe("TaskMessageScroller", () => { expect(scroller().scrollTop).toBe(150); }); + it("restores bottom-follow on Back after the viewport and message heights change", () => { + const configured = new WeakSet(); + function mount(restore: boolean, scrollHeight: number, clientHeight: number) { + flushSync(() => root.render( + + +
{ + const viewport = node?.parentElement; + if (viewport && !configured.has(viewport)) { + fakeGeometry(viewport, { scrollHeight, clientHeight }); + let top = 0; + Object.defineProperty(viewport, "scrollTop", { + configurable: true, + get: () => top, + set: (value: number) => { + top = Math.max(0, Math.min(value, viewport.scrollHeight - viewport.clientHeight)); + }, + }); + configured.add(viewport); + } + }}>Latest response
+ + , + )); + } + mount(false, 1000, 400); + flushSync(() => root.render(null)); + // Rotation changed wrapping and the available message viewport while away. + mount(true, 2000, 500); + expect(scroller().scrollTop).toBe(1500); + expect(pill()).toBeNull(); + Object.defineProperty(scroller(), "scrollHeight", { value: 2400, configurable: true }); + mount(true, 2400, 500); + expect(scroller().scrollTop).toBe(1900); + }); + it("renders children inside the scroll container, pill hidden, scrolled to bottom on mount", () => { render(); const el = scroller(); diff --git a/ui/src/components/task-chat/TaskMessageScroller.tsx b/ui/src/components/task-chat/TaskMessageScroller.tsx index 6c245df8ffc..5c5501fe29d 100644 --- a/ui/src/components/task-chat/TaskMessageScroller.tsx +++ b/ui/src/components/task-chat/TaskMessageScroller.tsx @@ -116,12 +116,17 @@ export function TaskMessageScroller({ children, contentKey, className }: TaskMes return true; }, [scrollToBottom]); - const rememberAnchor = useCallback(() => { + const rememberAnchor = useCallback((capture = true) => { const el = ref.current; if (!el || el.clientHeight === 0) return; - const rect = el.getBoundingClientRect(); - anchorRef.current = readThreadScrollAnchor(el, rect.top, rect.bottom); - if (initialPositionApplied.current) navigation.remember(el.scrollTop, anchorRef.current); + if (pinnedRef.current) { + // Following the bottom needs only an offset, not a scan of every old row. + anchorRef.current = null; + } else if (capture) { + const rect = el.getBoundingClientRect(); + anchorRef.current = readThreadScrollAnchor(el, rect.top, rect.bottom); + } + if (initialPositionApplied.current) navigation.remember(el.scrollTop, anchorRef.current, pinnedRef.current); }, [navigation.key, navigation.hash, navigation.ready]); const reconcileContent = useCallback(() => { @@ -140,18 +145,20 @@ export function TaskMessageScroller({ children, contentKey, className }: TaskMes const delta = threadScrollAnchorDelta(el, anchorRef.current, el.getBoundingClientRect().top); if (delta) el.scrollTop += delta; } - rememberAnchor(); + // Compensation preserves the existing reading anchor. Recapture only when + // the reader scrolls or navigation selects a different position. + rememberAnchor(false); }, [rememberAnchor, scrollToBottom, hidePill]); const handleScroll = useCallback(() => { if (!ref.current?.clientHeight) return; - rememberAnchor(); showScrollbarWhileScrolling(); // A growing composer shrinks this viewport. Some browsers dispatch the // resulting scroll event before ResizeObserver, so preserve the previous // pinned state here instead of mistaking the layout change for a user // scroll away from the bottom. if (followViewportResize()) { + rememberAnchor(false); hidePill(); return; } @@ -164,9 +171,11 @@ export function TaskMessageScroller({ children, contentKey, className }: TaskMes pinnedRef.current = true; hidePill(); } + rememberAnchor(); return; } pinnedRef.current = pinned; + rememberAnchor(); if (pinned) hidePill(); else showPill(); }, [ diff --git a/ui/src/components/task-chat/scroll-navigation.tsx b/ui/src/components/task-chat/scroll-navigation.tsx index 893daac0aab..03eee4af17c 100644 --- a/ui/src/components/task-chat/scroll-navigation.tsx +++ b/ui/src/components/task-chat/scroll-navigation.tsx @@ -16,7 +16,7 @@ export function taskChatScrollEntry(location: { key: string; hash: string; pathn // Scoped to browser-history entries, not issues: opening the same task from a // new Inbox click starts at latest, while Back restores the previous reading. -const positions = new Map(); +const positions = new Map(); export function useTaskChatScrollNavigation() { const navigation = useContext(TaskChatScrollNavigation); @@ -32,6 +32,9 @@ export function useTaskChatScrollNavigation() { if (!navigation) return null; if (navigation.restore && positionKey && positions.has(positionKey)) { const saved = positions.get(positionKey)!; + // A saved pixel offset goes stale when rotation changes text wrapping. + // Bottom-follow needs no row measurements, even across navigation. + if (saved.followBottom) return Math.max(0, root.scrollHeight - root.clientHeight); if (saved.anchor && [...root.querySelectorAll("[data-thread-anchor]")].some((row) => row.dataset.threadAnchor === saved.anchor?.id)) { return scrollTop + threadScrollAnchorDelta(root, saved.anchor, viewportTop); } @@ -45,9 +48,9 @@ export function useTaskChatScrollNavigation() { } return null; }, - remember(top: number, anchor: ThreadScrollAnchor | null) { + remember(top: number, anchor: ThreadScrollAnchor | null, followBottom: boolean) { if (!positionKey || !ready) return; - positions.set(positionKey, { top, anchor }); + positions.set(positionKey, { top, anchor, followBottom }); if (positions.size > 100) positions.delete(positions.keys().next().value!); }, }; diff --git a/ui/src/components/task-chat/useWindowAutoFollow.test.tsx b/ui/src/components/task-chat/useWindowAutoFollow.test.tsx deleted file mode 100644 index 4cfde316a28..00000000000 --- a/ui/src/components/task-chat/useWindowAutoFollow.test.tsx +++ /dev/null @@ -1,263 +0,0 @@ -// @vitest-environment jsdom - -import { flushSync } from "react-dom"; -import { createRoot, type Root } from "react-dom/client"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { useWindowAutoFollow } from "./useWindowAutoFollow"; -import { TaskChatScrollNavigation } from "./scroll-navigation"; - -function Host({ contentKey, enabled }: { contentKey: unknown; enabled: boolean }) { - useWindowAutoFollow(contentKey, enabled); - return
thread
; -} - -/** - * jsdom has no layout: document scroll geometry is faked on the scrolling - * element, window.scrollTo is stubbed to record calls and update scrollY. - */ -describe("useWindowAutoFollow", () => { - let container: HTMLDivElement; - let root: Root; - let scrollToCalls: number[]; - - function fakeWindowGeometry({ scrollHeight = 2000, innerHeight = 800 } = {}) { - const el = document.scrollingElement ?? document.documentElement; - let currentScrollHeight = scrollHeight; - Object.defineProperty(el, "scrollHeight", { - get: () => currentScrollHeight, - configurable: true, - }); - Object.defineProperty(window, "innerHeight", { value: innerHeight, configurable: true }); - return { - setScrollHeight(value: number) { - currentScrollHeight = value; - }, - }; - } - - function setWindowScrollY(y: number) { - Object.defineProperty(window, "scrollY", { value: y, configurable: true }); - } - - function render(contentKey: unknown, enabled = true) { - flushSync(() => { - root.render(); - }); - } - - /** Fire a window scroll event and flush the continuous-priority update. */ - function scrollWindowTo(y: number) { - setWindowScrollY(y); - window.dispatchEvent(new Event("scroll")); - return new Promise((resolve) => { - setTimeout(resolve); - }); - } - - /** Wait out the mount effect's rAF re-follow so it can't leak into asserts. */ - function flushRaf() { - return new Promise((resolve) => { - requestAnimationFrame(() => setTimeout(resolve)); - }); - } - - beforeEach(() => { - container = document.createElement("div"); - document.body.appendChild(container); - root = createRoot(container); - scrollToCalls = []; - fakeWindowGeometry(); - setWindowScrollY(0); - vi.stubGlobal("scrollTo", (options?: ScrollToOptions | number) => { - const top = typeof options === "number" ? options : (options?.top ?? 0); - scrollToCalls.push(top); - // Mirror the browser: a follow to the bottom leaves the window pinned. - setWindowScrollY(Math.max(0, top - window.innerHeight)); - }); - }); - - afterEach(() => { - flushSync(() => root.unmount()); - container.remove(); - vi.unstubAllGlobals(); - }); - - it("reapplies same-task targets and POP positions on mobile without remounting", async () => { - vi.stubGlobal("scrollTo", (options: ScrollToOptions) => setWindowScrollY(Math.min(1200, options.top ?? 0))); - function MobileThread() { - useWindowAutoFollow("unchanged", true); - return
{[100, 500].map((top, index) => ( -
{ - if (node) node.getBoundingClientRect = () => ({ top: top - window.scrollY, bottom: top + 100 - window.scrollY, height: 100 } as DOMRect); - }}>Comment {index}
- ))}
; - } - const navigate = (key: string, hash: string, restore = false) => flushSync(() => root.render( - , - )); - navigate("mobile-entry-one", "#mobile-comment-0"); - const thread = container.firstElementChild; - expect(window.scrollY).toBe(100); - await scrollWindowTo(150); - navigate("mobile-entry-two", "#mobile-comment-1"); - expect(container.firstElementChild).toBe(thread); - expect(window.scrollY).toBe(500); - navigate("mobile-entry-one", "#mobile-comment-0", true); - expect(window.scrollY).toBe(150); - navigate("mobile-entry-one", "#mobile-comment-1", true); - expect(window.scrollY).toBe(500); - }); - - it("owns browser restoration only while the mobile thread is enabled", () => { - window.history.scrollRestoration = "auto"; - render(0); - expect(window.history.scrollRestoration).toBe("manual"); - render(0, false); - expect(window.history.scrollRestoration).toBe("auto"); - }); - - it("scrolls the window to the bottom on mount", () => { - render(0); - expect(scrollToCalls).toContain(2000); - }); - - it("follows content growth while pinned to the bottom", async () => { - render(0); - await flushRaf(); - await scrollWindowTo(1200); // pinned: 2000 - 1200 - 800 = 0 - scrollToCalls = []; - render(1); - expect(scrollToCalls).toContain(2000); - }); - - it("holds position when the user has scrolled up", async () => { - render(0); - await flushRaf(); - await scrollWindowTo(100); // far from the bottom - scrollToCalls = []; - render(1); - expect(scrollToCalls).toHaveLength(0); - }); - - it("follows document growth while pinned on mobile", async () => { - let triggerResize = () => {}; - vi.stubGlobal( - "ResizeObserver", - class { - constructor(callback: ResizeObserverCallback) { - triggerResize = () => - callback([], this as unknown as ResizeObserver); - } - observe() {} - disconnect() {} - }, - ); - const geometry = fakeWindowGeometry(); - render(0); - await flushRaf(); - await scrollWindowTo(1200); - scrollToCalls = []; - - geometry.setScrollHeight(2300); - triggerResize(); - - expect(scrollToCalls).toContain(2300); - }); - - it("holds document position through mobile composer growth when scrolled up", async () => { - let triggerResize = () => {}; - vi.stubGlobal( - "ResizeObserver", - class { - constructor(callback: ResizeObserverCallback) { - triggerResize = () => - callback([], this as unknown as ResizeObserver); - } - observe() {} - disconnect() {} - }, - ); - const geometry = fakeWindowGeometry(); - render(0); - await flushRaf(); - await scrollWindowTo(100); - scrollToCalls = []; - - geometry.setScrollHeight(2300); - triggerResize(); - - expect(scrollToCalls).toHaveLength(0); - }); - - it("follows a late thread resize without a React content update or body resize", async () => { - const observed = new Set(); - let resizeThread = () => {}; - vi.stubGlobal("ResizeObserver", class { - constructor(callback: ResizeObserverCallback) { - resizeThread = () => { - const thread = container.querySelector('[data-testid="task-chat-thread"]')!; - if (observed.has(thread)) callback([{ target: thread } as ResizeObserverEntry], this as unknown as ResizeObserver); - }; - } - observe(element: Element) { observed.add(element); } - disconnect() { observed.clear(); } - }); - const geometry = fakeWindowGeometry(); - render(0); - scrollToCalls = []; - - geometry.setScrollHeight(2400); - resizeThread(); - - expect(scrollToCalls).toContain(2400); - expect(window.scrollY).toBe(1600); - }); - - it("keeps following when a scroll event arrives after growth before reconciliation", async () => { - const geometry = fakeWindowGeometry(); - render(0); - - geometry.setScrollHeight(2400); - await scrollWindowTo(1200); // Still at the old bottom; the user did not move up. - scrollToCalls = []; - render(1); - - expect(scrollToCalls).toContain(2400); - }); - - it("holds an upward user scroll during growth and resumes after returning to the bottom", async () => { - const geometry = fakeWindowGeometry(); - render(0); - - geometry.setScrollHeight(2400); - await scrollWindowTo(900); - scrollToCalls = []; - render(1); - expect(scrollToCalls).toHaveLength(0); - - await scrollWindowTo(1600); - geometry.setScrollHeight(2700); - render(2); - expect(scrollToCalls).toContain(2700); - }); - - it("follows viewport resizing while pinned and holds it while reading older output", async () => { - render(0); - Object.defineProperty(window, "innerHeight", { value: 600, configurable: true }); - window.dispatchEvent(new Event("resize")); - expect(window.scrollY).toBe(1400); - - await scrollWindowTo(500); - scrollToCalls = []; - Object.defineProperty(window, "innerHeight", { value: 700, configurable: true }); - window.dispatchEvent(new Event("resize")); - expect(scrollToCalls).toHaveLength(0); - }); - - it("does nothing when disabled", () => { - render(0, false); - expect(scrollToCalls).toHaveLength(0); - render(1, false); - expect(scrollToCalls).toHaveLength(0); - }); -}); diff --git a/ui/src/components/task-chat/useWindowAutoFollow.ts b/ui/src/components/task-chat/useWindowAutoFollow.ts deleted file mode 100644 index d971e94e42f..00000000000 --- a/ui/src/components/task-chat/useWindowAutoFollow.ts +++ /dev/null @@ -1,136 +0,0 @@ -import { readThreadScrollAnchor, threadScrollAnchorDelta, type ThreadScrollAnchor } from "./scroll-anchor"; -import { useEffect, useLayoutEffect, useRef } from "react"; -import { useTaskChatScrollNavigation } from "./scroll-navigation"; - -const PIN_THRESHOLD_PX = 48; - -function scrollingElement(): Element | null { - return document.scrollingElement ?? document.documentElement; -} - -function windowPinned(): boolean { - const el = scrollingElement(); - if (!el) return true; - return el.scrollHeight - window.scrollY - window.innerHeight <= PIN_THRESHOLD_PX; -} - -function scrollWindowToBottom(): void { - const el = scrollingElement(); - if (!el) return; - window.scrollTo({ top: el.scrollHeight, left: 0, behavior: "auto" }); -} - -/** - * Window-scroll counterpart of TaskMessageScroller's auto-follow rule, for the - * mobile document-flow thread (the mobile app shell scrolls the document, not - * an inner viewport — see Layout.tsx). While the window is pinned to the - * bottom (within a small threshold) content growth follows with INSTANT - * scroll; once the user scrolls up we hold their position. - * - * The conversation enables this hook after navigation has settled and before - * its coordinated reveal, so initial positioning happens before paint. - */ -export function useWindowAutoFollow(contentKey: unknown, enabled: boolean): void { - const pinnedRef = useRef(true); - const lastScrollYRef = useRef(window.scrollY); - const navigation = useTaskChatScrollNavigation(); - const initialPositionApplied = useRef(false); - const appliedNavigation = useRef({ key: navigation.key, hash: navigation.hash }); - const anchorRef = useRef(null); - const rememberAnchor = () => { - lastScrollYRef.current = window.scrollY; - const root = document.querySelector('[data-testid="task-chat-thread"]'); - if (root) anchorRef.current = readThreadScrollAnchor(root, 0, window.innerHeight); - if (initialPositionApplied.current) navigation.remember(window.scrollY, anchorRef.current); - }; - const reconcile = () => { - if (pinnedRef.current) scrollWindowToBottom(); - else { - const root = document.querySelector('[data-testid="task-chat-thread"]'); - if (root) { - const delta = threadScrollAnchorDelta(root, anchorRef.current, 0); - if (delta) window.scrollTo({ top: window.scrollY + delta, behavior: "auto" }); - } - } - rememberAnchor(); - }; - - useEffect(() => { - if (!enabled) return; - const onScroll = () => { - // Content or viewport growth can leave a gap below the old bottom before - // ResizeObserver runs. Only an upward scroll gives up existing follow - // intent; a scroll event at the same position must not strand the output. - pinnedRef.current = windowPinned() || (pinnedRef.current && window.scrollY >= lastScrollYRef.current); - rememberAnchor(); - }; - window.addEventListener("scroll", onScroll, { passive: true }); - window.addEventListener("resize", reconcile); - return () => { - window.removeEventListener("scroll", onScroll); - window.removeEventListener("resize", reconcile); - }; - }, [enabled, navigation.key, navigation.hash, navigation.ready]); - - useLayoutEffect(() => { - if (!enabled || typeof ResizeObserver === "undefined") return; - let previousScrollHeight = scrollingElement()?.scrollHeight ?? null; - const observer = new ResizeObserver(() => { - const nextScrollHeight = scrollingElement()?.scrollHeight ?? null; - if ( - nextScrollHeight == null || - nextScrollHeight === previousScrollHeight - ) { - return; - } - previousScrollHeight = nextScrollHeight; - reconcile(); - }); - // The mobile body is viewport-sized even when its contents overflow. Watch - // the thread box too, so streamed rows, images and expanding tools follow - // after their own layout changes without needing a new parent content key. - observer.observe(document.body); - const thread = document.querySelector('[data-testid="task-chat-thread"]'); - if (thread) observer.observe(thread); - return () => observer.disconnect(); - }, [enabled, navigation.key, navigation.hash, navigation.ready]); - - // Follow new content only when already pinned; otherwise hold position. - useLayoutEffect(() => { - if (!enabled) return; - if (appliedNavigation.current.key !== navigation.key || appliedNavigation.current.hash !== navigation.hash) { - appliedNavigation.current = { key: navigation.key, hash: navigation.hash }; - initialPositionApplied.current = false; - } - if (navigation.ready && !initialPositionApplied.current) { - const root = document.querySelector('[data-testid="task-chat-thread"]'); - const top = root ? navigation.initialPosition(root, 0, window.scrollY) : null; - if (top !== null) { - window.scrollTo({ top, behavior: "auto" }); - pinnedRef.current = windowPinned(); - rememberAnchor(); - } - initialPositionApplied.current = true; - } - reconcile(); - }, [contentKey, enabled, navigation.key, navigation.hash, navigation.ready]); - - useLayoutEffect(() => { - if (!enabled) return; - // One owner for document-flow compensation, as on the desktop viewport. - const previousRestoration = window.history.scrollRestoration; - window.history.scrollRestoration = "manual"; - document.documentElement.classList.add("task-chat-window-scroll"); - return () => { - document.documentElement.classList.remove("task-chat-window-scroll"); - window.history.scrollRestoration = previousRestoration; - }; - }, [enabled]); -} - -/** Mount below TaskChatScrollReady so mobile navigation also waits for targets - * fetched after the conversation's first reveal. */ -export function TaskChatWindowScroll({ contentKey, enabled }: { contentKey: unknown; enabled: boolean }) { - useWindowAutoFollow(contentKey, enabled); - return null; -} diff --git a/ui/src/components/transcript/useLiveRunTranscripts.test.tsx b/ui/src/components/transcript/useLiveRunTranscripts.test.tsx index df3419993d1..0636b1f6267 100644 --- a/ui/src/components/transcript/useLiveRunTranscripts.test.tsx +++ b/ui/src/components/transcript/useLiveRunTranscripts.test.tsx @@ -131,17 +131,18 @@ describe("useLiveRunTranscripts", () => { }); it("expands a compact log tail without hiding or duplicating the latest output", async () => { + let truncatedRunIds: ReadonlySet = new Set(); const row = (seq: number, chunk: string) => JSON.stringify({ seq, chunk, ts: `2026-10-24T00:00:0${seq}Z`, stream: "stdout" }); const result = (content: string) => ({ runId: "run-1", store: "memory", logRef: "log-1", content, nextOffset: 100_000 }); let resolveExpanded!: (value: ReturnType) => void; logMock.mockResolvedValueOnce(result(`${row(2, "Latest answer")}\n`)); logMock.mockImplementationOnce(() => new Promise(resolve => { resolveExpanded = resolve; })); function Harness({ limit, bytes = 100_000 }: { limit: number; bytes?: number }) { - useLiveRunTranscripts({ + ({ truncatedRunIds } = useLiveRunTranscripts({ runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logBytes: bytes }], enableRealtimeUpdates: false, logReadLimitBytes: limit, - }); + })); return null; } const root = createRoot(document.createElement("div")); @@ -149,11 +150,13 @@ describe("useLiveRunTranscripts", () => { try { await act(async () => root.render()); expect(logMock).toHaveBeenLastCalledWith("run-1", 68_000, 32_000, expect.objectContaining({ alignStartToRecord: true })); + expect(truncatedRunIds.has("run-1")).toBe(true); expect(chunks()).toEqual([expect.objectContaining({ chunk: "Latest answer" })]); await act(async () => root.render()); expect(logMock).toHaveBeenLastCalledWith("run-1", 0, 256_000, expect.anything()); expect(chunks()).toEqual([expect.objectContaining({ chunk: "Latest answer" })]); await act(async () => resolveExpanded(result(`${row(1, "Earlier activity")}\n${row(2, "Latest answer")}\n`))); + expect(truncatedRunIds.has("run-1")).toBe(false); expect(chunks()).toEqual([ expect.objectContaining({ chunk: "Earlier activity" }), expect.objectContaining({ chunk: "Latest answer" }), @@ -165,6 +168,30 @@ describe("useLiveRunTranscripts", () => { } }); + it("does not report omitted output when a run grows beyond the window after reading from the start", async () => { + let truncatedRunIds: ReadonlySet = new Set(); + const row = (seq: number) => `${JSON.stringify({ seq, chunk: `Output ${seq}`, stream: "stdout", ts: "2026-10-24T00:00:00Z" })}\n`; + const response = { runId: "run-1", store: "memory", logRef: "log-1" }; + logMock.mockResolvedValueOnce({ ...response, content: row(1), nextOffset: 100 }); + logMock.mockResolvedValueOnce({ ...response, content: row(2), nextOffset: 100_000 }); + function Harness({ bytes }: { bytes: number }) { + ({ truncatedRunIds } = useLiveRunTranscripts({ + runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logBytes: bytes }], + enableRealtimeUpdates: false, logReadLimitBytes: 32_000, + })); + return null; + } + const root = createRoot(document.createElement("div")); + try { + await act(async () => root.render()); + expect(logMock).toHaveBeenLastCalledWith("run-1", 0, 32_000, expect.anything()); + expect(truncatedRunIds.has("run-1")).toBe(false); + await act(async () => root.render()); + expect(logMock).toHaveBeenLastCalledWith("run-1", 100, 32_000, expect.anything()); + expect(truncatedRunIds.has("run-1")).toBe(false); + } finally { await act(async () => root.unmount()); } + }); + it("keeps record alignment pending when an expanded log read needs retry", async () => { const row = (seq: number, chunk: string) => JSON.stringify({ seq, chunk, ts: `2026-10-24T00:00:0${seq}Z`, stream: "stdout" }); const result = (content: string) => ({ runId: "run-1", store: "memory", logRef: "log-1", content, nextOffset: 500_000 }); diff --git a/ui/src/components/transcript/useLiveRunTranscripts.ts b/ui/src/components/transcript/useLiveRunTranscripts.ts index 978f0766479..134e88e766d 100644 --- a/ui/src/components/transcript/useLiveRunTranscripts.ts +++ b/ui/src/components/transcript/useLiveRunTranscripts.ts @@ -144,6 +144,7 @@ export function useLiveRunTranscripts({ const pendingLogRowsByRunRef = useRef(new Map()); const logOffsetByRunRef = useRef(new Map()); const logReadLimitByRunRef = useRef(new Map()); + const [truncatedRunIds, setTruncatedRunIds] = useState>(new Set()); const missingTerminalLogRunIdsRef = useRef(new Set()); // PAP-462 B3: buffered runs that dropped out of the `runs` list, mapped to the // wall-clock deadline (ms) after which their buffer may be pruned. A run still @@ -260,6 +261,10 @@ export function useLiveRunTranscripts({ const next = new Map([...previous].filter(([id]) => retainedRunIds.has(id))); return next.size === previous.size ? previous : next; }); + setTruncatedRunIds((previous) => { + const next = new Set([...previous].filter((id) => retainedRunIds.has(id))); + return next.size === previous.size ? previous : next; + }); for (const key of pendingLogRowsByRunRef.current.keys()) { const runId = key.replace(/:records$/, ""); @@ -340,6 +345,15 @@ export function useLiveRunTranscripts({ // Commit the expanded window only after success so retries still align // its first record and failed reads leave the previous parser intact. + if (previousReadLimit === undefined || rewinding) { + setTruncatedRunIds((previous) => { + if (previous.has(run.id) === (offset > 0)) return previous; + const next = new Set(previous); + if (offset > 0) next.add(run.id); + else next.delete(run.id); + return next; + }); + } logReadLimitByRunRef.current.set(run.id, logReadLimitBytes); if (rewinding) { pendingLogRowsByRunRef.current.delete(run.id); @@ -596,7 +610,7 @@ export function useLiveRunTranscripts({ return { transcriptByRun, - hydratedRunIds, errorsByRun, retry, + hydratedRunIds, truncatedRunIds, errorsByRun, retry, isInitialHydrating: normalizedRuns.some((run) => canReadPersistedLog(run) && !hydratedRunIds.has(run.id)), hasOutputForRun(runId: string) { return (chunksByRun.get(runId)?.length ?? 0) > 0 || runById.get(runId)?.hasStoredOutput === true; diff --git a/ui/src/hooks/useMobileVisualViewport.test.tsx b/ui/src/hooks/useMobileVisualViewport.test.tsx new file mode 100644 index 00000000000..8706820479c --- /dev/null +++ b/ui/src/hooks/useMobileVisualViewport.test.tsx @@ -0,0 +1,116 @@ +// @vitest-environment jsdom +import { act } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import { useMobileVisualViewport } from "./useMobileVisualViewport"; + +function Host({ mobile = true, conversation = true }: { mobile?: boolean; conversation?: boolean }) { + const ref = useMobileVisualViewport(mobile); + return
Chat shell
; +} + +let root: Root; +let container: HTMLDivElement; +let viewport: EventTarget & { height: number; offsetTop: number; scale: number }; +beforeEach(() => { + vi.stubGlobal("IS_REACT_ACT_ENVIRONMENT", true); + vi.stubGlobal("innerHeight", 695); + Object.defineProperty(document.documentElement, "clientHeight", { configurable: true, value: 695 }); + viewport = Object.assign(new EventTarget(), { height: 695, offsetTop: 0, scale: 1 }); + vi.stubGlobal("visualViewport", viewport); + container = document.createElement("div"); + document.body.append(container); + root = createRoot(container); +}); +afterEach(() => { + act(() => root.unmount()); + container.remove(); + Reflect.deleteProperty(document.documentElement, "clientHeight"); + document.documentElement.scrollTop = 0; + vi.unstubAllGlobals(); +}); + +it("resizes the shell from the visible height when iOS innerHeight shrinks too", () => { + act(() => root.render()); + const shell = container.firstElementChild as HTMLElement; + act(() => { + vi.stubGlobal("innerHeight", 358); + viewport.height = 358; + viewport.offsetTop = 337; + viewport.dispatchEvent(new Event("resize")); + }); + expect(shell.style.getPropertyValue("--mobile-chat-viewport-height")).toBe("358px"); + expect(shell.dataset.keyboardOpen).toBe("true"); + // Panning must not translate or resize the shell a second time. + act(() => { viewport.offsetTop = 185; viewport.dispatchEvent(new Event("scroll")); }); + expect(shell.style.top).toBe(""); + expect(shell.style.transform).toBe(""); + expect(shell.style.getPropertyValue("--mobile-chat-viewport-height")).toBe("358px"); + act(() => { viewport.height = 695; viewport.offsetTop = 0; viewport.dispatchEvent(new Event("resize")); }); + expect(shell.style.getPropertyValue("--mobile-chat-viewport-height")).toBe("695px"); + expect(shell.dataset.keyboardOpen).toBeUndefined(); +}); + +it("preserves pinch zoom and leaves desktop sizing to CSS", () => { + act(() => root.render()); + const shell = container.firstElementChild as HTMLElement; + act(() => { viewport.scale = 2; viewport.height = 350; viewport.dispatchEvent(new Event("resize")); }); + expect(shell.style.getPropertyValue("--mobile-chat-viewport-height")).toBe("695px"); + act(() => root.render()); + expect(shell.style.getPropertyValue("--mobile-chat-viewport-height")).toBe(""); + expect(shell.dataset.keyboardOpen).toBeUndefined(); +}); + +it("does not invalidate shell styles for typing, scrolling, or unchanged viewport events", () => { + act(() => root.render()); + const shell = container.firstElementChild as HTMLElement; + const observer = new MutationObserver(() => {}); + observer.observe(shell, { attributes: true }); + act(() => { + for (let i = 0; i < 30; i++) { + shell.dispatchEvent(new Event("input", { bubbles: true })); + viewport.dispatchEvent(new Event("scroll")); + viewport.dispatchEvent(new Event("resize")); + } + }); + expect(observer.takeRecords()).toHaveLength(0); + observer.disconnect(); +}); + +it("rejects Safari's stale document focus scroll after the keyboard has resized the conversation", () => { + act(() => root.render()); + act(() => { + viewport.height = 358; + // Some openings pan before the resize event; others pan one frame later. + document.documentElement.scrollTop = 337; + viewport.dispatchEvent(new Event("resize")); + }); + expect(document.documentElement.scrollTop).toBe(0); + document.documentElement.scrollTop = 337; + document.dispatchEvent(new Event("scroll")); + expect(document.documentElement.scrollTop).toBe(0); +}); + +it("leaves ordinary pages and inner message scrolling alone", () => { + viewport.height = 358; + act(() => root.render()); + document.documentElement.scrollTop = 337; + document.dispatchEvent(new Event("scroll")); + expect(document.documentElement.scrollTop).toBe(337); + act(() => root.render()); + const messages = container.querySelector('[data-task-chat-shell]') as HTMLElement; + messages.scrollTop = 250; + messages.dispatchEvent(new Event("scroll")); + expect(messages.scrollTop).toBe(250); +}); + +it("does not suppress zoom panning when the keyboard was already open", () => { + viewport.height = 358; + act(() => root.render()); + viewport.scale = 2; + viewport.height = 179; + viewport.dispatchEvent(new Event("resize")); + document.documentElement.scrollTop = 120; + document.dispatchEvent(new Event("scroll")); + expect(document.documentElement.scrollTop).toBe(120); +}); diff --git a/ui/src/hooks/useMobileVisualViewport.ts b/ui/src/hooks/useMobileVisualViewport.ts new file mode 100644 index 00000000000..9d975006d17 --- /dev/null +++ b/ui/src/hooks/useMobileVisualViewport.ts @@ -0,0 +1,47 @@ +import { useLayoutEffect, useRef } from "react"; + +/** Size the mobile conversation shell, never translate it with Safari's pan. + * CSS owns the fixed shell and its flex layout; messages own their scrolling. */ +export function useMobileVisualViewport(enabled: boolean) { + const ref = useRef(null); + useLayoutEffect(() => { + const shell = ref.current; + const viewport = window.visualViewport; + if (!enabled || !shell || !viewport) return; + const restoreDocumentPosition = () => { + if (viewport.scale !== 1 || shell.dataset.keyboardOpen !== "true" || !shell.querySelector("[data-task-chat-shell]")) return; + const page = document.scrollingElement ?? document.documentElement; + if (page.scrollTop !== 0) page.scrollTop = 0; + }; + const update = () => { + if (viewport.scale !== 1 || !Number.isFinite(viewport.height) || viewport.height <= 0) return; + // Read geometry before writing styles, avoiding a forced layout after our + // own height update. Nothing here subscribes to input or thread updates. + const keyboardOpen = document.documentElement.clientHeight - viewport.height > 120; + shell.style.setProperty("--mobile-chat-viewport-height", `${viewport.height}px`); + // innerHeight can shrink with visualViewport on iOS, so it is not the + // keyboard-closed baseline. The layout viewport remains stable. + if (keyboardOpen) { + shell.dataset.keyboardOpen = "true"; + } else { + delete shell.dataset.keyboardOpen; + } + restoreDocumentPosition(); + }; + update(); + viewport.addEventListener("resize", update); + window.addEventListener("resize", update); + // Safari can apply its pre-keyboard focus pan AFTER the resize. The fixed + // document is not a scroll surface. Listen without capture so scrolls in + // the message list/editor never reach this fallback. + document.addEventListener("scroll", restoreDocumentPosition); + return () => { + viewport.removeEventListener("resize", update); + window.removeEventListener("resize", update); + document.removeEventListener("scroll", restoreDocumentPosition); + shell.style.removeProperty("--mobile-chat-viewport-height"); + delete shell.dataset.keyboardOpen; + }; + }, [enabled]); + return ref; +} diff --git a/ui/src/index.css b/ui/src/index.css index a094db7de0c..c831b655bf1 100644 --- a/ui/src/index.css +++ b/ui/src/index.css @@ -583,6 +583,8 @@ leaves 6px below the dock; its pb-1 adds 4px for a 10px card-to-nav gap. Desktop and the classic thread use the safe-area default. */ --tc-composer-bottom: var(--sz-calc-8); + --mobile-chat-viewport-height: 100dvh; + --tc-composer-editor-max-height: min(var(--sz-28dvh), calc(var(--mobile-chat-viewport-height) * 0.4)); --tc-composer-visible-nav-offset: calc(4rem + var(--spacing) * 1.5 + var(--sz-safe-bottom)); /* With the dock's pb-1, leave 12px below the card when the nav hides. */ --tc-composer-hidden-nav-offset: calc(var(--sz-safe-bottom) + var(--spacing) * 2); @@ -887,6 +889,62 @@ select:where(:not([multiple]):not(.appearance-none)):where(:not([size]), [size=" transition: bottom var(--mobile-nav-motion-duration, var(--motion-mobile-nav-enter)) var(--mobile-nav-motion-ease, var(--motion-ease-out-expo)); } +/* Mobile conversations use the same single message scroller + normal footer as + desktop. Opt in from the rendered task shell, so classic tasks, lists, and + the empty agent-chat picker retain their existing document layout. */ +html:has(.paperclip-app-shell[data-mobile="true"] [data-task-chat-shell]), +html:has(.paperclip-app-shell[data-mobile="true"] [data-task-chat-shell]) body { + position: fixed; + inset: 0; + width: 100%; +} + +html:has(.paperclip-app-shell[data-mobile="true"] [data-task-chat-shell]), +html:has(.paperclip-app-shell[data-mobile="true"] [data-task-chat-shell]) body, +html:has(.paperclip-app-shell[data-mobile="true"] [data-task-chat-shell]) #root { + height: 100%; + overflow: hidden !important; + overscroll-behavior: none; +} + +.paperclip-app-shell[data-mobile="true"]:has([data-task-chat-shell]) { + position: fixed; + inset: 0; + display: flex; + flex-direction: column; + height: var(--mobile-chat-viewport-height); + min-height: 0; + overflow: hidden; +} + +.paperclip-app-shell[data-mobile="true"]:has([data-task-chat-shell]) :is(.app-main-row, .app-content-row) { + display: flex; + flex: 1; + min-height: 0; + overflow: hidden; +} + +.paperclip-app-shell[data-mobile="true"]:has([data-task-chat-shell]) .app-content-column { + flex: 1; + min-height: 0; +} + +.paperclip-app-shell[data-mobile="true"]:has([data-task-chat-shell]) .app-breadcrumb { + flex-shrink: 0; +} + +.paperclip-app-shell[data-mobile="true"]:has([data-task-chat-shell]) #main-content { + display: flex; + flex-direction: column; + min-height: 0; + overflow: hidden; + padding-bottom: var(--tc-composer-visible-nav-offset); +} + +.paperclip-app-shell[data-mobile="true"][data-keyboard-open="true"]:has([data-task-chat-shell]) #main-content { + padding-bottom: 0; +} + @media (prefers-reduced-motion: reduce) { :root { --motion-mobile-nav-enter: 0ms; @@ -3292,10 +3350,6 @@ span.paperclip-mention-chip[data-mention-kind="external-object"] { scrollbar-gutter: stable; } -.task-chat-window-scroll { - overflow-anchor: none; -} - .task-chat-loading-shell .animate-pulse { animation: none; } diff --git a/ui/src/pages/IssueDetail.test.tsx b/ui/src/pages/IssueDetail.test.tsx index 8543d8961aa..3d3726cc97f 100644 --- a/ui/src/pages/IssueDetail.test.tsx +++ b/ui/src/pages/IssueDetail.test.tsx @@ -1684,6 +1684,100 @@ describe("IssueDetail", () => { expect(windowOpen).not.toHaveBeenCalled(); }); + it.each([null, "queued", "running", "succeeded", "failed"])("hides the history disclosure on a fresh task with first run %s", async (status) => { + queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false); + mockIssuesApi.get.mockResolvedValue(createIssue()); + const runs = status ? [{ runId: "first-run", status, agentId: "agent-1", createdAt: new Date().toISOString() }] : []; + mockActivityApi.recentRunsForIssue.mockResolvedValue(runs); + await act(async () => renderIssueView()); + await waitForAssertion(() => { + const props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0]; + expect(mockActivityApi.recentRunsForIssue).toHaveBeenCalled(); + expect(props?.linkedRuns).toMatchObject(runs); + expect(props?.compactHistory).toBe(true); + expect(props?.historyControl).toBeNull(); + }); + expect(mockActivityApi.runsForIssue).not.toHaveBeenCalled(); + expect(mockIssuesApi.listComments).not.toHaveBeenCalled(); + }); + + it("loads full first-run activity from the task menu without an older-message disclosure", async () => { + queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false); + mockIssuesApi.get.mockResolvedValue(createIssue()); + const run = { runId: "first-run", runtimeMode: "native", status: "succeeded", agentId: "agent-1", createdAt: new Date().toISOString() }; + mockActivityApi.recentRunsForIssue.mockResolvedValue([run]); + mockActivityApi.runsForIssue.mockResolvedValue([run]); + await act(async () => renderIssueView()); + await waitForAssertion(() => { + const props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0]; + expect(props?.linkedRuns).toMatchObject([run]); + expect(props?.compactHistory).toBe(true); + expect(props?.historyControl).toBeNull(); + }); + expect(mockActivityApi.runsForIssue).not.toHaveBeenCalled(); + await act(async () => { + container.querySelector('button[aria-label="More task actions"]')!.click(); + }); + const activityButton = Array.from(document.body.querySelectorAll("button")) + .find((button) => button.textContent?.trim() === "Show task activity"); + expect(activityButton).toBeTruthy(); + await act(async () => activityButton!.click()); + await waitForAssertion(() => { + expect(mockActivityApi.runsForIssue).toHaveBeenCalled(); + expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0].compactHistory).toBe(false); + }); + await act(async () => { + container.querySelector('button[aria-label="More task actions"]')!.click(); + }); + expect(Array.from(document.body.querySelectorAll("button")) + .some((button) => button.textContent?.trim() === "Show task activity")).toBe(false); + }); + + it("does not add a history disclosure when a second run is observed while the page stays open", async () => { + queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false); + mockIssuesApi.get.mockResolvedValue(createIssue()); + const run = (runId: string) => ({ runId, status: "succeeded", agentId: "agent-1", createdAt: new Date().toISOString() }); + const first = run("first"), second = run("second"); + mockActivityApi.recentRunsForIssue.mockResolvedValue([first]); + await act(async () => renderIssueView()); + await waitForAssertion(() => expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0].linkedRuns).toMatchObject([first])); + mockActivityApi.recentRunsForIssue.mockResolvedValue([second, first]); + await act(async () => { await queryClient.invalidateQueries({ queryKey: queryKeys.issues.runs("PAP-1") }); }); + await waitForAssertion(() => { + const props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0]; + expect(props?.linkedRuns).toMatchObject([first, second]); + expect(props?.historyControl).toBeNull(); + }); + expect(mockActivityApi.runsForIssue).not.toHaveBeenCalled(); + }); + + it("reveals omitted runs through the history disclosure when there are no older comments", async () => { + queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false); + mockIssuesApi.get.mockResolvedValue(createIssue()); + const run = (runId: string) => ({ runId, status: "succeeded", agentId: "agent-1", createdAt: new Date().toISOString() }); + const latest = run("latest"), older = run("older"), next = run("next"); + mockActivityApi.recentRunsForIssue.mockResolvedValue([latest, older]); + mockActivityApi.runsForIssue.mockResolvedValue([next, latest, older]); + await act(async () => renderIssueView()); + await waitForAssertion(() => { + const props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0]; + expect(props?.linkedRuns).toMatchObject([latest]); + expect(props?.historyControl).toBeTruthy(); + }); + mockActivityApi.recentRunsForIssue.mockResolvedValue([next, latest]); + await act(async () => { await queryClient.invalidateQueries({ queryKey: queryKeys.issues.runs("PAP-1") }); }); + await waitForAssertion(() => expect(mockIssueChatThreadRender.mock.calls.at(-1)?.[0].linkedRuns).toMatchObject([latest, next])); + const disclosure = mockIssueChatThreadRender.mock.calls.at(-1)?.[0].historyControl; + expect(disclosure).toBeTruthy(); + await act(async () => disclosure.props.onClick()); + await waitForAssertion(() => { + const props = mockIssueChatThreadRender.mock.calls.at(-1)?.[0]; + expect(props?.linkedRuns).toMatchObject([next, latest, older]); + expect(props?.historyControl).toBeNull(); + expect(props?.compactHistory).toBe(false); + }); + }); + it("reveals a cold task and recent exchange while full detail and older history are stalled", async () => { queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false); const stalled = createDeferred(); diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index 144b2e10382..33d0cf9d6cb 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -1203,6 +1203,11 @@ function InboxMobileToolbar({ ); } +interface RecentIssueRunWindow { + runs: RunForIssue[]; + hasEarlierRuns: boolean; +} + type IssueDetailChatTabProps = { executionBlocked?: boolean; interruption?: Issue["interruption"]; @@ -1620,37 +1625,48 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ const hasLiveRuns = liveRunCount > 0 || !!resolvedActiveRun; const linkedRunsKey = historyExpanded ? queryKeys.issues.runs(issueQueryRef) - : [...queryKeys.issues.runs(issueQueryRef), "recent"]; + : [...queryKeys.issues.runs(issueQueryRef), "recent-window"]; const { - data: linkedRuns, + data: linkedRunHistory, isPending: linkedRunsPending, isError: linkedRunsError, refetch: refetchLinkedRuns, - } = useQuery({ + } = useQuery({ queryKey: linkedRunsKey, queryFn: async () => { if (historyExpanded) return activityApi.runsForIssue(issueQueryRef); let recent = await activityApi.recentRunsForIssue(issueQueryRef); - let previous = queryClient.getQueryData(linkedRunsKey) ?? []; + const previousWindow = queryClient.getQueryData(linkedRunsKey); + let previous = previousWindow?.runs ?? []; + let reconciledFullHistory = false; if (previous.some((run) => (run.status === "running" || run.status === "queued") && !recent.some((next) => next.runId === run.runId), )) { // After a reconnect, several turns may have completed beyond the small // window. Reconcile watched runs once against authoritative summaries. recent = await activityApi.runsForIssue(issueQueryRef); + reconciledFullHistory = true; const existingRunIds = new Set(recent.map((run) => run.runId)); previous = previous.filter((run) => existingRunIds.has(run.runId)); } // The second row refreshes a just-finished predecessor. Start with only // the latest run, then keep every run observed while this task is open. const visible = new Set([...previous.map((run) => run.runId), ...recent.slice(0, 1).map((run) => run.runId)]); - return [...new Map([...previous, ...recent].filter((run) => visible.has(run.runId)).map((run) => [run.runId, run])).values()]; + return { + runs: [...new Map([...previous, ...recent].filter((run) => visible.has(run.runId)).map((run) => [run.runId, run])).values()], + // A first run is already visible. The disclosure needs evidence of an + // omitted run, and must retain that evidence as later turns arrive. + hasEarlierRuns: recent.some((run) => !visible.has(run.runId)) + || (!reconciledFullHistory && recent.length >= 2 && previousWindow?.hasEarlierRuns === true), + }; }, enabled: !!issueId, refetchInterval: hasLiveRuns || issueStatus === "in_progress" ? 1000 : false, - placeholderData: historyExpanded ? queryClient.getQueryData([...queryKeys.issues.runs(issueQueryRef), "recent"]) : undefined, + placeholderData: historyExpanded ? queryClient.getQueryData([...queryKeys.issues.runs(issueQueryRef), "recent-window"])?.runs : undefined, }); + const linkedRuns = Array.isArray(linkedRunHistory) ? linkedRunHistory : linkedRunHistory?.runs; + const hasEarlierRuns = !Array.isArray(linkedRunHistory) && linkedRunHistory?.hasEarlierRuns === true; const resolvedActivity = useMemo( () => [...new Map([...(activity ?? []), ...(inputFacts.data ?? [])].map((event) => [event.id, event])).values()], [activity, inputFacts.data], @@ -2560,7 +2576,8 @@ const IssueDetailChatTab = memo(function IssueDetailChatTab({ 0) ? ( + onExpandHistory, + historyControl: !historyExpanded && (hasEarlierMessages || hasEarlierRuns) ? ( @@ -7661,6 +7678,18 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS + {taskChatShellEnabled && !historyExpanded ? ( + + ) : null} {streamlinedTaskDetailEnabled ? ( <>
; } @@ -56,7 +54,7 @@ const meta = { ], "implementation": "production", "boundary": "fixtures", - "notes": "The filter uses one lowercase label without helper text. All chat-channel tasks count as bot-created; the focused visibility stories cover human-initiated channel work." + "notes": "Production mobile viewport sizing, message scroller, and composer with bounded synthetic streaming. Physical iPhone keyboard acceptance is required." }, }, globals: { theme: "light", viewport: { value: "mobile", isRotated: false } },