From c91f7a6e5f53fc6fe5fc4388eb1a717e15a5e6fe Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 16 Oct 2026 13:35:26 -0500 Subject: [PATCH] perf(runner): overlap verified sandbox launch assets (#16585) Native sandbox launch preparation stages fresh harness files and immutable runtime context in sequence. Overlap them only under the verified concurrent-sync capability, settle every started transfer before launch or cleanup, and preserve serial SSH behavior and error priority. A matched real-Daytona comparison reduces staging from 1,552 ms to 1,017 ms. The larger active-run difference includes model variation and is not attributed to this change. Add an explicit single-attempt diagnostic and synchronize stale Pi evidence metadata with its shipped profile. Validation: 764 native runner/tracing cases, 2,764 support cases, repository typecheck/build, fresh exact-head full CI/E2E/security and Apex 5/5 pass. Observability remains a silent no-op when no endpoint is configured. Co-Authored-By: Paperclip --- .../2026-10-16-native-latency-diagnostic.md | 107 +++++++++++ .../native-runtime/native-session-executor.ts | 116 ++--------- .../remote-launch-assets.test.ts | 181 ++++++++++++++++++ .../native-runtime/remote-launch-assets.ts | 96 ++++++++++ tests/runner-e2e/catalog.test.ts | 6 +- tests/runner-e2e/catalog.ts | 25 +++ .../codie-latency-diagnostic.test.ts | 24 +++ tests/runner-e2e/pi-controls-evidence.test.ts | 8 +- 8 files changed, 459 insertions(+), 104 deletions(-) create mode 100644 doc/plans/2026-10-16-native-latency-diagnostic.md create mode 100644 server/src/services/native-runtime/remote-launch-assets.test.ts create mode 100644 server/src/services/native-runtime/remote-launch-assets.ts create mode 100644 tests/runner-e2e/codie-latency-diagnostic.test.ts diff --git a/doc/plans/2026-10-16-native-latency-diagnostic.md b/doc/plans/2026-10-16-native-latency-diagnostic.md new file mode 100644 index 00000000000..773f4e26298 --- /dev/null +++ b/doc/plans/2026-10-16-native-latency-diagnostic.md @@ -0,0 +1,107 @@ +# Native latency diagnostic — 2026-10-16 + +Status: the explicit local-controller/real-Daytona comparison passes. +This branch also overlaps verified sandbox launch assets. PAP is unchanged. + +Use the existing Product E2E basic-response journey and its independent exact +answer, run state, task state, runtime and environment checks. The new manual +suite selects only native Codex `gpt-6.1-sol`, a warm session with a five-minute +idle timeout, and the existing reusable Daytona fixture. It stays out of `--all` +and the ordinary `runner-codex` selector. It uses a 1,000-cent agent hard stop, +one expected run, the original deadlines, and no automatic retry. + +The staging comparison uses subscription authentication. This experiment uses +an API key. A fresh local database and fixture instructions do not reproduce +staging company data, Slack admission or Slack delivery. Separate controller +setup and provider timing. Do not report the result as exact staging or release +qualification. Keep the selected image and executable digests in the evidence. + +Credential-free discovery and the harness typecheck pass. The first typecheck +failed on an unknown fixture payload type; the final typecheck corrects that +cast without changing production types. The initial full support invocation +failed seven cases: two catalog totals changed, one existing Pi version +expectation differs from its source, and four installed-startup cases correctly +reject pnpm's injected `NODE_PATH`. The affected catalog totals are updated. +Direct Node invocation of the catalog, diagnostic, installed-startup and fixture +checks passes 63 cases. The Pi mismatch is retained as a failed attempt and repaired separately below. +No paid or live result is claimed by these checks. + +The local preparation-overlap candidate remains on a separate local branch. +It is not part of this timing baseline. + +## Real Daytona baseline and next slice + +The corrected current Core linux/amd64 runner artifact passes the exact answer, +run/task, native mode and Daytona matchers; cleanup passes. Its traced repetition +on source `ac86e9e7668b` has a 22.878-second active run and 10.233-second queue +while this newly created agent is activated. Whole-case time is 52.788 seconds. +This does not establish a sub-30-second submission-to-answer or Slack result. +The earlier stale local binary failed native provider identity validation and +its failed cleanup grade remains; an independently scoped follow-up confirmed +its nine owned Daytona sandboxes retired. Neither attempt changes PAP. + +The traced repetition stages fresh harness launch material for 621 ms, then +immutable runtime context for 615 ms. Both use disjoint session paths. Extract +this responsibility first with serial semantics, then overlap these two steps +only for a sandbox runner that explicitly advertises concurrent sync. Await both +outcomes before returning, retain launch-material error priority, keep CA staging +last, and leave SSH and undeclared runners serial with first-error termination. +Durable restore, artifact checks, ownership, permissions and process launch stay +in the existing orchestrator. + +Other defensible boundaries in the oversized executor are durable checkpoint +restore, verified artifact preparation, transport/process supervision and provider +credential lifecycle. They remain outside this bounded latency change. The +launch-asset extraction moves about 120 lines with explicit stage callbacks and +no new shared state or runtime dependency on the executor. Check its serial +behavior before introducing concurrency, then use barriers to verify overlap, +wait-for-all failure handling and unsupported transport behavior. Compare one +matched real-Daytona candidate against the saved baseline; provider variation +must remain separate from the launch phase saving. + + +## Candidate result + +On frozen source `e9404a585000`, the same model, Daytona image, Linux runner +binary and local trace setup pass the exact answer and all result matchers. +Cleanup passes. Launch material and immutable context both start at offset +12,785 ms and remain sibling spans under `stage.sync`. Their transfer times are +644 ms and 691 ms. Total staging falls from 1,552 ms to 1,017 ms (535 ms, 34.5%). +Runner bootstrap falls from 4,761 ms to 4,207 ms. + +The active run falls from 22,878 ms to 20,157 ms. This larger difference is not +attributed to staging: agent execution changes from 9,026 ms to 7,015 ms, +settlement increases from 1,600 ms to 2,769 ms, and preparation also varies. +Fresh-agent activation queue changes from 10,233 ms to 6,245 ms. This is one +matched pair, not a statistical tail-latency or Slack acceptance result. + +Before extraction, all 736 existing executor cases pass. The serial extraction +passes 741 cases. The concurrency assertion then fails on serial source as +expected. The candidate passes 764 executor, staging and tracing cases; the +12-case staging suite passes again after its fixture adds required process-result +fields. Full repository typecheck and build pass on the candidate. The earlier +server typecheck failed because that mock omitted signal, pid and startedAt; +production contracts were unchanged. The first sandboxed regression attempt +cannot start a loopback database and is interrupted; its authorized repetition +passes. The broad pre-existing support-suite failures above remain recorded. + +No additional paid iteration or unchanged PAP message is needed for this slice. +Complete final-branch CI and review before any PAP rollout. + + +## Review corrections + +The first review identifies that diagnostic metadata alone does not enforce its +single-attempt promise. The selected task now sets the existing `single_attempt` +policy. A test through the effective retry-limit function fails before this fix +and checks zero retries even when the command requests 99. The measured real +cases already pass explicit `--max-automatic-retries 0`; none is retried. + +The Pi profile pin has a separate stale expectation from the already-merged +preconfigured-home profile update. Update both metadata expectations from 22 to +23. Recomputing the old suite with only profileVersion 22 reproduces the old +pinned hash exactly; the new expected hash changes only that metadata value. +All 72 Pi identity/control/evidence tests pass. The full authorized support run +passes 2,763 cases before adding the retry regression. This does not rewrite the +original failed support invocation. Native production source remains identical +to the measured optimization. diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b79110b2156..2055cae7490 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { stageRemoteLaunchAssets } from "./remote-launch-assets.js"; import { createRunPreparationTiming } from "../execution-preparation-timing.js"; import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js"; import { apiKeyBillingType } from "@paperclipai/adapter-utils/api-billing-type"; @@ -12223,42 +12224,6 @@ async function createRunnerdBackendWithinSessionClaim( return backup; }; - const materializeRemoteHarnessLaunchState = async () => { - if (!remoteTarget || !remoteCommandRunner) return; - for (const directory of persistenceProfile.directories) { - if (directory.location !== "filesystem") continue; - const targetPath = remotePersistencePath(directory); - if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable"); - const escapedTarget = targetPath.replaceAll("'", "'\\''"); - const created = await remoteCommandRunner.execute({ - command: "sh", - args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`], - bypassSession: true, - timeoutMs: 10_000, - }); - if (created.exitCode !== 0 || created.timedOut) { - throw new Error("runner_remote_directory_staging_failed"); - } - - // Codex launch credentials are intentionally excluded from failover - // backups. Re-materialize only those launch-time files into a fresh or - // replacement sandbox after the durable history has been restored. - if (directory.name !== "codex-home") continue; - const localDirectory = resolve(root, directory.name); - for (const name of ["auth.json", "config.toml"] as const) { - const sourcePath = resolve(localDirectory, name); - if (!existsSync(sourcePath)) continue; - await stageRemoteRunnerFile({ - target: remoteTarget, - runner: remoteCommandRunner, - sourcePath, - targetPath: posix.join(targetPath, name), - mode: 0o600, - }); - } - } - }; - const ensureRemoteRunner = async (stageLaunchAssets = true) => { await measureNativeRunnerSpan( input.trace, @@ -12461,67 +12426,24 @@ async function createRunnerdBackendWithinSessionClaim( // this invocation's isolated Codex auth/config. The later launch must // still stage those fresh files even when history was already restored. // Launch material is never recovered from a failover backup. - await measureNativeRunnerSpan(input.trace, "harness_state.launch_material", () => materializeRemoteHarnessLaunchState()); - if ( - remoteTarget && - remoteCommandRunner && - remoteRunnerFilesystemRoot && - sourceRuntimeContext && - remoteRuntimeContext - ) { - await measureNativeRunnerSpan( - input.trace, - "stage.asset.runtime_context", - async () => { - await stageRemoteRunnerDirectory({ - target: remoteTarget, - runner: remoteCommandRunner, - sourcePath: sourceRuntimeContext.instructions.bundle.rootPath, - targetPath: remoteRuntimeContext.instructions.bundle.rootPath, - mode: 0o555, - }); - for ( - let index = 0; - index < sourceRuntimeContext.skills.length; - index += 1 - ) { - await stageRemoteRunnerDirectory({ - target: remoteTarget, - runner: remoteCommandRunner, - sourcePath: - sourceRuntimeContext.skills[index]!.bundle.rootPath, - targetPath: - remoteRuntimeContext.skills[index]!.bundle.rootPath, - mode: 0o555, - }); - } - await stageRemoteRunnerFile({ - target: remoteTarget, - runner: remoteCommandRunner, - sourcePath: resolve(root, "runtime-context.json"), - targetPath: posix.join( - remoteRunnerFilesystemRoot, - "runtime-context.json", - ), - mode: 0o600, - }); - }, - ); - } - if (remoteTarget && remoteCommandRunner && remoteCaBundleMapping) { - const caBundleMapping = remoteCaBundleMapping; - await measureNativeRunnerSpan( - input.trace, - "stage.asset.ca_bundle", - () => - stageRemoteRunnerFile({ - target: remoteTarget, - runner: remoteCommandRunner, - sourcePath: caBundleMapping.sourcePath, - targetPath: caBundleMapping.targetPath, - mode: 0o600, - }), - ); + if (remoteTarget && remoteCommandRunner) { + await stageRemoteLaunchAssets({ + root, + transport: remoteTarget.transport, + runner: remoteCommandRunner, + persistenceDirectories: persistenceProfile.directories.map(directory => ({ + name: directory.name, + location: directory.location, + targetPath: remotePersistencePath(directory), + })), + runtimeContext: remoteRunnerFilesystemRoot && sourceRuntimeContext && remoteRuntimeContext + ? { source: sourceRuntimeContext, target: remoteRuntimeContext, filesystemRoot: remoteRunnerFilesystemRoot } + : null, + caBundleMapping: remoteCaBundleMapping, + stageFile: asset => stageRemoteRunnerFile({ target: remoteTarget, runner: remoteCommandRunner, ...asset }), + stageDirectory: asset => stageRemoteRunnerDirectory({ target: remoteTarget, runner: remoteCommandRunner, ...asset }), + trace: input.trace, + }); } }, { diff --git a/server/src/services/native-runtime/remote-launch-assets.test.ts b/server/src/services/native-runtime/remote-launch-assets.test.ts new file mode 100644 index 00000000000..a322126e34d --- /dev/null +++ b/server/src/services/native-runtime/remote-launch-assets.test.ts @@ -0,0 +1,181 @@ +import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; +import { stageRemoteLaunchAssets } from "./remote-launch-assets.js"; + +function deferred() { + let resolve!: () => void; + const promise = new Promise((done) => { resolve = done; }); + return { promise, resolve }; +} + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +function fixture(options: { transport?: "sandbox" | "ssh"; concurrent?: boolean } = {}) { + const root = mkdtempSync(join(tmpdir(), "native-launch-assets-")); + roots.push(root); + mkdirSync(join(root, "codex-home")); + writeFileSync(join(root, "codex-home", "auth.json"), "fresh test auth"); + writeFileSync(join(root, "codex-home", "config.toml"), "fresh test config"); + const calls: string[] = []; + const source = nativeRuntimeContextFixture(); + const target = nativeRuntimeContextFixture(); + source.instructions.bundle.rootPath = join(root, "instructions"); + target.instructions.bundle.rootPath = "/session/filesystem/context/instructions"; + source.skills = [{ key: "test", runtimeName: "test", versionId: null, bundle: { ...source.instructions.bundle, rootPath: join(root, "skill") } }] as typeof source.skills; + target.skills = [{ ...source.skills[0]!, bundle: { ...source.skills[0]!.bundle, rootPath: "/session/filesystem/context/skills/0-digest" } }]; + const stageFile = vi.fn(async (input: { sourcePath: string; targetPath: string; mode: number }) => { + calls.push(`file:${input.targetPath}`); + }); + const stageDirectory = vi.fn(async (input: { sourcePath: string; targetPath: string; mode: number }) => { + calls.push(`directory:${input.targetPath}`); + }); + const execute = vi.fn(async (_input: { command: string; args?: string[]; bypassSession?: boolean; timeoutMs?: number }) => { + calls.push("mkdir:codex-home"); + return { exitCode: 0, signal: null, pid: null, startedAt: null, timedOut: false, stdout: "", stderr: "" }; + }); + const input: Parameters[0] = { + root, + transport: options.transport ?? "sandbox", + runner: { execute, allowConcurrentSyncOperations: options.concurrent }, + persistenceDirectories: [{ name: "codex-home", location: "filesystem", targetPath: "/session/filesystem/codex-home" }], + runtimeContext: { source, target, filesystemRoot: "/session/filesystem" }, + caBundleMapping: { sourcePath: join(root, "ca.pem"), targetPath: "/session/bin/ca.pem" }, + stageFile, + stageDirectory, + }; + return { input, root, calls, stageFile, stageDirectory, execute }; +} + +describe("remote launch asset staging", () => { + it("retains serial paths, private launch modes and immutable context modes without an opt-in", async () => { + const f = fixture(); + await stageRemoteLaunchAssets(f.input); + expect(f.calls).toEqual([ + "mkdir:codex-home", + "file:/session/filesystem/codex-home/auth.json", + "file:/session/filesystem/codex-home/config.toml", + "directory:/session/filesystem/context/instructions", + "directory:/session/filesystem/context/skills/0-digest", + "file:/session/filesystem/runtime-context.json", + "file:/session/bin/ca.pem", + ]); + expect(f.stageFile.mock.calls.map(([input]) => input.mode)).toEqual([0o600, 0o600, 0o600, 0o600]); + expect(f.stageDirectory.mock.calls.map(([input]) => input.mode)).toEqual([0o555, 0o555]); + expect(f.execute.mock.calls[0]?.[0]).toMatchObject({ bypassSession: true, timeoutMs: 10_000 }); + expect(f.execute.mock.calls[0]?.[0]?.args?.[1]).toContain("install -d -m 0700"); + expect(readFileSync(f.stageFile.mock.calls[0]![0].sourcePath, "utf8")).toBe("fresh test auth"); + }); + + it("stops serial staging before context and CA writes when launch material fails", async () => { + const f = fixture(); + f.stageFile.mockRejectedValueOnce(new Error("launch failure")); + await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("launch failure"); + expect(f.stageDirectory).not.toHaveBeenCalled(); + expect(f.stageFile).toHaveBeenCalledTimes(1); + }); + + it("fails closed before any transfer when the persistence path is unavailable", async () => { + const f = fixture(); + f.input.persistenceDirectories = [{ name: "codex-home", location: "filesystem", targetPath: null }]; + await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("bootstrap_target_unavailable"); + expect(f.execute).not.toHaveBeenCalled(); + expect(f.stageDirectory).not.toHaveBeenCalled(); + }); + + it("stops before CA staging when the immutable context fails", async () => { + const f = fixture(); + f.stageDirectory.mockRejectedValueOnce(new Error("context failure")); + await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("context failure"); + expect(f.stageFile.mock.calls.map(([input]) => input.targetPath)).toEqual([ + "/session/filesystem/codex-home/auth.json", "/session/filesystem/codex-home/config.toml", + ]); + }); + + it("does not require a context or launch credential files for preconfigured homes", async () => { + const f = fixture(); + f.input.persistenceDirectories = []; + f.input.runtimeContext = null; + f.input.caBundleMapping = null; + await stageRemoteLaunchAssets(f.input); + expect(f.calls).toEqual([]); + }); + + it("overlaps disjoint launch material and context only with verified sandbox concurrency", async () => { + const f = fixture({ concurrent: true }); + const started = deferred(), release = deferred(); + f.stageFile.mockImplementationOnce(async () => { started.resolve(); await release.promise; }); + const pending = stageRemoteLaunchAssets(f.input); + await started.promise; + try { + expect(f.stageDirectory).toHaveBeenCalled(); + expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false); + } finally { release.resolve(); await pending; } + expect(f.calls.at(-1)).toBe("file:/session/bin/ca.pem"); + }); + + it.each([ + { transport: "ssh" as const, concurrent: true }, + { transport: "sandbox" as const, concurrent: false }, + { transport: "sandbox" as const }, + ])("keeps unsupported transports serial: %j", async (options) => { + const f = fixture(options); + const started = deferred(), release = deferred(); + f.stageFile.mockImplementationOnce(async () => { started.resolve(); await release.promise; }); + const pending = stageRemoteLaunchAssets(f.input); + await started.promise; + try { expect(f.stageDirectory).not.toHaveBeenCalled(); } + finally { release.resolve(); await pending; } + expect(f.stageDirectory).toHaveBeenCalledTimes(2); + }); + + it("settles the context transfer before exposing a launch-material failure", async () => { + const f = fixture({ concurrent: true }); + const contextStarted = deferred(), releaseContext = deferred(); + const failure = new Error("launch failed first"); + f.stageFile.mockRejectedValueOnce(failure); + f.stageDirectory.mockImplementationOnce(async () => { contextStarted.resolve(); await releaseContext.promise; }); + let settled = false; + let observed: unknown; + const pending = stageRemoteLaunchAssets(f.input).then(() => { settled = true; }, error => { settled = true; observed = error; }); + await contextStarted.promise; + try { + await new Promise(resolve => setImmediate(resolve)); + expect(settled).toBe(false); + } finally { releaseContext.resolve(); await pending; } + expect(observed).toBe(failure); + expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false); + }); + + it("settles launch material before exposing a context failure", async () => { + const f = fixture({ concurrent: true }); + const launchStarted = deferred(), releaseLaunch = deferred(); + const failure = new Error("context failed first"); + f.stageFile.mockImplementationOnce(async () => { launchStarted.resolve(); await releaseLaunch.promise; }); + f.stageDirectory.mockRejectedValueOnce(failure); + let settled = false; + let observed: unknown; + const pending = stageRemoteLaunchAssets(f.input).then(() => { settled = true; }, error => { settled = true; observed = error; }); + await launchStarted.promise; + try { + await new Promise(resolve => setImmediate(resolve)); + expect(settled).toBe(false); + } finally { releaseLaunch.resolve(); await pending; } + expect(observed).toBe(failure); + expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false); + }); + + it("retains launch-material error priority when both parallel transfers fail", async () => { + const f = fixture({ concurrent: true }); + const failure = new Error("launch priority"); + f.stageFile.mockRejectedValueOnce(failure); + f.stageDirectory.mockRejectedValueOnce(new Error("context failure")); + await expect(stageRemoteLaunchAssets(f.input)).rejects.toBe(failure); + expect(f.stageDirectory).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/src/services/native-runtime/remote-launch-assets.ts b/server/src/services/native-runtime/remote-launch-assets.ts new file mode 100644 index 00000000000..fed76ce82a1 --- /dev/null +++ b/server/src/services/native-runtime/remote-launch-assets.ts @@ -0,0 +1,96 @@ +import { existsSync } from "node:fs"; +import { posix, resolve } from "node:path"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import type { NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js"; +import type { NativeRunTrace } from "./native-run-trace.js"; + +type StageAsset = { + sourcePath: string; + targetPath: string; + mode: number; +}; + +type LaunchAssetsInput = { + root: string; + transport: "ssh" | "sandbox"; + runner: CommandManagedRuntimeRunner; + persistenceDirectories: readonly { + name: string; + location: "runner" | "filesystem"; + targetPath: string | null; + }[]; + runtimeContext: { + source: NativeRuntimeContextSnapshot; + target: NativeRuntimeContextSnapshot; + filesystemRoot: string; + } | null; + caBundleMapping: { sourcePath: string; targetPath: string } | null; + stageFile: (input: StageAsset) => Promise; + stageDirectory: (input: StageAsset) => Promise; + trace?: NativeRunTrace; +}; + +export async function stageRemoteLaunchAssets(input: LaunchAssetsInput): Promise { + const measure = async (name: string, operation: () => Promise) => + input.trace ? input.trace.measure(name, operation) : operation(); + const materializeLaunchState = () => measure("harness_state.launch_material", async () => { + for (const directory of input.persistenceDirectories) { + if (directory.location !== "filesystem") continue; + const targetPath = directory.targetPath; + if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable"); + const escapedTarget = targetPath.replaceAll("'", "'\\''"); + const created = await input.runner.execute({ + command: "sh", + args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`], + bypassSession: true, + timeoutMs: 10_000, + }); + if (created.exitCode !== 0 || created.timedOut) { + throw new Error("runner_remote_directory_staging_failed"); + } + if (directory.name !== "codex-home") continue; + const localDirectory = resolve(input.root, directory.name); + for (const name of ["auth.json", "config.toml"] as const) { + const sourcePath = resolve(localDirectory, name); + if (!existsSync(sourcePath)) continue; + await input.stageFile({ sourcePath, targetPath: posix.join(targetPath, name), mode: 0o600 }); + } + } + }); + const stageRuntimeContext = async () => { + const context = input.runtimeContext; + if (!context) return; + await measure("stage.asset.runtime_context", async () => { + await input.stageDirectory({ + sourcePath: context.source.instructions.bundle.rootPath, + targetPath: context.target.instructions.bundle.rootPath, + mode: 0o555, + }); + for (let index = 0; index < context.source.skills.length; index += 1) { + await input.stageDirectory({ + sourcePath: context.source.skills[index]!.bundle.rootPath, + targetPath: context.target.skills[index]!.bundle.rootPath, + mode: 0o555, + }); + } + await input.stageFile({ + sourcePath: resolve(input.root, "runtime-context.json"), + targetPath: posix.join(context.filesystemRoot, "runtime-context.json"), + mode: 0o600, + }); + }); + }; + if (input.transport === "sandbox" && input.runner.allowConcurrentSyncOperations === true) { + const outcomes = await Promise.allSettled([materializeLaunchState(), stageRuntimeContext()]); + for (const outcome of outcomes) { + if (outcome.status === "rejected") throw outcome.reason; + } + } else { + await materializeLaunchState(); + await stageRuntimeContext(); + } + if (input.caBundleMapping) { + const mapping = input.caBundleMapping; + await measure("stage.asset.ca_bundle", () => input.stageFile({ ...mapping, mode: 0o600 })); + } +} diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 6920d750651..4c235fd2b36 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -176,10 +176,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 116, 2, + 1, 3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 116, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(730); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(730); + expect(validateRunnerCatalog()).toHaveLength(731); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(731); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 7dbfd7b7a86..fa4278a0bf7 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1106,6 +1106,31 @@ export const extendedHarnessFileTask: RunnerTaskFixture = { }; export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "codie-latency-diagnostic", + label: "Native Codex latency diagnostic", + description: "One real Daytona turn on the staging-selected model. API-key auth and a fresh fixture do not reproduce the staging subscription, data or Slack delivery.", + groups: ["native"], + manualOnly: true, + profiles: [(() => { + const profile = nativeProfile({ + id: "runner-codex-61-diagnostic", label: "Native Codex 6.1 diagnostic", + provider: "codex", model: "gpt-6.1-sol", credential: "OPENAI_API_KEY", + supportedEnvironments: ["daytona"], + modelQualification: { source: "candidate_runner_profile", qualificationId: "explicit-model-latency-diagnostic" }, + }); + return { ...profile, buildAgent(input: AgentFixtureBuildInput) { + const agent = profile.buildAgent(input); + return { ...agent, budgetMonthlyCents: 1000, + adapterConfig: { ...(agent.adapterConfig as Record), lifecycleMode: "warm", idleTimeoutMs: 300_000 } }; + } }; + })()], + environments: [daytonaWarmEnvironment], + tasks: runnerTasks.filter(task => task.id === "message-marker") + .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), + expectedMatrixSize: 1, + definitionMetadata: { diagnostic: true, modelSource: "observed-staging-run", providerAuth: "api_key", stagingAuth: "subscription", companyData: "fresh-fixture", slackDelivery: false, maximumAutomaticRetries: 0 }, + }, { id: TASK_WORKSPACES_SUITE, label: "Task workspaces without projects", manualOnly: true, description: "Three browser-driven admissions prove deferred repository preparation, stable projectless task files and nested Git persistence across a controller restart.", diff --git a/tests/runner-e2e/codie-latency-diagnostic.test.ts b/tests/runner-e2e/codie-latency-diagnostic.test.ts new file mode 100644 index 00000000000..9e1449ba426 --- /dev/null +++ b/tests/runner-e2e/codie-latency-diagnostic.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from "vitest"; +import { runnerSuites } from "./catalog.js"; +import { effectiveAutomaticRetryLimit } from "./automatic-retry.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; + +describe("explicit native latency diagnostic", () => { + const suiteId = "codie-latency-diagnostic"; + it("selects only one explicit real-Daytona cell and stays out of all", () => { + const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", suiteId])); + expect(cells).toHaveLength(1); + expect(cells[0]).toMatchObject({ profile: { model: "gpt-6.1-sol", generation: "native" }, environment: { id: "daytona" } }); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === suiteId)).toBe(false); + expect(selectRunnerExecutions(parseRunnerSelectors(["--profile", "runner-codex"])).some(cell => cell.suite.id === suiteId)).toBe(false); + }); + it("enforces one paid attempt even when the command requests retries", () => { + const [cell] = selectRunnerExecutions(parseRunnerSelectors(["--suite", suiteId])); + expect(effectiveAutomaticRetryLimit(cell!.task, 99)).toBe(0); + }); + it("declares the authentication and staging-fidelity limits", () => { + expect(runnerSuites.find(suite => suite.id === suiteId)?.definitionMetadata).toMatchObject({ + diagnostic: true, providerAuth: "api_key", stagingAuth: "subscription", companyData: "fresh-fixture", slackDelivery: false, maximumAutomaticRetries: 0, + }); + }); +}); diff --git a/tests/runner-e2e/pi-controls-evidence.test.ts b/tests/runner-e2e/pi-controls-evidence.test.ts index 9cc6ea1c2ae..261e32e1c67 100644 --- a/tests/runner-e2e/pi-controls-evidence.test.ts +++ b/tests/runner-e2e/pi-controls-evidence.test.ts @@ -180,7 +180,7 @@ describe("Pi controls catalog admission", () => { expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true); expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false); - expect(validateRunnerCatalog()).toHaveLength(730); + expect(validateRunnerCatalog()).toHaveLength(731); for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined(); expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow(); }); @@ -189,8 +189,8 @@ describe("Pi controls catalog admission", () => { // Extended v3 states the strict file oracle's task-wide Bash limit. const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile; expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter"); - expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 22, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); - expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 22, + expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 23, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); + expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 23, remoteProcessIdentity: "observer-pid-startTicks-bootId", controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" }); expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 5, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" }); @@ -200,7 +200,7 @@ describe("Pi controls catalog admission", () => { } const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)])); expect(hashes).toEqual({ - "pi-native": "f88d1648fe02ac07b9f43cd5d747131d98844add29fda73746bb1b2bceec0df9", + "pi-native": "3c77ced7edfb99aa12052f451398da7580466a54abadcaf287d31ee12c16382d", "native-active-stop": "2d4fdeeb75bd531b309bd1b35cfa5680ceefdeee6b7155b068dd011ce9901980", "rich-acp-warm-continuity": "331b88d9538a132670789fb7864df7df5f4bf294ae19b62f53d8a02f901774f0", "extended-harnesses": "5de4fac78d4d581bc0954f07b5cf2df2862d37f8b0205325eede6d9ef6d9f5e6",