mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 20:46:28 +01:00
perf(runner): overlap verified sandbox launch assets (#16585)
Native sandbox launch preparation stages fresh harness files and immutable runtime context in sequence. Overlap them only under the verified concurrent-sync capability, settle every started transfer before launch or cleanup, and preserve serial SSH behavior and error priority. A matched real-Daytona comparison reduces staging from 1,552 ms to 1,017 ms. The larger active-run difference includes model variation and is not attributed to this change. Add an explicit single-attempt diagnostic and synchronize stale Pi evidence metadata with its shipped profile. Validation: 764 native runner/tracing cases, 2,764 support cases, repository typecheck/build, fresh exact-head full CI/E2E/security and Apex 5/5 pass. Observability remains a silent no-op when no endpoint is configured. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
f5534717b0
commit
c91f7a6e5f
8 files changed
+459
-104
No files matched your search
@@ -0,0 +1,107 @@
|
||||
# Native latency diagnostic — 2026-10-16
|
||||
|
||||
Status: the explicit local-controller/real-Daytona comparison passes.
|
||||
This branch also overlaps verified sandbox launch assets. PAP is unchanged.
|
||||
|
||||
Use the existing Product E2E basic-response journey and its independent exact
|
||||
answer, run state, task state, runtime and environment checks. The new manual
|
||||
suite selects only native Codex `gpt-6.1-sol`, a warm session with a five-minute
|
||||
idle timeout, and the existing reusable Daytona fixture. It stays out of `--all`
|
||||
and the ordinary `runner-codex` selector. It uses a 1,000-cent agent hard stop,
|
||||
one expected run, the original deadlines, and no automatic retry.
|
||||
|
||||
The staging comparison uses subscription authentication. This experiment uses
|
||||
an API key. A fresh local database and fixture instructions do not reproduce
|
||||
staging company data, Slack admission or Slack delivery. Separate controller
|
||||
setup and provider timing. Do not report the result as exact staging or release
|
||||
qualification. Keep the selected image and executable digests in the evidence.
|
||||
|
||||
Credential-free discovery and the harness typecheck pass. The first typecheck
|
||||
failed on an unknown fixture payload type; the final typecheck corrects that
|
||||
cast without changing production types. The initial full support invocation
|
||||
failed seven cases: two catalog totals changed, one existing Pi version
|
||||
expectation differs from its source, and four installed-startup cases correctly
|
||||
reject pnpm's injected `NODE_PATH`. The affected catalog totals are updated.
|
||||
Direct Node invocation of the catalog, diagnostic, installed-startup and fixture
|
||||
checks passes 63 cases. The Pi mismatch is retained as a failed attempt and repaired separately below.
|
||||
No paid or live result is claimed by these checks.
|
||||
|
||||
The local preparation-overlap candidate remains on a separate local branch.
|
||||
It is not part of this timing baseline.
|
||||
|
||||
## Real Daytona baseline and next slice
|
||||
|
||||
The corrected current Core linux/amd64 runner artifact passes the exact answer,
|
||||
run/task, native mode and Daytona matchers; cleanup passes. Its traced repetition
|
||||
on source `ac86e9e7668b` has a 22.878-second active run and 10.233-second queue
|
||||
while this newly created agent is activated. Whole-case time is 52.788 seconds.
|
||||
This does not establish a sub-30-second submission-to-answer or Slack result.
|
||||
The earlier stale local binary failed native provider identity validation and
|
||||
its failed cleanup grade remains; an independently scoped follow-up confirmed
|
||||
its nine owned Daytona sandboxes retired. Neither attempt changes PAP.
|
||||
|
||||
The traced repetition stages fresh harness launch material for 621 ms, then
|
||||
immutable runtime context for 615 ms. Both use disjoint session paths. Extract
|
||||
this responsibility first with serial semantics, then overlap these two steps
|
||||
only for a sandbox runner that explicitly advertises concurrent sync. Await both
|
||||
outcomes before returning, retain launch-material error priority, keep CA staging
|
||||
last, and leave SSH and undeclared runners serial with first-error termination.
|
||||
Durable restore, artifact checks, ownership, permissions and process launch stay
|
||||
in the existing orchestrator.
|
||||
|
||||
Other defensible boundaries in the oversized executor are durable checkpoint
|
||||
restore, verified artifact preparation, transport/process supervision and provider
|
||||
credential lifecycle. They remain outside this bounded latency change. The
|
||||
launch-asset extraction moves about 120 lines with explicit stage callbacks and
|
||||
no new shared state or runtime dependency on the executor. Check its serial
|
||||
behavior before introducing concurrency, then use barriers to verify overlap,
|
||||
wait-for-all failure handling and unsupported transport behavior. Compare one
|
||||
matched real-Daytona candidate against the saved baseline; provider variation
|
||||
must remain separate from the launch phase saving.
|
||||
|
||||
|
||||
## Candidate result
|
||||
|
||||
On frozen source `e9404a585000`, the same model, Daytona image, Linux runner
|
||||
binary and local trace setup pass the exact answer and all result matchers.
|
||||
Cleanup passes. Launch material and immutable context both start at offset
|
||||
12,785 ms and remain sibling spans under `stage.sync`. Their transfer times are
|
||||
644 ms and 691 ms. Total staging falls from 1,552 ms to 1,017 ms (535 ms, 34.5%).
|
||||
Runner bootstrap falls from 4,761 ms to 4,207 ms.
|
||||
|
||||
The active run falls from 22,878 ms to 20,157 ms. This larger difference is not
|
||||
attributed to staging: agent execution changes from 9,026 ms to 7,015 ms,
|
||||
settlement increases from 1,600 ms to 2,769 ms, and preparation also varies.
|
||||
Fresh-agent activation queue changes from 10,233 ms to 6,245 ms. This is one
|
||||
matched pair, not a statistical tail-latency or Slack acceptance result.
|
||||
|
||||
Before extraction, all 736 existing executor cases pass. The serial extraction
|
||||
passes 741 cases. The concurrency assertion then fails on serial source as
|
||||
expected. The candidate passes 764 executor, staging and tracing cases; the
|
||||
12-case staging suite passes again after its fixture adds required process-result
|
||||
fields. Full repository typecheck and build pass on the candidate. The earlier
|
||||
server typecheck failed because that mock omitted signal, pid and startedAt;
|
||||
production contracts were unchanged. The first sandboxed regression attempt
|
||||
cannot start a loopback database and is interrupted; its authorized repetition
|
||||
passes. The broad pre-existing support-suite failures above remain recorded.
|
||||
|
||||
No additional paid iteration or unchanged PAP message is needed for this slice.
|
||||
Complete final-branch CI and review before any PAP rollout.
|
||||
|
||||
|
||||
## Review corrections
|
||||
|
||||
The first review identifies that diagnostic metadata alone does not enforce its
|
||||
single-attempt promise. The selected task now sets the existing `single_attempt`
|
||||
policy. A test through the effective retry-limit function fails before this fix
|
||||
and checks zero retries even when the command requests 99. The measured real
|
||||
cases already pass explicit `--max-automatic-retries 0`; none is retried.
|
||||
|
||||
The Pi profile pin has a separate stale expectation from the already-merged
|
||||
preconfigured-home profile update. Update both metadata expectations from 22 to
|
||||
23. Recomputing the old suite with only profileVersion 22 reproduces the old
|
||||
pinned hash exactly; the new expected hash changes only that metadata value.
|
||||
All 72 Pi identity/control/evidence tests pass. The full authorized support run
|
||||
passes 2,763 cases before adding the retry regression. This does not rewrite the
|
||||
original failed support invocation. Native production source remains identical
|
||||
to the measured optimization.
|
||||
@@ -1,3 +1,4 @@
|
||||
import { stageRemoteLaunchAssets } from "./remote-launch-assets.js";
|
||||
import { createRunPreparationTiming } from "../execution-preparation-timing.js";
|
||||
import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js";
|
||||
import { apiKeyBillingType } from "@paperclipai/adapter-utils/api-billing-type";
|
||||
@@ -12223,42 +12224,6 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
return backup;
|
||||
};
|
||||
|
||||
const materializeRemoteHarnessLaunchState = async () => {
|
||||
if (!remoteTarget || !remoteCommandRunner) return;
|
||||
for (const directory of persistenceProfile.directories) {
|
||||
if (directory.location !== "filesystem") continue;
|
||||
const targetPath = remotePersistencePath(directory);
|
||||
if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
|
||||
const escapedTarget = targetPath.replaceAll("'", "'\\''");
|
||||
const created = await remoteCommandRunner.execute({
|
||||
command: "sh",
|
||||
args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`],
|
||||
bypassSession: true,
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
if (created.exitCode !== 0 || created.timedOut) {
|
||||
throw new Error("runner_remote_directory_staging_failed");
|
||||
}
|
||||
|
||||
// Codex launch credentials are intentionally excluded from failover
|
||||
// backups. Re-materialize only those launch-time files into a fresh or
|
||||
// replacement sandbox after the durable history has been restored.
|
||||
if (directory.name !== "codex-home") continue;
|
||||
const localDirectory = resolve(root, directory.name);
|
||||
for (const name of ["auth.json", "config.toml"] as const) {
|
||||
const sourcePath = resolve(localDirectory, name);
|
||||
if (!existsSync(sourcePath)) continue;
|
||||
await stageRemoteRunnerFile({
|
||||
target: remoteTarget,
|
||||
runner: remoteCommandRunner,
|
||||
sourcePath,
|
||||
targetPath: posix.join(targetPath, name),
|
||||
mode: 0o600,
|
||||
});
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const ensureRemoteRunner = async (stageLaunchAssets = true) => {
|
||||
await measureNativeRunnerSpan(
|
||||
input.trace,
|
||||
@@ -12461,67 +12426,24 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
// this invocation's isolated Codex auth/config. The later launch must
|
||||
// still stage those fresh files even when history was already restored.
|
||||
// Launch material is never recovered from a failover backup.
|
||||
await measureNativeRunnerSpan(input.trace, "harness_state.launch_material", () => materializeRemoteHarnessLaunchState());
|
||||
if (
|
||||
remoteTarget &&
|
||||
remoteCommandRunner &&
|
||||
remoteRunnerFilesystemRoot &&
|
||||
sourceRuntimeContext &&
|
||||
remoteRuntimeContext
|
||||
) {
|
||||
await measureNativeRunnerSpan(
|
||||
input.trace,
|
||||
"stage.asset.runtime_context",
|
||||
async () => {
|
||||
await stageRemoteRunnerDirectory({
|
||||
target: remoteTarget,
|
||||
runner: remoteCommandRunner,
|
||||
sourcePath: sourceRuntimeContext.instructions.bundle.rootPath,
|
||||
targetPath: remoteRuntimeContext.instructions.bundle.rootPath,
|
||||
mode: 0o555,
|
||||
});
|
||||
for (
|
||||
let index = 0;
|
||||
index < sourceRuntimeContext.skills.length;
|
||||
index += 1
|
||||
) {
|
||||
await stageRemoteRunnerDirectory({
|
||||
target: remoteTarget,
|
||||
runner: remoteCommandRunner,
|
||||
sourcePath:
|
||||
sourceRuntimeContext.skills[index]!.bundle.rootPath,
|
||||
targetPath:
|
||||
remoteRuntimeContext.skills[index]!.bundle.rootPath,
|
||||
mode: 0o555,
|
||||
});
|
||||
}
|
||||
await stageRemoteRunnerFile({
|
||||
target: remoteTarget,
|
||||
runner: remoteCommandRunner,
|
||||
sourcePath: resolve(root, "runtime-context.json"),
|
||||
targetPath: posix.join(
|
||||
remoteRunnerFilesystemRoot,
|
||||
"runtime-context.json",
|
||||
),
|
||||
mode: 0o600,
|
||||
});
|
||||
},
|
||||
);
|
||||
}
|
||||
if (remoteTarget && remoteCommandRunner && remoteCaBundleMapping) {
|
||||
const caBundleMapping = remoteCaBundleMapping;
|
||||
await measureNativeRunnerSpan(
|
||||
input.trace,
|
||||
"stage.asset.ca_bundle",
|
||||
() =>
|
||||
stageRemoteRunnerFile({
|
||||
target: remoteTarget,
|
||||
runner: remoteCommandRunner,
|
||||
sourcePath: caBundleMapping.sourcePath,
|
||||
targetPath: caBundleMapping.targetPath,
|
||||
mode: 0o600,
|
||||
}),
|
||||
);
|
||||
if (remoteTarget && remoteCommandRunner) {
|
||||
await stageRemoteLaunchAssets({
|
||||
root,
|
||||
transport: remoteTarget.transport,
|
||||
runner: remoteCommandRunner,
|
||||
persistenceDirectories: persistenceProfile.directories.map(directory => ({
|
||||
name: directory.name,
|
||||
location: directory.location,
|
||||
targetPath: remotePersistencePath(directory),
|
||||
})),
|
||||
runtimeContext: remoteRunnerFilesystemRoot && sourceRuntimeContext && remoteRuntimeContext
|
||||
? { source: sourceRuntimeContext, target: remoteRuntimeContext, filesystemRoot: remoteRunnerFilesystemRoot }
|
||||
: null,
|
||||
caBundleMapping: remoteCaBundleMapping,
|
||||
stageFile: asset => stageRemoteRunnerFile({ target: remoteTarget, runner: remoteCommandRunner, ...asset }),
|
||||
stageDirectory: asset => stageRemoteRunnerDirectory({ target: remoteTarget, runner: remoteCommandRunner, ...asset }),
|
||||
trace: input.trace,
|
||||
});
|
||||
}
|
||||
},
|
||||
{
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { stageRemoteLaunchAssets } from "./remote-launch-assets.js";
|
||||
|
||||
function deferred() {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>((done) => { resolve = done; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function fixture(options: { transport?: "sandbox" | "ssh"; concurrent?: boolean } = {}) {
|
||||
const root = mkdtempSync(join(tmpdir(), "native-launch-assets-"));
|
||||
roots.push(root);
|
||||
mkdirSync(join(root, "codex-home"));
|
||||
writeFileSync(join(root, "codex-home", "auth.json"), "fresh test auth");
|
||||
writeFileSync(join(root, "codex-home", "config.toml"), "fresh test config");
|
||||
const calls: string[] = [];
|
||||
const source = nativeRuntimeContextFixture();
|
||||
const target = nativeRuntimeContextFixture();
|
||||
source.instructions.bundle.rootPath = join(root, "instructions");
|
||||
target.instructions.bundle.rootPath = "/session/filesystem/context/instructions";
|
||||
source.skills = [{ key: "test", runtimeName: "test", versionId: null, bundle: { ...source.instructions.bundle, rootPath: join(root, "skill") } }] as typeof source.skills;
|
||||
target.skills = [{ ...source.skills[0]!, bundle: { ...source.skills[0]!.bundle, rootPath: "/session/filesystem/context/skills/0-digest" } }];
|
||||
const stageFile = vi.fn(async (input: { sourcePath: string; targetPath: string; mode: number }) => {
|
||||
calls.push(`file:${input.targetPath}`);
|
||||
});
|
||||
const stageDirectory = vi.fn(async (input: { sourcePath: string; targetPath: string; mode: number }) => {
|
||||
calls.push(`directory:${input.targetPath}`);
|
||||
});
|
||||
const execute = vi.fn(async (_input: { command: string; args?: string[]; bypassSession?: boolean; timeoutMs?: number }) => {
|
||||
calls.push("mkdir:codex-home");
|
||||
return { exitCode: 0, signal: null, pid: null, startedAt: null, timedOut: false, stdout: "", stderr: "" };
|
||||
});
|
||||
const input: Parameters<typeof stageRemoteLaunchAssets>[0] = {
|
||||
root,
|
||||
transport: options.transport ?? "sandbox",
|
||||
runner: { execute, allowConcurrentSyncOperations: options.concurrent },
|
||||
persistenceDirectories: [{ name: "codex-home", location: "filesystem", targetPath: "/session/filesystem/codex-home" }],
|
||||
runtimeContext: { source, target, filesystemRoot: "/session/filesystem" },
|
||||
caBundleMapping: { sourcePath: join(root, "ca.pem"), targetPath: "/session/bin/ca.pem" },
|
||||
stageFile,
|
||||
stageDirectory,
|
||||
};
|
||||
return { input, root, calls, stageFile, stageDirectory, execute };
|
||||
}
|
||||
|
||||
describe("remote launch asset staging", () => {
|
||||
it("retains serial paths, private launch modes and immutable context modes without an opt-in", async () => {
|
||||
const f = fixture();
|
||||
await stageRemoteLaunchAssets(f.input);
|
||||
expect(f.calls).toEqual([
|
||||
"mkdir:codex-home",
|
||||
"file:/session/filesystem/codex-home/auth.json",
|
||||
"file:/session/filesystem/codex-home/config.toml",
|
||||
"directory:/session/filesystem/context/instructions",
|
||||
"directory:/session/filesystem/context/skills/0-digest",
|
||||
"file:/session/filesystem/runtime-context.json",
|
||||
"file:/session/bin/ca.pem",
|
||||
]);
|
||||
expect(f.stageFile.mock.calls.map(([input]) => input.mode)).toEqual([0o600, 0o600, 0o600, 0o600]);
|
||||
expect(f.stageDirectory.mock.calls.map(([input]) => input.mode)).toEqual([0o555, 0o555]);
|
||||
expect(f.execute.mock.calls[0]?.[0]).toMatchObject({ bypassSession: true, timeoutMs: 10_000 });
|
||||
expect(f.execute.mock.calls[0]?.[0]?.args?.[1]).toContain("install -d -m 0700");
|
||||
expect(readFileSync(f.stageFile.mock.calls[0]![0].sourcePath, "utf8")).toBe("fresh test auth");
|
||||
});
|
||||
|
||||
it("stops serial staging before context and CA writes when launch material fails", async () => {
|
||||
const f = fixture();
|
||||
f.stageFile.mockRejectedValueOnce(new Error("launch failure"));
|
||||
await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("launch failure");
|
||||
expect(f.stageDirectory).not.toHaveBeenCalled();
|
||||
expect(f.stageFile).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("fails closed before any transfer when the persistence path is unavailable", async () => {
|
||||
const f = fixture();
|
||||
f.input.persistenceDirectories = [{ name: "codex-home", location: "filesystem", targetPath: null }];
|
||||
await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("bootstrap_target_unavailable");
|
||||
expect(f.execute).not.toHaveBeenCalled();
|
||||
expect(f.stageDirectory).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("stops before CA staging when the immutable context fails", async () => {
|
||||
const f = fixture();
|
||||
f.stageDirectory.mockRejectedValueOnce(new Error("context failure"));
|
||||
await expect(stageRemoteLaunchAssets(f.input)).rejects.toThrow("context failure");
|
||||
expect(f.stageFile.mock.calls.map(([input]) => input.targetPath)).toEqual([
|
||||
"/session/filesystem/codex-home/auth.json", "/session/filesystem/codex-home/config.toml",
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not require a context or launch credential files for preconfigured homes", async () => {
|
||||
const f = fixture();
|
||||
f.input.persistenceDirectories = [];
|
||||
f.input.runtimeContext = null;
|
||||
f.input.caBundleMapping = null;
|
||||
await stageRemoteLaunchAssets(f.input);
|
||||
expect(f.calls).toEqual([]);
|
||||
});
|
||||
|
||||
it("overlaps disjoint launch material and context only with verified sandbox concurrency", async () => {
|
||||
const f = fixture({ concurrent: true });
|
||||
const started = deferred(), release = deferred();
|
||||
f.stageFile.mockImplementationOnce(async () => { started.resolve(); await release.promise; });
|
||||
const pending = stageRemoteLaunchAssets(f.input);
|
||||
await started.promise;
|
||||
try {
|
||||
expect(f.stageDirectory).toHaveBeenCalled();
|
||||
expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false);
|
||||
} finally { release.resolve(); await pending; }
|
||||
expect(f.calls.at(-1)).toBe("file:/session/bin/ca.pem");
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ transport: "ssh" as const, concurrent: true },
|
||||
{ transport: "sandbox" as const, concurrent: false },
|
||||
{ transport: "sandbox" as const },
|
||||
])("keeps unsupported transports serial: %j", async (options) => {
|
||||
const f = fixture(options);
|
||||
const started = deferred(), release = deferred();
|
||||
f.stageFile.mockImplementationOnce(async () => { started.resolve(); await release.promise; });
|
||||
const pending = stageRemoteLaunchAssets(f.input);
|
||||
await started.promise;
|
||||
try { expect(f.stageDirectory).not.toHaveBeenCalled(); }
|
||||
finally { release.resolve(); await pending; }
|
||||
expect(f.stageDirectory).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("settles the context transfer before exposing a launch-material failure", async () => {
|
||||
const f = fixture({ concurrent: true });
|
||||
const contextStarted = deferred(), releaseContext = deferred();
|
||||
const failure = new Error("launch failed first");
|
||||
f.stageFile.mockRejectedValueOnce(failure);
|
||||
f.stageDirectory.mockImplementationOnce(async () => { contextStarted.resolve(); await releaseContext.promise; });
|
||||
let settled = false;
|
||||
let observed: unknown;
|
||||
const pending = stageRemoteLaunchAssets(f.input).then(() => { settled = true; }, error => { settled = true; observed = error; });
|
||||
await contextStarted.promise;
|
||||
try {
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(settled).toBe(false);
|
||||
} finally { releaseContext.resolve(); await pending; }
|
||||
expect(observed).toBe(failure);
|
||||
expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false);
|
||||
});
|
||||
|
||||
it("settles launch material before exposing a context failure", async () => {
|
||||
const f = fixture({ concurrent: true });
|
||||
const launchStarted = deferred(), releaseLaunch = deferred();
|
||||
const failure = new Error("context failed first");
|
||||
f.stageFile.mockImplementationOnce(async () => { launchStarted.resolve(); await releaseLaunch.promise; });
|
||||
f.stageDirectory.mockRejectedValueOnce(failure);
|
||||
let settled = false;
|
||||
let observed: unknown;
|
||||
const pending = stageRemoteLaunchAssets(f.input).then(() => { settled = true; }, error => { settled = true; observed = error; });
|
||||
await launchStarted.promise;
|
||||
try {
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(settled).toBe(false);
|
||||
} finally { releaseLaunch.resolve(); await pending; }
|
||||
expect(observed).toBe(failure);
|
||||
expect(f.stageFile.mock.calls.some(([input]) => input.targetPath === "/session/bin/ca.pem")).toBe(false);
|
||||
});
|
||||
|
||||
it("retains launch-material error priority when both parallel transfers fail", async () => {
|
||||
const f = fixture({ concurrent: true });
|
||||
const failure = new Error("launch priority");
|
||||
f.stageFile.mockRejectedValueOnce(failure);
|
||||
f.stageDirectory.mockRejectedValueOnce(new Error("context failure"));
|
||||
await expect(stageRemoteLaunchAssets(f.input)).rejects.toBe(failure);
|
||||
expect(f.stageDirectory).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { posix, resolve } from "node:path";
|
||||
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
import type { NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
|
||||
import type { NativeRunTrace } from "./native-run-trace.js";
|
||||
|
||||
type StageAsset = {
|
||||
sourcePath: string;
|
||||
targetPath: string;
|
||||
mode: number;
|
||||
};
|
||||
|
||||
type LaunchAssetsInput = {
|
||||
root: string;
|
||||
transport: "ssh" | "sandbox";
|
||||
runner: CommandManagedRuntimeRunner;
|
||||
persistenceDirectories: readonly {
|
||||
name: string;
|
||||
location: "runner" | "filesystem";
|
||||
targetPath: string | null;
|
||||
}[];
|
||||
runtimeContext: {
|
||||
source: NativeRuntimeContextSnapshot;
|
||||
target: NativeRuntimeContextSnapshot;
|
||||
filesystemRoot: string;
|
||||
} | null;
|
||||
caBundleMapping: { sourcePath: string; targetPath: string } | null;
|
||||
stageFile: (input: StageAsset) => Promise<void>;
|
||||
stageDirectory: (input: StageAsset) => Promise<void>;
|
||||
trace?: NativeRunTrace;
|
||||
};
|
||||
|
||||
export async function stageRemoteLaunchAssets(input: LaunchAssetsInput): Promise<void> {
|
||||
const measure = async (name: string, operation: () => Promise<void>) =>
|
||||
input.trace ? input.trace.measure(name, operation) : operation();
|
||||
const materializeLaunchState = () => measure("harness_state.launch_material", async () => {
|
||||
for (const directory of input.persistenceDirectories) {
|
||||
if (directory.location !== "filesystem") continue;
|
||||
const targetPath = directory.targetPath;
|
||||
if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
|
||||
const escapedTarget = targetPath.replaceAll("'", "'\\''");
|
||||
const created = await input.runner.execute({
|
||||
command: "sh",
|
||||
args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`],
|
||||
bypassSession: true,
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
if (created.exitCode !== 0 || created.timedOut) {
|
||||
throw new Error("runner_remote_directory_staging_failed");
|
||||
}
|
||||
if (directory.name !== "codex-home") continue;
|
||||
const localDirectory = resolve(input.root, directory.name);
|
||||
for (const name of ["auth.json", "config.toml"] as const) {
|
||||
const sourcePath = resolve(localDirectory, name);
|
||||
if (!existsSync(sourcePath)) continue;
|
||||
await input.stageFile({ sourcePath, targetPath: posix.join(targetPath, name), mode: 0o600 });
|
||||
}
|
||||
}
|
||||
});
|
||||
const stageRuntimeContext = async () => {
|
||||
const context = input.runtimeContext;
|
||||
if (!context) return;
|
||||
await measure("stage.asset.runtime_context", async () => {
|
||||
await input.stageDirectory({
|
||||
sourcePath: context.source.instructions.bundle.rootPath,
|
||||
targetPath: context.target.instructions.bundle.rootPath,
|
||||
mode: 0o555,
|
||||
});
|
||||
for (let index = 0; index < context.source.skills.length; index += 1) {
|
||||
await input.stageDirectory({
|
||||
sourcePath: context.source.skills[index]!.bundle.rootPath,
|
||||
targetPath: context.target.skills[index]!.bundle.rootPath,
|
||||
mode: 0o555,
|
||||
});
|
||||
}
|
||||
await input.stageFile({
|
||||
sourcePath: resolve(input.root, "runtime-context.json"),
|
||||
targetPath: posix.join(context.filesystemRoot, "runtime-context.json"),
|
||||
mode: 0o600,
|
||||
});
|
||||
});
|
||||
};
|
||||
if (input.transport === "sandbox" && input.runner.allowConcurrentSyncOperations === true) {
|
||||
const outcomes = await Promise.allSettled([materializeLaunchState(), stageRuntimeContext()]);
|
||||
for (const outcome of outcomes) {
|
||||
if (outcome.status === "rejected") throw outcome.reason;
|
||||
}
|
||||
} else {
|
||||
await materializeLaunchState();
|
||||
await stageRuntimeContext();
|
||||
}
|
||||
if (input.caBundleMapping) {
|
||||
const mapping = input.caBundleMapping;
|
||||
await measure("stage.asset.ca_bundle", () => input.stageFile({ ...mapping, mode: 0o600 }));
|
||||
}
|
||||
}
|
||||
@@ -176,10 +176,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 116, 2,
|
||||
1, 3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 116, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(730);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(730);
|
||||
expect(validateRunnerCatalog()).toHaveLength(731);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(731);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1106,6 +1106,31 @@ export const extendedHarnessFileTask: RunnerTaskFixture = {
|
||||
};
|
||||
|
||||
export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "codie-latency-diagnostic",
|
||||
label: "Native Codex latency diagnostic",
|
||||
description: "One real Daytona turn on the staging-selected model. API-key auth and a fresh fixture do not reproduce the staging subscription, data or Slack delivery.",
|
||||
groups: ["native"],
|
||||
manualOnly: true,
|
||||
profiles: [(() => {
|
||||
const profile = nativeProfile({
|
||||
id: "runner-codex-61-diagnostic", label: "Native Codex 6.1 diagnostic",
|
||||
provider: "codex", model: "gpt-6.1-sol", credential: "OPENAI_API_KEY",
|
||||
supportedEnvironments: ["daytona"],
|
||||
modelQualification: { source: "candidate_runner_profile", qualificationId: "explicit-model-latency-diagnostic" },
|
||||
});
|
||||
return { ...profile, buildAgent(input: AgentFixtureBuildInput) {
|
||||
const agent = profile.buildAgent(input);
|
||||
return { ...agent, budgetMonthlyCents: 1000,
|
||||
adapterConfig: { ...(agent.adapterConfig as Record<string, unknown>), lifecycleMode: "warm", idleTimeoutMs: 300_000 } };
|
||||
} };
|
||||
})()],
|
||||
environments: [daytonaWarmEnvironment],
|
||||
tasks: runnerTasks.filter(task => task.id === "message-marker")
|
||||
.map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })),
|
||||
expectedMatrixSize: 1,
|
||||
definitionMetadata: { diagnostic: true, modelSource: "observed-staging-run", providerAuth: "api_key", stagingAuth: "subscription", companyData: "fresh-fixture", slackDelivery: false, maximumAutomaticRetries: 0 },
|
||||
},
|
||||
{
|
||||
id: TASK_WORKSPACES_SUITE, label: "Task workspaces without projects", manualOnly: true,
|
||||
description: "Three browser-driven admissions prove deferred repository preparation, stable projectless task files and nested Git persistence across a controller restart.",
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerSuites } from "./catalog.js";
|
||||
import { effectiveAutomaticRetryLimit } from "./automatic-retry.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
|
||||
describe("explicit native latency diagnostic", () => {
|
||||
const suiteId = "codie-latency-diagnostic";
|
||||
it("selects only one explicit real-Daytona cell and stays out of all", () => {
|
||||
const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", suiteId]));
|
||||
expect(cells).toHaveLength(1);
|
||||
expect(cells[0]).toMatchObject({ profile: { model: "gpt-6.1-sol", generation: "native" }, environment: { id: "daytona" } });
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === suiteId)).toBe(false);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--profile", "runner-codex"])).some(cell => cell.suite.id === suiteId)).toBe(false);
|
||||
});
|
||||
it("enforces one paid attempt even when the command requests retries", () => {
|
||||
const [cell] = selectRunnerExecutions(parseRunnerSelectors(["--suite", suiteId]));
|
||||
expect(effectiveAutomaticRetryLimit(cell!.task, 99)).toBe(0);
|
||||
});
|
||||
it("declares the authentication and staging-fidelity limits", () => {
|
||||
expect(runnerSuites.find(suite => suite.id === suiteId)?.definitionMetadata).toMatchObject({
|
||||
diagnostic: true, providerAuth: "api_key", stagingAuth: "subscription", companyData: "fresh-fixture", slackDelivery: false, maximumAutomaticRetries: 0,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -180,7 +180,7 @@ describe("Pi controls catalog admission", () => {
|
||||
expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true);
|
||||
expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false);
|
||||
expect(validateRunnerCatalog()).toHaveLength(730);
|
||||
expect(validateRunnerCatalog()).toHaveLength(731);
|
||||
for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
|
||||
expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow();
|
||||
});
|
||||
@@ -189,8 +189,8 @@ describe("Pi controls catalog admission", () => {
|
||||
// Extended v3 states the strict file oracle's task-wide Bash limit.
|
||||
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
|
||||
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 22, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 22,
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 23, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 23,
|
||||
remoteProcessIdentity: "observer-pid-startTicks-bootId",
|
||||
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" });
|
||||
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 5, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" });
|
||||
@@ -200,7 +200,7 @@ describe("Pi controls catalog admission", () => {
|
||||
}
|
||||
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
|
||||
expect(hashes).toEqual({
|
||||
"pi-native": "f88d1648fe02ac07b9f43cd5d747131d98844add29fda73746bb1b2bceec0df9",
|
||||
"pi-native": "3c77ced7edfb99aa12052f451398da7580466a54abadcaf287d31ee12c16382d",
|
||||
"native-active-stop": "2d4fdeeb75bd531b309bd1b35cfa5680ceefdeee6b7155b068dd011ce9901980",
|
||||
"rich-acp-warm-continuity": "331b88d9538a132670789fb7864df7df5f4bf294ae19b62f53d8a02f901774f0",
|
||||
"extended-harnesses": "5de4fac78d4d581bc0954f07b5cf2df2862d37f8b0205325eede6d9ef6d9f5e6",
|
||||
|
||||
Reference in new issue
Block a user