Files
PaperClipAI/server/src/services/native-runtime/remote-launch-assets.ts
T
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00

206 lines
9.7 KiB
TypeScript

import { existsSync } from "node:fs";
import { lstat, readdir } from "node:fs/promises";
import { posix, resolve } from "node:path";
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
import type { NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
import type { NativeRunTrace } from "./native-run-trace.js";
type StageAsset = {
sourcePath: string;
targetPath: string;
mode: number;
};
type LaunchAssetsInput = {
scope?: "launch" | "recovery";
root: string;
transport: "ssh" | "sandbox";
runner: CommandManagedRuntimeRunner;
persistenceDirectories: readonly {
name: string;
location: "runner" | "filesystem";
targetPath: string | null;
}[];
runtimeContext: {
source: NativeRuntimeContextSnapshot;
target: NativeRuntimeContextSnapshot;
filesystemRoot: string;
} | null;
caBundleMapping: { sourcePath: string; targetPath: string } | null;
stageFile: (input: StageAsset) => Promise<void>;
stageDirectory: (input: StageAsset) => Promise<void>;
isolatedCodexSkills?: { workspaceRoot: string };
trace?: NativeRunTrace;
};
async function assertSelectedSkillsTree(root: string, names: string[]): Promise<void> {
const found = new Set<string>();
const reject = () => { throw new Error("runner_remote_codex_skills_invalid"); };
async function visit(relativePath: string) {
const path = relativePath ? resolve(root, relativePath) : root;
const stat = await lstat(path);
if (stat.isSymbolicLink()) reject();
if (stat.isDirectory()) {
if (relativePath && !names.some(name => relativePath === name || relativePath.startsWith(`${name}/`) || name.startsWith(`${relativePath}/`))) reject();
for (const entry of await readdir(path)) await visit(relativePath ? `${relativePath}/${entry}` : entry);
} else {
if (!stat.isFile() || (stat.mode & 0o222) !== 0 || !names.some(name => relativePath.startsWith(`${name}/`))) reject();
for (const name of names) if (relativePath === `${name}/SKILL.md`) found.add(name);
}
}
await visit("");
if (found.size !== names.length) reject();
}
// Launch preparation owns this isolated directory and runs only after the old
// provider has stopped. Never follow a retained symlink while removing stale
// assignments; auth, config and provider conversation history are not touched.
const validateRemoteCodexHome = `
const fs = require("node:fs"), path = require("node:path");
const home = path.resolve(process.argv[1]), workspace = path.resolve(process.argv[2]);
const relative = path.relative(workspace, home);
if (!relative || relative === ".." || relative.startsWith(".." + path.sep) || path.isAbsolute(relative)) throw Error("unsafe_codex_home");
// The execution target authorizes the workspace anchor. Resolve only that
// anchor (which can have an OS alias), then refuse links below it.
let current = fs.realpathSync(workspace);
if (!fs.lstatSync(current).isDirectory()) throw Error("unsafe_workspace_root");
for (const segment of relative.split(path.sep)) {
current = path.join(current, segment);
if (!fs.lstatSync(current, { throwIfNoEntry: false })) fs.mkdirSync(current, { mode: 0o700 });
const stat = fs.lstatSync(current);
if (!stat.isDirectory() || stat.isSymbolicLink()) throw Error("unsafe_codex_home");
}
for (const name of ["auth.json", "config.toml"]) {
const file = fs.lstatSync(path.join(current, name), { throwIfNoEntry: false });
if (file && (!file.isFile() || file.isSymbolicLink() || file.nlink > 1)) throw Error("unsafe_codex_launch_file");
}
`;
const clearRemoteCodexSkills = `${validateRemoteCodexHome}
const skills = path.join(current, "skills");
function unseal(directory) {
const stat = fs.lstatSync(directory);
if (stat.isSymbolicLink() || !stat.isDirectory()) throw Error("unsafe_skills_root");
fs.chmodSync(directory, 0o700);
for (const name of fs.readdirSync(directory)) {
const child = path.join(directory, name), stat = fs.lstatSync(child);
if (stat.isDirectory() && !stat.isSymbolicLink()) unseal(child);
}
}
if (fs.lstatSync(skills, { throwIfNoEntry: false })) {
unseal(skills);
fs.rmSync(skills, { recursive: true });
}
// Native directory transfers omit the source root and may ignore its mapping
// mode. Create the private discovery root before their mkdir/extract step.
fs.mkdirSync(skills, { mode: 0o700 });
fs.chmodSync(skills, 0o700);
`;
const prepareRemoteCodexRecoveryHome = `${validateRemoteCodexHome}
if (process.argv[3] === "remove-auth") fs.rmSync(path.join(current, "auth.json"), { force: true });
`;
export async function stageRemoteLaunchAssets(input: LaunchAssetsInput): Promise<void> {
const measure = async (name: string, operation: () => Promise<void>) =>
input.trace ? input.trace.measure(name, operation) : operation();
const materializeLaunchState = () => measure("harness_state.launch_material", async () => {
for (const directory of input.persistenceDirectories) {
if (directory.location !== "filesystem") continue;
const targetPath = directory.targetPath;
if (!targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
const escapedTarget = targetPath.replaceAll("'", "'\\''");
const created = await input.runner.execute({
command: "sh",
args: ["-c", `umask 077; install -d -m 0700 '${escapedTarget}'`],
bypassSession: true,
timeoutMs: 10_000,
});
if (created.exitCode !== 0 || created.timedOut) {
throw new Error("runner_remote_directory_staging_failed");
}
if (directory.name !== "codex-home") continue;
const localDirectory = resolve(input.root, directory.name);
for (const name of input.scope === "recovery" ? ["config.toml", "auth.json"] : ["auth.json", "config.toml"]) {
const sourcePath = resolve(localDirectory, name);
if (input.scope === "recovery" && name === "config.toml" && !existsSync(sourcePath)) throw new Error("runner_remote_codex_recovery_config_missing");
if (!existsSync(sourcePath)) continue;
await input.stageFile({ sourcePath, targetPath: posix.join(targetPath, name), mode: 0o600 });
}
}
});
if (input.scope === "recovery") {
const home = input.persistenceDirectories.find(directory => directory.name === "codex-home" && directory.location === "filesystem");
if (!home?.targetPath || !input.isolatedCodexSkills) throw new Error("runner_remote_codex_credentials_path_invalid");
const validated = await input.runner.execute({
command: "node",
args: ["-e", prepareRemoteCodexRecoveryHome, home.targetPath, input.isolatedCodexSkills.workspaceRoot,
existsSync(resolve(input.root, "codex-home", "auth.json")) ? "keep-auth" : "remove-auth"],
bypassSession: true,
timeoutMs: 10_000,
});
if (validated.exitCode !== 0 || validated.timedOut) throw new Error("runner_remote_codex_credentials_path_invalid");
await materializeLaunchState();
return;
}
const stageRuntimeContext = async () => {
const context = input.runtimeContext;
if (!context) return;
await measure("stage.asset.runtime_context", async () => {
await input.stageDirectory({
sourcePath: context.source.instructions.bundle.rootPath,
targetPath: context.target.instructions.bundle.rootPath,
mode: 0o555,
});
for (let index = 0; index < context.source.skills.length; index += 1) {
await input.stageDirectory({
sourcePath: context.source.skills[index]!.bundle.rootPath,
targetPath: context.target.skills[index]!.bundle.rootPath,
mode: 0o555,
});
}
await input.stageFile({
sourcePath: resolve(input.root, "runtime-context.json"),
targetPath: posix.join(context.filesystemRoot, "runtime-context.json"),
mode: 0o600,
});
});
};
const stageCodexSkills = async () => {
if (!input.isolatedCodexSkills) return;
const directory = input.persistenceDirectories.find(entry => entry.name === "codex-home" && entry.location === "filesystem");
if (!directory?.targetPath) throw new Error("runner_harness_state_mismatch: bootstrap_target_unavailable");
const sourcePath = resolve(input.root, "codex-home", "skills");
await measure("stage.asset.codex_skills.verify", () =>
assertSelectedSkillsTree(sourcePath, input.runtimeContext?.source.skills.map(skill => skill.runtimeName) ?? []));
await measure("stage.asset.codex_skills", async () => {
const cleared = await input.runner.execute({
command: "node",
args: ["-e", clearRemoteCodexSkills, directory.targetPath!, input.isolatedCodexSkills!.workspaceRoot],
bypassSession: true,
timeoutMs: 10_000,
});
if (cleared.exitCode !== 0 || cleared.timedOut) throw new Error("runner_remote_codex_skills_reset_failed");
// Match prepareIsolatedCodexHome: the discovery root is private/writable
// for Codex's own built-ins; each selected package remains sealed.
await input.stageDirectory({ sourcePath, targetPath: posix.join(directory.targetPath!, "skills"), mode: 0o700 });
});
};
// Resolve the owned home and remove obsolete assignments before any launch
// file can be written through a retained, redirected path.
await stageCodexSkills();
if (input.transport === "sandbox" && input.runner.allowConcurrentSyncOperations === true) {
const outcomes = await Promise.allSettled([materializeLaunchState(), stageRuntimeContext()]);
for (const outcome of outcomes) {
if (outcome.status === "rejected") throw outcome.reason;
}
} else {
await materializeLaunchState();
await stageRuntimeContext();
}
if (input.caBundleMapping) {
const mapping = input.caBundleMapping;
await measure("stage.asset.ca_bundle", () => input.stageFile({ ...mapping, mode: 0o600 }));
}
}