Files
PaperClipAI/server/src/services/native-runtime/native-github-access.ts
T
Devin FoleyandPaperclip cc9ff75cb8 fix(runtime): keep managed GitHub access across a controller restart (#21928)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs give agents managed GitHub access through `gh` and `git`
launchers on the provider's `PATH`
> - A provider process can keep running while the server restarts, and
the next controller adopts it
> - The detaching controller deleted the launcher directory, and the
adopting controller staged launchers in a new random directory
> - The running provider still names the old directory, so every later
`gh` or `git` call ran without credentials
> - This pull request keeps the launcher directory stable for one
provider session and task, and re-arms it after the restart
> - The benefit is that a long run keeps its GitHub identity across a
server deploy or restart

## Linked Issues or Issue Description

No public issue exists. This describes the bug in-PR, with the bug
template labels.

**What happened?**
An agent run on a Daytona sandbox used `gh` with managed GitHub access.
The server restarted during the run (a rolling deploy), and the
controller re-attached to the live provider process. After that, every
`gh` call failed with `You are not logged into any GitHub hosts`. `git
credential fill` returned no token. The agent reported that its token
had "expired" and used the unauthenticated public API for the rest of
the run.

**Steps to reproduce**
1. Start a native run with managed GitHub access on a remote sandbox.
2. Run a `gh` command in the run. It succeeds.
3. Restart the server while the provider process is still running. The
new controller re-attaches to the process.
4. Run a `gh` command again in the same run. It fails with "not logged
in", and no `Paperclip: GitHub ...` diagnostic appears.

**Expected behavior**
After the new controller adopts the provider, `gh` and `git` get
credentials from the new controller, as they did before the restart.

**Root cause**
- `createNativeGitHubAccess` staged the launchers in
`native-session-<random UUID>` and put that path in the provider's
`PATH`, `BASH_ENV`, and `GH_CONFIG_DIR`.
- On a restart detach, `githubAccess.stop()` ran `rm -rf` on that
directory (`detachReleasedWarmNativeSession`, and the run's final
`github_stop` while `nativeRunsDetachingForRestart` is set).
- The adopting controller made a new random directory. The provider
environment cannot change after spawn, so the provider never saw it.
- The shims were gone. `gh` resolved to the real binary with an empty
`GH_CONFIG_DIR`, so no Paperclip diagnostic appeared.

Refs #15354. That PR keeps ACPX sessions across launcher rotation in the
legacy adapter path. This PR covers the native runtime path.

## What Changed

- `createNativeGitHubAccess` takes an optional `launcherKey`. The
launcher directory is `native-github-<sha256(key)>`, so the same
provider session and task always get the same directory.
- The native session executor passes a key built from
`nativeSessionScopeKey` and the issue id.
- `prepareGitHubOperationLaunchers` takes an optional `broker`. It then
stages broker-file launchers and an owner-only `broker.json`. The
launchers already read `broker.json` on each operation in this mode
(`githubLauncherSource({ brokerFile: true })`). A restaged directory
therefore gives the running provider the new endpoint.
- Native access passes `broker` only for remote sandbox targets. A
sandbox provider is the one that outlives a controller restart. Local
launchers live in the shared host temp directory, so they keep the
capability in the provider environment only, as before.
- `stop({ preserveLaunchers: true })` keeps the launcher directory. A
restart detach of an active run and of a released warm session use this
option. A normal stop still removes the directory.
- A final stop does not remove a directory that a newer access in the
same process has already restaged.
- The callback bridge runtime stays in its own per-access directory. A
new bridge never shares a queue directory with an old bridge.
- `syncRemoteTextFiles` takes a `private` flag. It sets `umask 077` for
the whole staging script, so the upload copy, the partial file, and the
final file are never readable by group or other users. Local launcher
files are written to a temporary file and renamed, so a running launcher
never reads a partly written file.

## Verification

- New `native-github-access` test on a sandbox target: a live parent
process gets credentials, the access stops with `preserveLaunchers`, and
a new access with the same key re-arms the directory. The same parent
PID then gets the new controller's credentials, with empty stderr.
`broker.json` has mode `600`. The final stop removes the directory.
- New previous-release cutover test: a provider launched with the old
random-directory layout keeps working without managed GitHub after its
controller removes the directory. The new controller does not resolve
credentials for it. A provider that the new controller launches gets
credentials.
- New local regression: local launcher files do not contain the broker
capability, and there is no `broker.json`.
- New local restart test: a local provider adopted after a restart gets
a Paperclip diagnostic and no credentials. The new controller resolves
nothing for the old endpoint.
- New `sandbox-callback-bridge` test: staging a private file runs with
umask `077` and leaves the file at mode `600`.
- New `native-github-access` test: when a superseded access stops, its
successor's launchers stay usable. A different key gives a different
directory.
- New `native-session-executor` test: a run detached for restart while
active stops its GitHub access with `{ preserveLaunchers: true }`, and
creates it with a `launcherKey`. The existing overlapping-failure test
now asserts `{ preserveLaunchers: false }`.
- The new tests fail on the old behavior. I checked this by forcing
`brokerFile: false`, forcing `preserveLaunchers = false`, removing the
`umask 077` line, and writing `broker.json` for local targets.
- Passed: `native-github-access` (26), `native-session-executor` +
`github-operation-credentials` + `heartbeat-github-launchers` +
`native-stop-accounting` + `heartbeat-project-env` (1640 passed, 7
skipped), `native-runner-restart-recovery.integration` and
`heartbeat-native-restart-suspension`.
- Passed: adapter-utils `execution-target`, `github-launcher`,
`github-launcher-environment`, `sandbox-callback-bridge`; runner
`acpx/github-launcher`, `acpx/environment`,
`codex/codex-security-config`.
- `pnpm --filter @paperclipai/adapter-utils typecheck` and the server
`pnpm typecheck` pass.
- Not run locally: `native-warm-checkpoint-ownership.integration` needs
the compiled `paperclip-runnerd` binary. CI builds it.

## Previous-release story

- No persisted run, session, checkpoint, or database state changes. Only
the in-sandbox launcher layout changes.
- A provider launched by the previous release has a random
`native-session-*` launcher directory in its environment. The previous
release's controller removes that directory at the cutover restart. The
new controller cannot find it, so it does not re-arm it.
- That provider carries forward without managed GitHub until its next
spawn, which is the current behavior. The run is not refused and is not
retried. `gh` and `git` run without managed credentials, as they do
today after a restart. The cutover test covers this.
- A provider that the new release launches gets the stable directory and
survives later restarts.

## Risks

- On sandbox targets, the provider can now read the bridge capability in
`broker.json`, as well as in its environment. The file holds the same
per-access transport token that the environment already holds, not a run
token or a GitHub credential. The file and its staging copies are mode
`600`, and the broker still checks the active run and grants on each
request. Local targets write no `broker.json`.
- Local providers that outlive a restart still lose managed GitHub. This
is not a regression: master deletes the launcher directory at the
detach, so the old local provider also runs without credentials there.
With this PR, the local launchers stay in place and print a `Paperclip:
GitHub ...` diagnostic instead of a silent "not logged in". Re-arming
local providers needs a broker channel that other local runs under the
same user cannot read. That is out of scope here, because `broker.json`
in the shared temp directory was a security finding on this PR.
- On sandbox targets, `git` and `gh` children no longer inherit
`PAPERCLIP_API_URL`, `PAPERCLIP_API_KEY`, or
`PAPERCLIP_GITHUB_BRIDGE_TOKEN`. Broker-file mode removes them, as it
already did in its existing tests.
- After a restart detach, a launcher directory stays in the workspace
runtime directory. If no controller adopts the provider, that directory
stays until a later access with the same key stops, or until the
workspace is removed. It contains no GitHub credential.
- Between the detach and the adoption, `gh` gets a transport error from
the old endpoint. It then prints `broker_transport_unavailable` and
continues without credentials, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-16 12:13:04 -08:00

208 lines
9.6 KiB
TypeScript

import { createHash, randomBytes, randomUUID, timingSafeEqual } from "node:crypto";
import { createServer } from "node:http";
import {
cleanupGitHubOperationLaunchers,
adapterExecutionTargetEnablesSandboxDuplexBridge,
adapterExecutionTargetDuplexObservabilityRecorder,
githubOperationLauncherDirectory,
prepareGitHubOperationLaunchers,
runAdapterExecutionTargetProcess,
startAdapterExecutionTargetPaperclipBridge,
} from "@paperclipai/adapter-utils/execution-target";
import { githubBrokerEnvironment } from "@paperclipai/adapter-utils/github-launcher";
type Binding = { companyId: string; agentId: string; issueId: string; runId: string };
type LauncherInput = Parameters<typeof prepareGitHubOperationLaunchers>[0];
export type NativeGitHubAccess = Awaited<ReturnType<typeof createNativeGitHubAccess>>;
const launcherOwners = new Map<string, symbol>();
/** A process-lifetime transport, with authority only while its controller owns a run.
* No run token or GitHub credential is stored in the provider's environment/files.
* The resolver still checks the active run and current identity/grants per operation.
*/
export async function createNativeGitHubAccess(input: {
scope: Omit<Binding, "runId">;
target: LauncherInput["target"];
cwd: string;
env: NodeJS.ProcessEnv;
resolveCredentials: (binding: Binding) => Promise<unknown>;
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
launcherKey?: string;
}, startBridge = startAdapterExecutionTargetPaperclipBridge) {
const duplexRequested = adapterExecutionTargetEnablesSandboxDuplexBridge(input.target);
const token = randomBytes(32).toString("hex");
const location = { runId: `native-session-${randomUUID()}`, target: input.target };
const launcherLocation = input.launcherKey === undefined ? location : {
runId: `native-github-${createHash("sha256").update(input.launcherKey).digest("hex").slice(0, 32)}`,
target: input.target,
};
const launcherDirectory = githubOperationLauncherDirectory(launcherLocation);
const launcherOwner = Symbol("GitHub launcher owner");
let active: Binding | null = null;
let stopped = false;
let ready = true;
let stopping: Promise<void> | undefined;
let bridge: Awaited<ReturnType<typeof startAdapterExecutionTargetPaperclipBridge>> = null;
const stopBridge = async () => {
bridge?.markOrderlyCompletion?.();
await bridge?.stop();
bridge = null;
};
const server = createServer(async (req, res) => {
res.setHeader("Cache-Control", "no-store");
res.setHeader("Content-Type", "application/json");
req.resume();
const reply = (status: number, body: unknown) => {
res.writeHead(status);
res.end(JSON.stringify(body));
};
const bearer = req.headers.authorization ?? "";
const expected = `Bearer ${token}`;
if (req.headers.origin || req.headers.cookie || req.headers["sec-fetch-site"] ||
Buffer.byteLength(bearer) !== Buffer.byteLength(expected) ||
!timingSafeEqual(Buffer.from(bearer), Buffer.from(expected))) {
reply(403, { error: "GitHub session authentication required" });
return;
}
if (req.method !== "POST" || req.url !== "/runtime-tools/github/credentials") {
reply(404, { error: "Unknown GitHub session operation" });
return;
}
const binding = active;
if (!binding || stopped) {
reply(403, { error: "No active GitHub run" });
return;
}
try {
// Bind at receipt; body/headers cannot select a run or responsible user.
const result = await input.resolveCredentials({ ...binding });
if (active !== binding || stopped) {
reply(403, { error: "GitHub run ended during credential acquisition" });
return;
}
reply(200, result);
} catch (error) {
const status = (error as { status?: number })?.status;
// Never leak secret-store/provider errors. Preserve steering's retry signal.
reply(status === 409 ? 409 : status === 403 ? 403 : 503,
{ error: "GitHub credentials unavailable" });
}
});
server.requestTimeout = 15_000;
server.headersTimeout = 10_000;
const stop = (options: { preserveLaunchers?: boolean } = {}) => stopping ??= (async () => {
stopped = true;
active = null;
server.closeAllConnections();
const removeLaunchers = !options.preserveLaunchers && launcherOwners.get(launcherDirectory) === launcherOwner;
if (removeLaunchers) launcherOwners.delete(launcherDirectory);
const results = await Promise.allSettled([
stopBridge(),
new Promise<void>(resolve => server.close(() => resolve())),
...(launcherLocation !== location ? [cleanupGitHubOperationLaunchers(location)] : []),
...(removeLaunchers ? [cleanupGitHubOperationLaunchers(launcherLocation)] : []),
]);
if (results.some(result => result.status === "rejected")) {
await input.onLog?.("stderr", "[paperclip] GitHub session cleanup incomplete.\n").catch(() => undefined);
}
})();
try {
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", () => { server.off("error", reject); resolve(); });
});
const address = server.address();
if (!address || typeof address === "string") throw new Error("GitHub broker did not listen");
const url = `http://127.0.0.1:${address.port}`;
const startVerifiedBridge = async (enableSandboxDuplexBridge: boolean) => {
bridge = await startBridge({
...location,
runtimeRootDir: input.target?.kind === "remote"
? githubOperationLauncherDirectory(location)
: null,
adapterKey: "native-github",
hostApiToken: token,
hostApiUrl: url,
enableSandboxDuplexBridge,
duplexObservabilityRecorder: adapterExecutionTargetDuplexObservabilityRecorder(input.target),
onLog: input.onLog,
});
if (input.target?.kind === "remote" && !bridge) {
throw new Error("GitHub session requires a remote callback bridge");
}
if (!bridge) return null;
// A gateway greeting does not prove that HTTP/2 can carry a request.
// Probe from the execution target before publishing its endpoint. No run
// is active yet, so this checks transport/authentication without acquiring
// a GitHub credential or performing any provider operation.
const probe = await runAdapterExecutionTargetProcess(location.runId, input.target, "node", ["-e", `
(async () => {
const response = await fetch(process.env.PAPERCLIP_API_URL + '/runtime-tools/github/credentials', {
method: 'POST', redirect: 'error', signal: AbortSignal.timeout(10000),
headers: { authorization: 'Bearer ' + process.env.PAPERCLIP_API_KEY, 'content-type': 'application/json' },
body: '{}',
});
const body = await response.json();
if (response.status !== 403 || body.error !== 'No active GitHub run') process.exitCode = 1;
})().catch(() => { process.exitCode = 1; });
`], { cwd: input.cwd, env: bridge.env, timeoutSec: 15, graceSec: 1, onLog: async () => {} });
if (probe.exitCode !== 0 || probe.timedOut || bridge.readRunDisposition?.().failed) {
throw new Error("GitHub session transport verification failed");
}
return bridge;
};
try {
try {
bridge = await startVerifiedBridge(duplexRequested);
} catch (error) {
if (!duplexRequested || bridge?.env.PAPERCLIP_API_BRIDGE_MODE !== "http2_v1") throw error;
// Only the inactive readiness probe can have crossed this bridge.
// Replacing it cannot replay an agent command or change its identity.
await stopBridge();
await input.onLog?.("stderr", "[paperclip] GitHub HTTP/2 startup check failed; retrying with the file bridge.\n").catch(() => undefined);
bridge = await startVerifiedBridge(false);
}
} catch {
// GitHub remains optional. Stage anonymous wrappers for this session;
// the supervisor retries transport setup on the next run.
ready = false;
// Stop an unverified transport now, including its remote queue polling.
// Retain ownership if cleanup throws so the outer teardown can retry it.
await stopBridge();
await new Promise<void>(resolve => server.close(() => resolve()));
await input.onLog?.("stderr", "[paperclip] GitHub runtime transport unavailable; continuing without managed GitHub access.\n").catch(() => undefined);
}
const broker = {
url: ready ? bridge?.env.PAPERCLIP_API_URL ?? url : "",
token: ready ? bridge?.env.PAPERCLIP_API_KEY ?? token : "",
};
launcherOwners.set(launcherDirectory, launcherOwner);
const env = await prepareGitHubOperationLaunchers({
...launcherLocation, cwd: input.cwd, ...(input.target?.kind === "remote" ? { broker } : {}),
env: {
...githubBrokerEnvironment({ PATH: input.env.PATH }, broker),
// Never retain an old run's bridge authentication override.
PAPERCLIP_GITHUB_BRIDGE_TOKEN: broker.token,
},
});
return {
env,
get ready() { return ready && !stopped && bridge?.readRunDisposition?.().failed !== true; },
duplexRequested,
activate(binding: Binding) {
if (stopped || active) throw new Error("GitHub session is closed or busy");
if (binding.companyId !== input.scope.companyId || binding.agentId !== input.scope.agentId ||
binding.issueId !== input.scope.issueId) throw new Error("GitHub session scope mismatch");
const owner = { ...binding };
active = owner;
return () => { if (active === owner) active = null; };
},
stop,
};
} catch (error) {
await stop().catch(() => undefined);
throw error;
}
}