From 57e85d0d10152c3ec81e23d03b3aa0dfe3c6b4b1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 18 Nov 2026 15:24:11 -0600 Subject: [PATCH] fix(adapter-utils): wait for a replacement GitHub broker after a controller restart (#22209) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in sandbox tasks use managed `git` and `gh` launchers. Each launcher gets a GitHub credential from a broker for each command. > - A controller restart stops the broker of a task. The provider in the sandbox continues to run. The restored controller then writes a replacement broker location to `broker.json`. > - The launcher read `broker.json` one time only. It retried a transport failure for about 1.5 seconds. Then it ran the command without credentials. > - Thus, commands in the restart window failed with "You are not logged into any GitHub hosts" or "could not read Username". Agents lost pull-request checks and pushes, but the GitHub connection was healthy. > - This pull request makes broker-file launchers wait for a bounded time and read `broker.json` again before each retry. > - The benefit is that GitHub commands continue with managed credentials through a controller restart. ## Linked Issues or Issue Description No public issue exists. This is the bug description. **What happened?** A sandbox task ran `gh` or `git` while the controller restarted. The launcher wrote `Paperclip: GitHub broker_transport_unavailable; continuing without managed credentials.` Then `gh` wrote "You are not logged into any GitHub hosts", and `git` wrote "could not read Username". Private reads, pushes and pull-request commands failed. Failures occurred in many tasks at the same time, for some minutes. **Expected behavior** The launcher waits for the replacement broker for a bounded time. Then it uses the managed credential. If the broker does not return, the launcher continues without credentials and shows the same diagnostic as before. **Steps to reproduce** 1. Write a broker-file launcher. Set `broker.json` to a closed local port. 2. Run `gh` through the launcher. 3. After 3 seconds, write a reachable broker location to `broker.json`. 4. The command ran without credentials before the replacement was written. **Relevant logs or output** `Paperclip: GitHub broker_transport_unavailable; continuing without managed credentials.` followed by `You are not logged into any GitHub hosts. To log in, run: gh auth login`. ## What Changed - `packages/adapter-utils/src/github-launcher.ts`: Broker-file launchers retry for up to 60 seconds. Before each retry, they read `broker.json` again. A replacement broker is used when the restored controller publishes it. - The launcher retries two failure types in this wait: a refused or failed connection, and an HTTP 503 answer. With HTTP/2, the sandbox gateway can continue to listen after its controller connection closes. It then answers `503 {"error":"bridge_unavailable"}`. - HTTP 401 and 403 answers are real denials. The launcher does not wait for them. - If `broker.json` cannot be read or parsed (for example during a rewrite), the launcher keeps the last broker location and continues the wait. If `broker.json` is missing, has no broker, or has an empty URL or token, the launcher stops the wait at once, because the access of the task ended. (When a replacement bridge cannot start, the controller writes empty values.) - After the third failure, the launcher writes one line to stderr: it waits for the credential service to return. - Launchers with an inherited broker URL keep the old short retry budget and the old 503 behavior. Their URL cannot change, so a longer wait cannot help. - `packages/adapter-utils/src/github-launcher.test.ts`: Regression tests for three outage types (a closed port, a 503 `bridge_unavailable` gateway, and a partly written `broker.json`). Other tests remove `broker.json` or write empty values to it during the wait, and check that 401 and 403 answers do not wait. - `doc/execution-github-identity.md`: One sentence describes the wait. ## Verification - Failing before the first fix: `waits for a replacement task broker instead of running without credentials (closed port)` failed with `expected 'anonymous' to be 'reattached-run-credential'`. - Failing before the second fix: the `(bridge unavailable)` and `(partly written broker file)` cases failed with `expected 'anonymous' to be 'reattached-run-credential'`. The `(closed port)` case passed. - Failing before the third fix: `stops waiting for an unreachable task broker once its broker file is emptied` timed out after 20 seconds, because the launcher continued to wait. The `removed` case passed. - Passing after the fixes: `npx vitest run packages/adapter-utils/src/github-launcher.test.ts packages/adapter-utils/src/execution-target.test.ts packages/adapter-utils/src/github-launcher-environment.test.ts server/src/services/heartbeat-github-launchers.test.ts server/src/services/native-runtime/native-github-access.test.ts` passed 138 of 138 tests in 5 files. `native-github-access.test.ts` includes the controller restart and launcher re-arm tests. - `npx tsc --noEmit -p packages/adapter-utils` passed. - I did not run the full `pnpm verify`. The local disk was too small for a full install. - Note: Run these tests outside a managed sandbox, or remove the sandbox GitHub launcher from `PATH` and unset the sandbox broker variables. The sandbox values otherwise leak into the tests on `master` too. ## Risks - If the broker does not return, a GitHub command that needs credentials now waits up to about 60 seconds (plus one 10-second request timeout) before it continues without credentials. Before, it waited about 1.5 seconds. Local Git commands do not ask the broker, so they do not wait. - A broker that answers 503 for a long time (for example, when its credential lookup fails) also causes this 60-second wait. Then the launcher continues without credentials, as before. - The launcher sends no credential in the new stderr line. The tests check that broker tokens do not appear in stderr. - Low risk otherwise. The change is in the generated launcher source only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5.5 --- doc/execution-github-identity.md | 2 +- .../adapter-utils/src/github-launcher.test.ts | 86 +++++++++++++++++++ packages/adapter-utils/src/github-launcher.ts | 50 ++++++++--- 3 files changed, 126 insertions(+), 12 deletions(-) diff --git a/doc/execution-github-identity.md b/doc/execution-github-identity.md index 8fcf22ccb00..867e873b2b2 100644 --- a/doc/execution-github-identity.md +++ b/doc/execution-github-identity.md @@ -56,7 +56,7 @@ including the native input required by project creation authorization. Native runners retain a session-owned broker and launcher path across warm turns. The provider keeps an opaque transport token, not a GitHub credential or the previous run's signed capability. After acquiring exclusive session ownership, the controller binds the broker to the current company, agent, task, and run. Requests cannot choose another run or responsible person. The broker rejects requests while idle and discards credential responses if their run binding changed during acquisition. Each operation still rechecks the live run, accepted identity, grants, and trust policy through the shared credential resolver. Changing run IDs alone no longer replaces the provider process; changes to authentication mode, provider credentials, permissions, or other session configuration retain their existing retirement rules. -The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption. +The session broker listens only on controller loopback and accepts only its authenticated GitHub credential operation. Remote executions reach it through the existing authenticated callback bridge. Its transport and launchers are retired with the provider session. If remote bridge startup fails, anonymous launchers keep ordinary work available; the next run retries setup with a fresh session. Controller restart/cold recovery still uses the existing checkpoint and process-recovery rules; the broker's in-memory authority is not persisted for adoption. A remote launcher reads its broker file again while it retries an unreachable broker, for up to 60 seconds. A command that runs during a controller restart thus uses the replacement broker when the restored controller publishes it. A removed broker file ends the wait at once. The native GitHub callback bridge uses HTTP/2 only when the server-selected `enableSandboxDuplexBridge` flag and the sandbox's verified duplex capability diff --git a/packages/adapter-utils/src/github-launcher.test.ts b/packages/adapter-utils/src/github-launcher.test.ts index 3bc2f4dbfd7..de8bc7ebc22 100644 --- a/packages/adapter-utils/src/github-launcher.test.ts +++ b/packages/adapter-utils/src/github-launcher.test.ts @@ -440,3 +440,89 @@ it("captures the current task broker file once per invocation and never falls ba expect((await run()).stdout).toBe("anonymous"); expect(captured).toEqual(["Bearer first-run", "Bearer next-run"]); }); + +// A control-plane restart stops the task's broker while its provider keeps +// running. The replacement broker is published to broker.json a few seconds later. +// An HTTP/2 gateway can outlive its controller and answer 503 instead of refusing. +// The replacement file can also be read while it is only partly written. +it.each(["closed port", "bridge unavailable", "partly written broker file"])( + "waits for a replacement task broker instead of running without credentials (%s)", async (outage) => { + const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-task-github-outage-")); + cleanups.push(() => rm(root, { recursive: true, force: true })); + const bin = path.join(root, "managed"), real = path.join(root, "real"); + await mkdir(bin); await mkdir(real); + await writeFile(path.join(bin, "gh"), githubLauncherSource({ brokerFile: true }), { mode: 0o700 }); + await writeFile(path.join(real, "gh"), '#!/usr/bin/env node\nprocess.stdout.write(process.env.GH_TOKEN || "anonymous");\n', { mode: 0o700 }); + const stopped = createServer((_req, res) => { + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: "bridge_unavailable" })); + }); + await new Promise(resolve => stopped.listen(0, "127.0.0.1", resolve)); + const stoppedUrl = `http://127.0.0.1:${(stopped.address() as { port: number }).port}`; + if (outage === "bridge unavailable") cleanups.push(() => new Promise(resolve => stopped.close(() => resolve()))); + else await new Promise(resolve => stopped.close(() => resolve())); + const replacement = createServer((req, res) => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ status: "available", env: { GH_TOKEN: req.headers.authorization!.slice(7) + "-credential" } })); + }); + await new Promise(resolve => replacement.listen(0, "127.0.0.1", resolve)); + cleanups.push(() => new Promise(resolve => replacement.close(() => resolve()))); + const replacementUrl = `http://127.0.0.1:${(replacement.address() as { port: number }).port}`; + await writeFile(path.join(bin, "broker.json"), JSON.stringify({ url: stoppedUrl, token: "stopped-run" })); + const operation = exec(path.join(bin, "gh"), [], { env: { ...process.env, PATH: `${bin}:${real}:${process.env.PATH}` } }); + if (outage === "partly written broker file") { + await new Promise(resolve => setTimeout(resolve, 1000)); + await writeFile(path.join(bin, "broker.json"), '{"url":'); + } + await new Promise(resolve => setTimeout(resolve, outage === "partly written broker file" ? 2000 : 3000)); + await writeFile(path.join(bin, "broker.json"), JSON.stringify({ url: replacementUrl, token: "reattached-run" })); + const result = await operation; + expect(result.stdout).toBe("reattached-run-credential"); + expect(result.stderr).not.toContain("continuing without managed credentials"); + expect(result.stderr).not.toMatch(/stopped-run|reattached-run/); +}, 20_000); + +// A failed replacement bridge publishes empty broker values instead of removing the file. +it.each(["removed", "emptied"])("stops waiting for an unreachable task broker once its broker file is %s", async (ending) => { + const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-task-github-ended-")); + cleanups.push(() => rm(root, { recursive: true, force: true })); + const bin = path.join(root, "managed"), real = path.join(root, "real"); + await mkdir(bin); await mkdir(real); + await writeFile(path.join(bin, "gh"), githubLauncherSource({ brokerFile: true }), { mode: 0o700 }); + await writeFile(path.join(real, "gh"), '#!/usr/bin/env node\nprocess.stdout.write(process.env.GH_TOKEN || "anonymous");\n', { mode: 0o700 }); + const stopped = createServer(); + await new Promise(resolve => stopped.listen(0, "127.0.0.1", resolve)); + const stoppedUrl = `http://127.0.0.1:${(stopped.address() as { port: number }).port}`; + await new Promise(resolve => stopped.close(() => resolve())); + await writeFile(path.join(bin, "broker.json"), JSON.stringify({ url: stoppedUrl, token: "ended-run" })); + const started = Date.now(); + const operation = exec(path.join(bin, "gh"), [], { env: { ...process.env, PATH: `${bin}:${real}:${process.env.PATH}` } }); + await new Promise(resolve => setTimeout(resolve, 2500)); + if (ending === "removed") await rm(path.join(bin, "broker.json")); + else await writeFile(path.join(bin, "broker.json"), JSON.stringify({ url: "", token: "" })); + const result = await operation; + expect(result.stdout).toBe("anonymous"); + expect(result.stderr).toContain("broker_transport_unavailable"); + expect(Date.now() - started).toBeLessThan(10_000); +}, 20_000); + +it.each([401, 403])("does not wait when the task broker denies the capability (%s)", async (status) => { + const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-task-github-denied-")); + cleanups.push(() => rm(root, { recursive: true, force: true })); + const bin = path.join(root, "managed"), real = path.join(root, "real"); + await mkdir(bin); await mkdir(real); + await writeFile(path.join(bin, "gh"), githubLauncherSource({ brokerFile: true }), { mode: 0o700 }); + await writeFile(path.join(real, "gh"), '#!/usr/bin/env node\nprocess.stdout.write(process.env.GH_TOKEN || "anonymous");\n', { mode: 0o700 }); + let requests = 0; + const server = createServer((_req, res) => { requests++; res.writeHead(status); res.end(); }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + cleanups.push(() => new Promise(resolve => server.close(() => resolve()))); + const url = `http://127.0.0.1:${(server.address() as { port: number }).port}`; + await writeFile(path.join(bin, "broker.json"), JSON.stringify({ url, token: "denied-run" })); + const started = Date.now(); + const result = await exec(path.join(bin, "gh"), [], { env: { ...process.env, PATH: `${bin}:${real}:${process.env.PATH}` } }); + expect(result.stdout).toBe("anonymous"); + expect(result.stderr).toContain("capability_rejected"); + expect(requests).toBe(1); + expect(Date.now() - started).toBeLessThan(3000); +}); diff --git a/packages/adapter-utils/src/github-launcher.ts b/packages/adapter-utils/src/github-launcher.ts index c0da0d578ca..88583516326 100644 --- a/packages/adapter-utils/src/github-launcher.ts +++ b/packages/adapter-utils/src/github-launcher.ts @@ -6,6 +6,10 @@ export function githubLauncherSource(options: { brokerFile?: boolean; interprete || !isAbsolute(options.interpreterPath) || /[\s\0]/u.test(options.interpreterPath))) { throw new Error("GitHub launcher interpreter must be an absolute path without whitespace"); } + // A controller restart stops a task's broker while its provider keeps running, + // then publishes a replacement to broker.json. Only that file can change, so + // only broker-file launchers wait for it; an inherited URL never recovers. + const brokerRecoveryMs = options.brokerFile ? 60_000 : 0; return String.raw`#!${options.interpreterPath ?? "/usr/bin/env node"} const fs = require('node:fs'); const path = require('node:path'); @@ -58,14 +62,22 @@ ${options.brokerFile ? String.raw` delete env.PAPERCLIP_GITHUB_BRIDGE_TOKEN; env.PAPERCLIP_GITHUB_BROKER_URL = ''; env.PAPERCLIP_GITHUB_BROKER_TOKEN = ''; - try { - const broker = JSON.parse(fs.readFileSync(path.join(directory, 'broker.json'), 'utf8')); - if (typeof broker.url === 'string' && typeof broker.token === 'string') { - env.PAPERCLIP_GITHUB_BROKER_URL = broker.url; - env.PAPERCLIP_GITHUB_BROKER_TOKEN = broker.token; - } - } catch {} -` : ''} + // An unreadable file (for example during a rewrite) keeps the last location. + // Only a missing file, or one without a usable broker, means the task's access ended. + const readBroker = () => { + let broker; + try { broker = JSON.parse(fs.readFileSync(path.join(directory, 'broker.json'), 'utf8')); } + catch (error) { return error && error.code === 'ENOENT' ? 'absent' : 'unreadable'; } + // A failed replacement bridge publishes empty values, which also means no broker. + if (!broker || typeof broker.url !== 'string' || typeof broker.token !== 'string' || !broker.url || !broker.token) return 'absent'; + env.PAPERCLIP_GITHUB_BROKER_URL = broker.url; + env.PAPERCLIP_GITHUB_BROKER_TOKEN = broker.token; + return 'present'; + }; + readBroker(); +` : String.raw` + const readBroker = () => 'present'; +`} const diagnostic = (code) => process.stderr.write('Paperclip: GitHub ' + code + '; continuing without managed credentials.\n'); let connected = false; const configRoot = env.GH_CONFIG_DIR || os.tmpdir(); @@ -102,20 +114,30 @@ ${options.brokerFile ? String.raw` try { let response; if (askBroker && base && env.PAPERCLIP_GITHUB_BROKER_TOKEN) { - const url = base.replace(/\/+$/, '').replace(/\/api$/, '') + '/runtime-tools/github/credentials'; // A slow or restarting control plane must not cost the operation its // managed identity, so a failed request is retried before giving up. // Busy (409) responses and transport failures keep separate budgets, and // the body is read inside the retry so a failed read is retried too. + // Transport failures and 503 responses retry until the recovery deadline, + // reading the broker location again so a replacement is used once published. + // Authentication denials (401/403) still end the request at once. + const recoveryDeadline = Date.now() + ${brokerRecoveryMs}; let transportFailures = 0, conflicts = 0, result; for (;;) { try { + const url = (env.PAPERCLIP_GITHUB_BROKER_URL || env.PAPERCLIP_API_URL).replace(/\/+$/, '').replace(/\/api$/, '') + '/runtime-tools/github/credentials'; response = await fetch(url, { method: 'POST', redirect: 'error', signal: AbortSignal.timeout(10000), headers: { authorization: 'Bearer ' + (env.PAPERCLIP_GITHUB_BRIDGE_TOKEN || env.PAPERCLIP_API_KEY || env.PAPERCLIP_GITHUB_BROKER_TOKEN), 'x-paperclip-github-capability': env.PAPERCLIP_GITHUB_BROKER_TOKEN, 'content-type': 'application/json' }, body: '{}', }); + // A sandbox gateway that outlived its controller answers 503 until + // the replacement broker is published, so it waits like a refusal. + if (response.status === 503 && Date.now() < recoveryDeadline) { + await response.arrayBuffer().catch(() => {}); + throw new Error('broker unavailable'); + } if (response.status === 409 && conflicts < 29) { conflicts += 1; await response.arrayBuffer(); @@ -126,8 +148,14 @@ ${options.brokerFile ? String.raw` break; } catch (error) { transportFailures += 1; - if (transportFailures >= 3) throw error; - await new Promise(resolve => setTimeout(resolve, 500 * transportFailures)); + if (transportFailures >= 3 && Date.now() >= recoveryDeadline) throw error; + if (transportFailures === 3) { + process.stderr.write('Paperclip: GitHub credential service is not reachable; waiting up to ' + + Math.ceil((recoveryDeadline - Date.now()) / 1000) + ' seconds for it to return.\n'); + } + await new Promise(resolve => setTimeout(resolve, Math.min(500 * transportFailures, 2000))); + // A removed broker file means the task's access ended; nothing will return. + if (readBroker() === 'absent') throw error; } } if (!response.ok) {